Sabermetrics by the sea: Evaluating college players with the Cape Cod Baseball League

1.1 Previous work on the metric

Our project began in the 2019 Cape Cod season, and several players who played on the Cape that summer and were drafted the following year have recently made their debuts in the Major Leagues. We developed the first version of the cWAR metric, largely following the algorithm of FanGraphs, but not including fielding (as accurate hit location data have been unavailable) and using a replacement-level baseline based on the average runs created per plate appearance among temporary players (Kilanowski, 2020). The most notable find from our study, as it turned out, was noticing that Brandon Pfaadt of Bellarmine University, a Division II school at the time, ranked fifth in the league among all pitchers in WAR, even though he was overlooked for the All-Star team and post-season awards; this fall, he became the first player in the study to play in the World Series, as a rookie with the Arizona Diamondbacks. Later, we made refinements to the model, such as replacing fielding-independent pitching (FIP) with weighted on-base average (wOBA) allowed as the basis for calculating pitchers’ WAR and expanded our data to every Cape League season since the offense-dominated summer of 2012, in which Aaron Judge, Kyle Schwarber, and Sean Manaea played (Kilanowski, 2021). At this point, we focused our efforts on investigating the relationship between a player’s cWAR one summer and his rank in the Major League draft the following year.

This version of WAR differs from those formulated from Major League data in three significant ways. First, the Cape League has a markedly different run expectancy matrix and set of weights for wOBA, as seen in Table 9 of the Appendix (Drummey, 2023). The league tends to be lower-scoring than the Majors, mainly because many of the batters are using wooden bats for the first time. This results in lower values in the run expectancy matrix, particularly for the states with no runners on base, as well as higher weights for wOBA (for example, 0.95 instead of 0.90 for a single in a typical year). The weights are also higher due to the greater occurrence of wild pitches, passed balls, and errors on the Cape as compared to the Majors, meaning that reaching base is at a premium in this top-end summer collegiate league.

Second, many Major League algorithms, such as that of FanGraphs, calculate WAR differently for starting pitchers and relief pitchers, but since the same pitchers often play both roles on the Cape, we make no such distinction for cWAR. While the various algorithms use different base statistics for calculating WAR for pitchers (for example, FanGraphs uses FIP while Baseball Reference uses runs allowed per nine innings), we use wOBA allowed, converting this to runs the same way that we do for batters, since this correlates better with team wins and draft position.

Third, the baseline for replacement level for cWAR is taken to be the average of all temporary players (who fill out the rosters early in the season while the full-time players are still playing in the College World Series or with Team USA), rather than an arbitrarily defined baseline as most Major League WAR algorithms do. The result is that the replacement level varies from season to season, but the metric more accurately reflects the player’s performance relative to the talent pool that year.

1.2 Purpose of this study

The goal of this article is to refine and extend the cWAR metric further, which we aim to do in three ways. First, we will investigate the use of skill-interactive earned-run average (SIERA) to calculate pitchers’ WAR, as this statistic takes into account results over which the pitcher has more control, namely, strikeouts, walks, ground balls, and fly balls. Second, we will modify the baseline for replacement level and the calculation of park factors, while also accounting for positional effects, in order to compare batters more accurately to other players at the same position. Third, we will test whether these modifications to the model improve its fit to the actual data. Our main considerations for this paper are to compare players’ cWAR in the 2021 season with their positions in the 2022 Major League draft, and the WAR figures for the 2022 season with teams’ actual winning percentage the same year. Our study shows that wOBA, for pitchers as well as hitters, remains the most effective measure for evaluating players as it provides the best correlation with both draft position and team results, although in some years, expected fielding-independent pitching statistics (xFIP) and SIERA can provide a better fit for the pitchers. Section 2 of the article will detail these modifications to cWAR, while Section 3 will test the new model both qualitatively and quantitatively, while Section 4 will observe conclusions about the model and the directions in which this study can go in the future.

2.1 Introducing SIERA

Previously (Kilanowski, 2021), we found that replacing FIP with wOBA allowed by each pitcher provided a stronger fit between WAR and draft position and team wins than in the original model (Kilanowski, 2020). Wishing to improve the fit even further, we recently considered xFIP and skill-interactive ERA (SIERA) as alternatives to FIP, as they consider each pitcher’s rate of fly balls (and in the latter case, ground balls) allowed. This consideration is especially important in the Cape League, because the field dimensions vary greatly from one park to another, and SIERA is useful for evaluating amateur talent because it measures the marketable skill of inducing ground balls on contact. The calculations for xFIP used the league average proportion of fly balls and line drives that went for home runs from the ten seasons 2012–2022, 2 and for SIERA, we ran a multiple regression over the data from the same ten seasons to find the coefficients, based on how each pitcher’s runs allowed per nine innings correlates with his number of strikeouts, walks, and net ground balls (ground balls minus fly balls, including line drives and pop-ups) per batter faced. Following the example of Baseball Prospectus, we include some, but not all, of the second-order terms (Swartz, 2011). One small adjustment that we made this year was to reduce the criterion for replacement level for pitchers from 40 batters faced to 20, so as to avoid counting some stronger pitchers as replacement level. We can use a lower threshold for pitchers, since pitchers tend to have less playing time than hitters during the summer, as college coaches are wary about preserving their pitchers’ arms for the next spring season. The regression coefficients used for SIERA, along with comparable ones used for the Majors, appear in Table 1.

A word must be said also on our refinement of the calculation of park factors. Previously, we used the ratio of each team’s runs scored at home to runs scored on the road; however, this assumes that every game is of the same length. This is not often the case on the Cape, as three parks (Cotuit, Yarmouth-Dennis, and Brewster) do not have lights, and games are more frequently called short on account of darkness there; several other games are called for weather, and doubleheaders (whether scheduled or containing games that had to be made up) have games of only seven innings each. League rules regarding extra innings have also changed throughout the years. Therefore, our park factor calculations now normalize each team’s number of runs per inning and use the data from the past ten seasons, 2012 through 2022. The park factors for the ten teams from these years appear in Table 2.

One more addition to the model to standardize it and compare each batter more fairly to other players at his position is that of the positional adjustment. Most algorithms for WAR include such an adjustment, often in units of runs, for each position based not only on the average offensive output from batters at that position, but also on the defensive difficulty of the position. While our model does not yet include fielding data, we have decided to consider how to compare each batter to the average hitter at his position (or positions if he played many during the season) rather than the average of all batters, so that the mode will better reflect who is the best available player at a given position, as Major League teams often aim to draft such a player.

To do this, we calculated the average wOBA for all plate appearances, grouped by position, over the ten seasons which we analyzed, and compared it to the league average wOBA (.332) over the same period. The differences appear in Table 3.

As expected with such a large data set (160,499 plate appearances), we observed several anomalies. Pitchers do not ordinarily bat in the Cape League, but when they did (89 times), they showed a difference of –.039, or 39 points of wOBA lower than league average. Eighteen plate appearances were mistakenly attributed to pinch runners with an average difference of +.115; four were labeled “DP” 3 with an average of -.139; and four others were left blank and had a large positive average difference of +.628. Anomalies aside, the data show that the differences from the norm based on positions are small but noticeable: catchers tend to have the lowest wOBA while first basemen and right fielders have the highest; and there is a marked “DH penalty,” and an even larger pinch-hitting penalty (Tango, Lichtman, Dolphin, 2006), giving evidence to the theory that players who do not play the field are less prepared to bat.

When this adjustment is added to the WAR model for the 2022 season, which we did by calculating the average positional adjustment for each batter and subtracting it from his wOBA, some small differences emerge. For most players, especially those who were not primarily catchers, first basemen, right fielders, or designated hitters, their WAR values only changed slightly, as the standard deviation of differences in WAR between the adjusted and non-adjusted models was only 0.0417. Among top players, league MVP Matt Shaw shows a decrease of less than 0.001, while Chris Brito’s WAR decreased by 0.122, and Cooper Ingle’s increased by 0.153, showing the range of adjustments based on one’s primary position. All differences were between -0.2 and +0.2, and were concentrated around 0 in a distribution with lower kurtosis than the normal distribution, as indicated on the histogram in Fig. 1.

OPEN IN VIEWER

Fig. 1

Histogram of the difference between each batter’s WAR with and without offensive positional baseline modifications (the positional adjustment) in the Cape Cod Baseball League, 2022.

4.1 Final observations

The cWAR metric, using wOBA as the basis for evaluating both batters and pitchers, continues to be an effective tool in measuring players’ performance and ability, as it places many of the nation’s top college players on an even playing field so that they can be compared fairly for the Major League draft. For the most recent season, it shows a strong correlation between team WAR and winning percentage, and as before, it agrees with the end-of-season awards more for batters than for pitchers. This is often because several pitchers can impress the scouts more with just a few appearances, while WAR, as a counting statistic, rewards pitchers who throw more innings across the whole season. For example, Jackson Kelley of the Cotuit Kettleers only pitched in five games, but his fastball impressed the scouts and fans and helped him make the All-League team, and his high strikeout rates boosted his xFIP and SIERA numbers. Speaking of which, based on the baselines for these statistics this year, the cWAR numbers based on xFIP appear to be too low, and those based on SIERA to be too high, on the whole. Most striking is the 3.29 value for Ben Hampton, which exceeds the highest WAR value for any player in the study so far, namely, second baseman Nick Gonzales in 2019, 7 but this, combined with the weak (and even negative) correlation with team wins leads us to believe that wOBA allowed is still the best way to evaluate pitchers.

The 2021 season did show a marked increase in the correlation coefficient with SIERA-based WAR (just as xFIP did in 2019), but the advantage for that year disappears when the 2022 draft is considered. For batters, cWAR led the way with a weak yet significant negative correlation with draft position, as players with higher WAR ought to be drafted earlier, while for pitchers, the wOBA-based cWAR formula had the only significant fit of the three, but wOBA itself and even WHIP exhibited a stronger fit. The Spearman correlations were weaker for pitchers across the board, likely because there were more pitchers drafted who pitched fewer innings on the Cape, and thus the distribution was more highly concentrated around 0. For both batters and pitchers, several players were drafted early who had limited action or subpar performance in the CCBL, demonstrating that a player’s collegiate statistics are still more of a factor for scouts, but the Cape League statistics still show a significant correlation with draft position, thus showing that our metric is a useful tool for evaluating college players.

4.2 Future considerations

To improve further on the model, we hope to refine our positional adjustments to include fielding. To do this, we would use the best defensive data available, with a linear weighting system for putouts and assists, as well as wild pitches for pitchers and passed balls for catchers, possibly even including a pitch-framing metric based on the proportions of balls and strikes called when each catcher is behind the plate. Additionally, we can examine the draft further into the past, and even search for a relationship between players’ WAR on the Cape in the Majors for those who make it that far.

The most promising innovation to the model, however, would be for us to include data from TrackMan that have recently been made available with the 2023 season. While on-field performance during a short summer season and with a small sample size may not be the best indicator of a player’s skill, the information from TrackMan will enable us to measure each player’s intrinsic ability, such as with exit velocity for batters and spin rate for pitchers. Adding these values to our cWAR model ought to provide a fuller picture of each prospect, which we can call Total Prospect Rating, and we can use the methods of machine learning to build the best predictive model for when each player should be drafted. A particular application of our study would be to identify which players from smaller colleges or lesser-known conferences, such as Nick Gonzales and Brandon Pfaadt, ought to be given consideration from scouts who may otherwise overlook them. With these refinements, we propose that our model be considered when Major League scouts evaluate collegiate players for the annual draft, which remains one of the last arenas for finding a competitive advantage in acquiring players’ skills in an analytically driven age.