On the relationship between +/– ratings and event-level performance statistics

1 Introduction

The development of ratings for individual players in team sports has attracted increased attention from researchers in recent years, as the availability and quality of data have improved. Such individual ratings are not straightforward to derive, but potentially offer valuable inputs to fans (McHale et al., 2012), media (Tiedemann et al., 2011), clubs (Macdonald, 2012), coaches (Schultze and Wellbrock, 2018), associations (Hass and Craig, 2018), tournament organizers (McHale et al., 2012), gamblers and bookmakers (Engelmann, 2017), and the players themselves (Tiedemann et al., 2011).

This paper contributes to the understanding of a particular type of rating system, collectively known as plus-minus (PM) ratings. PM ratings have been developed for basketball players (Engelmann, 2017), ice-hockey players (Gramacy et al., 2017), volleyball players (Hass and Craig, 2018), and association football players (Sæbø and Hvattum, 2019). For additional historical references and a discussion of key publications on PM ratings, the reader can consult a recently published review paper (Hvattum, 2019). While simpler forms of PM ratings exist, the focus in this paper is on (regularized) adjusted PM ratings, calculated using some form of (regularized) regression model.

A particular feature of PM ratings is that they only use information about how the score changes during matches and about which players appear on the playing field at different points in time. This contrasts to methods that apply detailed event-based data to derive player ratings, typically focusing on particular aspects of play, such as goal scoring or passing (Szczepański, 2015).

Arguments against the top-down perspective of PM ratings include that it is unreasonable to distribute credit of performance onto players without knowing the actual actions taken by players. However, bottom-up ratings that depend on the actions taken by players typically also miss some aspects of play that the top-down perspective could capture, such as the motivating influence on teammates. In addition, unless tracking-data is available, bottom-up ratings would have to ignore typical off-the-ball actions that may be important in many team sports.

The purpose of this work is to increase our understanding about the relationship between PM ratings and event-based performance statistics of individual players in association football. This relationship has not previously been examined in football, but there is some related work from basketball.

Rosenbaum (2004, 2005) examined adjusted PM ratings for basketball. He also calculated a set of game statistics, measured per 40 minutes of playing time. The adjusted PM ratings were regressed on the game statistics, to estimate how different actions can explain the players’ PM ratings. Using in total 14 independent variables, including pure and derived game statistics, the multiple linear regression model could explain 44% of the variance in the PM rating. Furthermore, the regression coefficients were used to directly calculate a statistical PM rating, which turned out to provide ratings with smaller standard errors, and which in turn could be combined with the initial PM ratings for yet another rating measure.

Fearnhead and Taylor (2011) used a Bayesian framework to calculate adjusted PM ratings, where each player received both an offensive and a defensive rating. Regressing the PM ratings on ten different dependent variables, they were able to explain 41% of the variance of the offensive ratings, but only 3% of the variance of the defensive ratings.

There are few papers comparing the quality of PM ratings and alternative ratings. Engelmann (2011) developed an ad hoc player rating for basketball players that made use of information about individual player actions. The ad hoc model was better at predicting point differentials than a regularized PM rating. Matano et al. (2018) compared regularized PM ratings for football with ratings from the FIFA computer game series, which are partly based on subjective inputs from scouts. They showed that the PM ratings outperformed the FIFA ratings in two out of three seasons. By using the FIFA ratings as priors for the PM ratings, the resulting ratings outperformed the other two ratings in all three seasons.

Given the context of using PM ratings to evaluate individual players in football, the goal of this paper is three-fold. One goal is to test whether PM ratings capture the information contained in event-level player performance statistics. The second goal is to test the extent to which past event-level statistics contain information that is not captured by PM ratings and that can be used to improve predictions for outcomes of future football matches. The third goal is to test whether improved player ratings can be derived by combining the PM ratings with information from player events.

The remainder of the paper is structured as follows. Section 2 provides information about the data used, the calculation of the plus-minus ratings that are analyzed, and the variables derived from event data. Results from regressing PM ratings on the performance statistics are presented in Section 3. Section 4 contains results from tests using PM ratings and performance statistics to predict match outcomes. The attempts at deriving improved player ratings are discussed in Section 5, and Section 6 concludes the paper.

2 Experimental setup

The data consist of individual on-the-ball player events recorded for the seasons 2009–2017 in five European Tier 1 domestic competitions (English Premier League, French Ligue 1, German Bundesliga, Italian Serie A and Spanish La Liga). Cartesian pitch coordinates are given for each event, and passes are encoded with origin and end-point coordinates. Four matches were incomplete and excluded from the analysis, leaving 16,430 matches in the analysis dataset. There were approximately 1,514 on-the-ball events per match. In addition to the on-the-ball player events, the data contain information about substitutions and players sent off, so that at any time the players present on the pitch are known.

2.1 Plus-minus ratings

The PM ratings that we analyze in this work are based on a model presented by Pantuso and Hvattum (2019), which is an extension of the work by Sæbø and Hvattum (2015, 2019). The model is formulated as an unconstrained quadratic program, which generalizes a multiple linear regression model estimated using ridge regression (Tikhonov regularization).

Let M be a set of matches, where each match m ∈ M is split into a set of segments S _m . For our PM ratings this is done by creating one segment for each interval with no changes to the players appearing on the pitch. In other words, segments are created for each substitution and each red card. The duration of segment s ∈ S _m is denoted by d (m, s). Let t = t ^MATCH  (m) denote the time that match m is played, and let T be the current time, at which ratings are calculated. The parameters of the model associated to the timing of events are illustrated in Fig. 1. Each match belongs to a given competition type, such as the English Premier League, or the Spanish La Liga. Denote the competition type of match m by c (m).

OPEN IN VIEWER

Fig. 1

Illustration of the parameters referring to the time of events.

Let h = h (m) and a = a (m) be the two teams involved in match m, where h is the home team, unless the match is played on neutral ground. The goal difference in favour of h in segment s ∈ S _m of match m is denoted by g (m, s). We also define g⁰ (m, s) as the goal difference in favour of h at the beginning of the segment, and g¹ (m, s) as the goal difference at the end of the segment.

The set of players of team t∈ { h, a } that appear on the pitch for segment s is denoted by P_m,s,t. For n = 1, . . . , 4, define r (m, s, n) = 1 if team h has received n red cards and team a has not, r (m, s, n) = -1 if team a has received n red cards and team h has not, and r (m, s, n) = 0 otherwise. We let C be the set of competition types and define C _p as the set of competition types in which player p has participated. Each player p is associated to a set P p SIM of players that are considered to be similar. This set is based on which players have appeared together on the same team for the most number of minutes. The last time that players p and p′ appeared together is denoted by t ^SIM  (p, p′).

The model includes an age profile, as the playing strength of players is expected to improve while young and then decline when getting older. To this end, t ^AGE  (m, p) is defined as the age of player p at the time of match m. A discrete set of possible age values Y ={ y ^MIN , …, y ^MAX  } is defined, and for a given match and player, the age of the player is taken as a convex combination of the possible age values. That is, by introducing parameters u (y, t ^MATCH  (m) , p) to select among the age combinations, it holds that max { min { t ^AGE  (m, p) , y ^MAX  } , y ^MIN  } = ∑_y∈Yu (y, t ^MATCH  (m) , p) y, where at most two values u (y, t ^MATCH  (m) , p) are non-zero, ∑_y∈Yu (y, t ^MATCH  (m) , p) = 1, and 0 ≤ u (y, t ^MATCH  (m) , p) ≤1. As an example, t ^MATCH  (m) = 25.4 would result in the unique combination of u (25, t ^MATCH  (m) , p) = 0.6, u (26, t ^MATCH  (m) , p) = 0.4, and u (y, t ^MATCH  (m) , p) =0. for y∉ { 25, 26 }.

Define the following parameters: λ is a regularization factor, ρ₁ is a discount factor for older observations, ρ₂ and ρ₃ are parameters regarding the importance of the duration of a segment, and ρ₄ is a factor for the importance of a segment depending on the goal difference at the start of the segment. The parameter w ^AGE is a weight to balance the importance of the age factors when considering similarity of players. Finally, w ^SIM is a weight that controls the extent to which ratings of players with few minutes played are shrunk towards 0 or towards the ratings of similar players. The values of the parameters are set as in (Pantuso and Hvattum, 2019).

The variables used in the model can be stated as follows. Let β _p correspond to the base rating of player p. The value of the home advantage in competition c (m) is represented by β c ( m ) H . For a given age y ∈ Y, the rating of a player is adjusted by β y AGE . An adjustment for each competition c ∈ C is given by the variables β c COMP . Let β be the vector of all the decision variables of the model and let V be the set of elements from β.

The full model for our PM ratings can now be written as follows: min β Z = ∑ m ∈ M ∑ s ∈ S m ( w ( m , s ) f LHS ( m , s ) - w ( m , s ) f RHS ( m , s ) ) 2 + ∑ j ∈ V ( f REG ( β j ) ) 2

This is a way to express the estimation of a multiple linear regression model with Tikhonov regularization. In particular, w (m, s) f ^LHS  (m, s) corresponds to the values of the independent variables, w (m, s) f ^RHS  (m, s) represents the value of the dependent variable, and f ^REG  (β _j ) expresses the regularization penalties. The rating model defined here generalizes this by allowing f ^REG  (β _j ) to deviate from the standard regularization terms.

Defining the details of the rating model, w (m, s) is a weight that is used to express the importance of a given segment s of match m:

w ( m , s ) = w TIME ( m , s ) w DURATION ( m , s ) w GOALS ( m , s ) w TIME ( m , s ) = exp ( ρ 1 ( T - t MATCH ( m ) ) ) w DURATION ( m , s ) = ( d ( m , s ) + ρ 2 ) ρ 3 w GOALS ( m , s ) = { ρ 4 if | g 0 ( m , s ) | ≥ 2 and | g 1 ( m , s ) | ≥ 2 1 otherwise

The dependent variable is simply the observed goal difference from the perspective of the home team, multiplied by the weight of the observation: f RHS ( m , s ) = g ( m , s )

The independent variables can be split into those that describe properties of each player, f ^PLAYER  (m, s, p), those that describe properties of a given segment f ^SEGMENT  (m, s), and those that describe a given match, f ^MATCH  (m). These are defined as follows:

90 d ( m , s ) f LHS ( m , s ) = 11 | P m , s , h | ∑ p ∈ P m , s , h f PLAYER ( m , s , p ) - 11 | P m , s , a | ∑ p ∈ P m , s , a f PLAYER ( m , s , p ) + f SEGMENT ( m , s ) + f MATCH ( m ) f SEGMENT ( m , s ) = { ∑ n = 1 , … , 4 r ( m , s , n ) β n HOMERED if ∑ n = 1 , … , 4 r ( m , s , n ) ≥ 0 ∑ n = 1 , … , 4 r ( m , s , n ) β n AWAYRED if ∑ n = 1 , … , 4 r ( m , s , n ) < 0 f MATCH ( m ) = { β c ( m ) H if team h ( m ) has home advantage 0 otherwise . f PLAYER ( m , s , p ) = β p + ∑ y ∈ Y u ( y , t MATCH ( m ) , p ) β y AGE + 1 | C p | ∑ c ∈ C p β c COMP .

Finally, the form of f ^REG  (β _j ) varies for the different regression coefficients. For the competition adjustment coefficients, β c COMP , and the home field advantage coefficients, β c H , the terms correspond to standard ridge regression. The player rating coefficients, β _p , and the age adjustments, β y AGE , are modified by taking advantage of formulating the model as an unconstrained quadratic program: f REG ( β y AGE ) = { λ ( β y AGE - ( β y - 1 AGE + β y + 1 AGE ) / 2 ) if y ∈ Y \ { y MIN , y MAX } 0 if y ∈ { y MIN , y MAX } f REG ( β c COMP ) = λ β c COMP . f REG ( β c H ) = λ β c H . f REG ( β p ) = λ ( f AUX ( p , T , 1 ) - w SIM | P p SIM | ( ∑ p ′ ∈ P p SIM f AUX ( p ′ , t SIM ( p , p ′ ) , w AGE ) ) ) . f AUX ( p , t , w ) = β p + w ∑ y ∈ Y u ( y , t , p ) β y AGE + 1 | C p | ∑ c ∈ C p β c COMP

The estimated rating for player p at time T can now be written as f ^AUX  (p, T, 1). For large data sets, solving the unconstrained quadratic programming can become time consuming. Sæbø and Hvattum (2015) described how a simpler version of the ratings can be estimated using commercially available software. However, in this model, regularization terms for the player ratings f ^REG  (β _p ) and for the age effects f REG ( β y AGE ) have been replaced by more complex expressions. Thus, it may no longer be possible to implement the model in frameworks targeted at estimating multiple linear regression models with ridge regression, as would otherwise be the case. Therefore, an ad hoc method has been implemented to solve the new model, using a gradient search where the calculation of the gradient is implemented using parallel computing.

The calculations outlined above provides a point estimate for the rating of each player. To provide a measure of the uncertainty in the rating of each player, we sugge to use bootstrapping (Fox, 2008, Chapter 21). In this procedure, observations from a regression model are sampled with replacement, until the bootstrap sample has the same size as the original data set. The model is then estimated on the bootstrap sample and the relevant values are recorded, which in our case means recording f ^AUX  (p, T, 1) for each player p. By repeating the bootstrap procedure multiple times, confidence intervals for the rating of a given player can be derived by referring to the percentiles of the recorded ratings for that player.

For this phase of the analysis we use matches in the seasons 2009–2016. Because per 90 KPIs are unstable when a player has played only a few minutes, players with fewer than 540 minutes playing time (equivalent to six full-time matches) are excluded. Twenty-four KPIs that are commonly used to evaluate players are entered into the analysis. One KPI (Saves-to-Shots Ratio) is only defined for goalkeepers. The correlations of the KPIs with PM ratings are reported in Table 3. Statistical significance levels are indicated by *** for p <0.001, ** for p <0.01, and * for p <0.05.

Most of the KPIs correlate positively with PM ratings, indicating some convergence between the two measures of player quality. This is further explored by a series of ordinary least squares regression analyses below. Separate regression equations were estimated for each player position, and in each case, the predictors were selected by a filtering process (Saeys, Inza, & Larrañaga, 2007). The data was resampled according to a five-fold cross-validation scheme, and a generalized additive model was used to screen for a significant relationship between the plus-minus ratings and a spline on each of the candidate predictors (23 KPIs for outfield players, 24 for goalkeepers). The procedure was repeated three times, producing 15 cross-validation samples. Table 4 reports statistics for the R² values obtained.

Predictors that survived this filtering process were used in the positional regression equations reported below. Residuals in each positional equation were tested for normality and heteroscedasticity and outlying/influential observations removed (Peña & Slate, 2006). After removal of these observations (only necessary for defenders and midfielders), all tests for heteroscedasticity and departures from skewness and kurtosis were non-significant at p ≤ 0.05, indicating that the linear model assumptions of normality and constant variance were satisfied.

Tables 5 10 show the regression models for the full positional datasets. Table 11 shows the relative importance of the predictors for each position.

Tables 5 through 10 show that player PM ratings are related to a wide variety of KPIs which are conditional upon player position, and suggest that PM ratings capture a range of positionally relevant skills.

Table 11 shows the relative importance (CAR scores; Zuber and Strimmer, 2011) of the KPIs in each regression equation, and the values in each column sum to 100%. So for example, the value of 8.4% for Successful Aerial Duels/90 in the defenders column means that this KPI contributes 8.4% of the explained variance in PM ratings for defenders, while the value of 1.8% for Assists/90 means that this KPI contributes a much smaller proportion, and is accordingly a less important predictor of PM ratings.

The overall pattern of results seems plausible. For instance, the most important predictors of PM ratings for forwards are Goals Scored/90, Key Passes/90 and Shots on Target/90. For attacking midfielders, the most important predictors are Key Passes/90, Successful Take-Ons/90, and Successful Passes/90. For the other outfield positions, Goals, Key Passes and Shots are less important, and Successful Passes/90 and % Passes Completed dominate the other predictors of PM ratings. For midfielders the dominant predictor is Touches/90, but if this KPI is forced out of the regression equation, it is replaced by Successful Passes/90 as the most dominant predictor.

The shared variance between KPIs and PM ratings indicates a modest degree of convergent validity (Campbell and Fiske, 1959) between the top-down and bottom-up metrics of evaluating players. Nevertheless, it is clear that most of the variance in PM ratings cannot be explained by the bottom-up indicators we have examined here.

PM ratings can be used as a basis for predicting outcomes of football matches where the starting line-ups are known (Sæbø and Hvattum, 2019), by using the difference of the average ratings of home team players and away team players as a single covariate in an ordered logit regression (OLR) model (Greene, 2012). In this section we test whether information from KPIs can be used as additional covariates to improve the prediction quality.

To this end, seasons 2009 through 2015 are used to calculate initial values for PM ratings and KPIs for each player. Seasons 2016 and 2017 provide initial observations for the OLR, where player ratings and KPIs are updated before each new day with matches played. Finally, season 2017 is used as the validation set: before each day of matches, player ratings and KPIs are updated and the regression model is re-estimated using all current information. Then, the predictions are generated for each match of the day, and the quadratic loss (Witten et al., 2011) of the prediction is calculated.

Five models are considered. Model 1 has only one covariate, based on the PM ratings of the players in the starting line-up. Model 2 has 32 covariates, one based on PM ratings, and the rest derived from KPIs. The KPIs included are the 24 from Table 1, in addition to Aerial Duels Won/90, Aerial Duels Lost/90, Unsuccessful Passes/90, Unsuccessful Take-Ons/90, Challenges/90, Saves by Goalkeeper/90, and Fouls Attracted/90. For the KPI-based covariates, the values are first calculated for each individual player in the starting line-up, and then the covariate is calculated as the difference between the averages for the two teams.

To improve the second model, a form of recursive feature elimination (Guyon et al., 2002) is used. The feature selection is performed before considering the data in the validation set and is based on recursively eliminating the covariate that leads to the greatest improvement in the Bayesian information criterion (BIC) until no covariates are left. The final set of covariates retained is the set that provided the best value of BIC throughout the procedure. Starting from Model 2 and performing such feature selection results in Model 3. Models 4 and 5 are created in the same manner as Model 2 and Model 3, respectively, but without including the PM ratings as one of the initial covariates.

Table 12 presents, for each model, the log-likelihood when estimating the model on the full data set, the quadratic loss from out-of-sample predictions in the validation set, and the final regression coefficients. Model 2 and Model 4 are omitted from the table. Their log-likelihood measures are –5112.2 and –5157.6, respectively, while their quadratic loss, which is more importantly addressing their out-of-sample performance, is 0.5684 and 0.5731. Despite their performance thus being slightly better than that of Model 3 and Model 5, we prefer to focus our further analysis on the more parsimonious models. The main reason for this is that the regression coefficients in Model 2 and Model 4 are largely not statistically significant.

The models without PM ratings are worse at predicting future matches, with a higher quadratic loss. On the other hand, Model 3, which adds covariates based on the number of saves and the number of key passes to the model with only PM ratings, is able to predict future matches slightly better. For comparison, a model with no covariates, thus predicting future matches based on past frequencies of outcomes only, obtains a quadratic loss of 0.6442, and a log-likelihood value of –5668.1.

This indicates that the PM ratings do not fully incorporate useful information derived from looking at the number of saves and the number of key passes during a match. As the regression coefficient of Saves/90 is positive, it means that a higher number of saves is associated with worse future results. An increased number of key passes, on the other hand, is associated with higher probabilities of better results. In total, 24 different KPIs were included in Model 2 and Model 4, but were removed by the feature selection for both Model 3 and Model 5

In this section, we discuss two attempts at creating improved PM ratings for individual football players. The first attempt is based on the regression results in Section 3, where the PM ratings were regressed on player KPIs. Rosenbaum (2004, 2005) did this for basketball players, and then suggested a statistical PM rating, where the rating of each player is calculated directly from the regression formulas obtained and the observed KPIs. As a next step, a convex combination of the statistical PM and the regular PM was calculated so as to minimize an error measure.

Rosenbaum (2004, 2005) proposed the statistical PM and its combination with regular PM ratings mainly to reduce the noise in the ratings. This work was performed on adjusted PM, before the use of regularization was popularized (Sill, 2010). As noise is a much less significant issue in the PM ratings used here, the combination of statistical PM and regular PM ratings may be less effective. A difference to the statistical PM ratings of Rosenbaum (2004, 2005), is that our regressions are performed for each player position separately.

To calibrate the overall rating, as a convex combination of statistical PM and regular PM, we consider the log-likelihood of an OLR model fitted on observed match results in the training data, with a single covariate based on the overall rating. The relative weight of statistical PM is varied in a grid search from 0 to 1 in intervals of 0.05. Figure 2 shows the obtained log-likelihoods. The best log-likelihood is obtained for a weight equal to 0.05, which means that there is some value in the statistical PM, but not much. For comparison, in (Rosenbaum, 2005), a weight of 0.4 was used for the statistical PM.

OPEN IN VIEWER

Fig. 2

Tuning of the overall rating combining statistical PM and regular PM.

Table 12 shows the results of Model 6, which uses the combined rating as the basis of a single co-variate to predict future matches. Comparing with the results of Model 1 in Table 12, we can see that the new rating performs only slightly better. Furthermore, Table 14 and Table 15 show the top 20 rated players according to PM ratings and the new combined rating, calculated based on the whole data set. The top-rated players are mostly identical for the two ratings, which is as expected, given that a weight of 0.95 is used on the regular PM ratings in the combined rating.

To evaluate the uncertainty of the rating estimates, Table 14 provides PM ratings based on 500 bootstrap samples (Fox, 2008). For the bootstrapped ratings, the median rating is shown together with the 5th and the 95th percentile ratings, thus providing a 90 % confidence interval for the PM rating. Additionally, we show a 90 % confidence interval for the difference in rating to Thomas Muller, the player with the highest PM rating. The latter is useful when determining whether a given player has a rating that is significantly different from the rating of another player, although only as a descriptive statistic of the players’ performances in the data set.

A second attempt at finding an improved rating is based on the regression results in Section 4. The results of Model 3 presented there, suggested that information about saves and key passes could be helpful to improve the regular PM ratings. To this end, we define an enhanced PM rating by taking a linear combination of the regular PM rating and the individual values for Saves/90 and Key passes/90 for each player. The weights of the linear combination are determined by maximizing the log-likelihood for an OLR model on the training data. A variant of coordinate search (Kolda et al., 2003; Schwefel, 1995) is used as a direct search method to optimize the log-likelihood as a function of the weights of the components in the enhanced PM rating. Starting the search with weights (1.0, 0.0, 0.0), the best weights found are (0.681, –0.0643, 0.0363), with a log-likelihood of –3427.87, which is slightly better than what was obtained using the combination of statistical PM and regular PM ratings.

Table 13 provides results for Model 7, using the enhanced PM rating as the basis for the sole covariate included in OLR to predict future match results. This model performs slightly better than the other models with only a single rating covariate, and even marginally better than Model 3. This confirms that information about saves and key passes can be used to improve the predictive ability of PM ratings.

However, it is also worthwhile to look at Table 14, showing the top 20 ranked players according to the enhanced PM ratings. For the enhanced PM ratings, it was necessary to enforce a cut-off based on the number of minutes played to produce the top 20 list. Whereas the regular PM ratings ensure that players with few minutes played are not represented in the extremes of the rating lists, this is not the case for enhanced PM ratings. Some players with few minutes recorded may obtain very high values for Key Passes/90, and thus will dominate the other players on the list.

When listing only players with more than 540 minutes of playing time, the top 20 list looks very reasonable, and perhaps even more so than the list based on regular PM ratings. However, while the top list for PM ratings contain players of all positions, the enhanced PM ratings seem to favour offensive players, and there are no goalkeepers and no defenders in the top 20.

Our first main finding is that top-down and bottom-up performance indicators for football players share a non-trivial amount of variance. For goalkeepers, regression of KPIs on PM ratings finds an R² of 0.22, while for outfield players the R² values range from 0.31 for defensive midfielders to 0.38 for forwards. These findings are comparable to previous research on NBA player ratings. Rosenbaum (2004, 2005) regressed PM ratings on fourteen different KPIs, reporting an R² of 0.44. Fearnhead and Taylor (2011) regressed offensive and defensive PM ratings on a common set of ten KPIs, finding R² values of 0.42 and 0.03 respectively.

Although these results show that some of the variance of PM ratings for football players can be explained by event data, most of the variance remains unexplained. It may be that some of the variance is due to properties not covered by the event data, such as a player’s ability to create or cover space, pressing that does not lead to ball interaction, and physical abilities such as pace. However, it may also be that a substantial part of the variance is simply noise, as was the case in early versions of PM ratings (Hvattum, 2019).

Our second main finding is that marginal improvements in the prediction of match results can be achieved by combining information from player top-down and bottom-up ratings. However, the application of multiple rating perspectives produces comparatively little change in the predicted probabilities of match results. Correlations between the modelled probabilities of a home win for the pure (Model 1), statistical (Model 6) and enhanced PM models (Model 7) are all over 0.98, as are the correlations between away win probabilities, while the correlations between draw probabilities are all over 0.97.

As the improvement in prediction quality for the versions of PM ratings incorporating event-based data is modest, it may be argued that the PM ratings, aggregated on a team level, provides a reasonable assessment of team quality. However, on a player level, the new enhanced PM rating has some interesting differences to the standard PM rating; while the two methods evaluate players within a given player position similarly, the different player positions are ranked differently. Within position correlations between the standard and enhanced ratings are uniformly high, ranging from 0.85 for goalkeepers to 0.94 for forwards. However, taken over all players, the correlation is 0.61.

For standard PM ratings and the mix of standard and statistical PM ratings, the average ranks for each player position are similar, as shown in Table 16. On the other hand, the enhanced PM rating suggests that forwards are the most influential, followed by attacking midfielders, whereas defenders and goalkeepers are less influential and receive lower ratings.

As two opposing teams will use a similar number of players from each position at any time, this shift in relative evaluation of players of different positions does not in itself lead to significant differences in predictions. For example, since all teams will almost always use exactly one goalkeeper, any shift in ratings of goalkeepers will not have any consequences for predictions made. Therefore, although enhanced PM has slightly better predictive power than the other two variants, it would be premature to conclude that goalkeepers and defenders are less important for winning football matches.

Combining different base learners often leads to reductions in noise and improvements in predictive accuracy (see e.g. Hanson and Salamon, 1990). A natural extension to the present research would therefore be to investigate a stacked ensemble that combines match predictions from a model based on PM ratings alone and a model based only on player KPIs.