FIFA ranking: Evaluation and path forward

In statistics, a conventional approach to performance evaluation is to rely on a metric, called a scoring function, which relates the result y _t to its prediction obtained from the estimates at hand (here, θ _t ), (Gelman, Hwang, & Vehtari, 2014).

At this point we want to use only the elements that are clearly defined in the FIFA ranking and since the only explicit predictive element defined in the FIFA algorithm is the expected score (3), F ( z t / s ) = 𝔼 [ y ˇ t | z t ] , we will base the evaluation on the metric affected by the mean. Later we will abandon this simplistic approach.

Using the squared prediction error,

m ( z t , y t ) = ( y ˇ t - F ( z t / s ) ) 2 ,(8) averaged over the large number of games, we obtain the Mean Squared Error (MSE) estimate

MSE = 1 T - T ′ ∑ t = T ′ + 1 T m ( z t , y t ) ,(9) where T′ =⌈ T/2 ⌉ is the time-index separating the observations into two approximately equal parts. So the games in the first part are used to initialize the algorithm and the second part is used to calculate the metrics. This separation, which aims to attenuate the initialization effects, is somewhat arbitrary, of course, but should not significantly affect the results for large T.

The MSE in (9) may be treated as an estimate of the expectation, MSE ≈ 𝔼 z t [ 𝔼 y t | z t [ ( y ˇ t - F ( z t / s ) ) 2 ] ] = 𝕧 [ y ˇ t ] + 𝔼 z t [ ( B ( z t , y t ) ) 2 ] ,(10) which highlights the bias-variance decomposition, (Duda, Hart, & Stork, 2001, Ch. 9.3.2) and where 𝕧 [ y ˇ t ] = 𝔼 z t [ 𝕧 [ y ˇ t | z t ] ] is the average conditional variance of y ˇ t , and B ( z t , y t ) = F ( z t / s ) - 𝔼 [ y ˇ t | z t ] is the bias in the estimation of the mean. 5

Therefore, by reducing the (absolute value of the) bias B (z _t , y _t ), that is, by improving the calculation of the expected score F (z _t /s), should manifest itself in a lower value of the MSE, which is calculated as in (9). 6

Using the MSE, we are now able to assess how the values of the importance parameters I _c (or alternatively, K and ξ _c ) affect the expected value of the score (i.e., the estimate of the mean).

We find the coefficients K and/or ξ _c by minimizing the MSE (9) using the following alternate optimization which turned out to converge quickly (and be independent of the initialization) ξ c ← argmin ξ c MSE , c = 1 , … , 8(11) K ← argmin K MSE ;(12) MSE was thus optimized with respect to only one variable at a time, while all the others were kept fixed. We perform optimization (11) on eight weights ξ₁, …, ξ₈ (recall that we set ξ₀ = 1), one optimization (12) on K, and repeat these steps until convergence, defined as an insignificant change in MSE (less than 0.01%). The arrow ← means that, once optimization over a variable ξ _c is finished, the optimal result is used in the following optimizations; the same applies to K, of course. 7

The one-dimensional optimizations (11) and (12) only require one-dimensional line search (e.g., over a grid) and we preferred to avoid derivative-based methods which are not well suited to deal with the complicated functional relationship resulting from the recursive rating algorithm.

The results are shown in Table 2 and we observe the following:

The common update step K increases ten-fold in the optimized solution and it seems that it is the most important contributor to the improvement of the MSE (which changes from MSE _opt  = 0.1340 in the original algorithm to MSE _opt  = 0.1295 in the algorithm with fixed weights ξ _c but larger common adaptation step).

Using a very simple MSE criterion derived from the definitions used by the FIFA algorithm, we obtain results that cast doubt on the optimality/utility of the game-importance parameters, I _c proposed by FIFA.

However, drawing conclusions at this point may be premature. For example, regarding K (which, after optimization should be much larger than 5), it is possible that the relatively short period of observation time (34 months) is not sufficient for small K to guarantee the sufficient convergence but may pay off in a long run, when smaller values of K will improve the performance after the convergence is reached. We will return to this issue in Section 5.1 when analyzing the interaction between the initialization of the algorithm and the scale s.

On the other hand, to address concerns about the weights ξ _c , the situation is quite different. Even after the convergence, the weights associated with different categories should affect the results in a meaningful way. To elucidate this point, we will take a more formal approach and, in Section 3, go back to the “drawing board” to derive the rating algorithm from the first principles.

Before that, however, we will evaluate the impact of the knockout/shootout rules.

The basic algorithmic Equations (1)–(2) guarantee that the teams “exchange” the rating points so that their total stays constant, i.e., ∑ m = 1 M θ t , m = ∑ m = 1 M θ t + 1 , m : this is a well-known property of the Elo rating algorithm, (Langville & Meyer, 2012, Ch. 5). On the other hand, by applying the knockout/shootout rules, we always obtain a new δ_t,m that is equal to or greater than the original one in (2). Thus, the shootout/knockout rules are a source of “inflation” in the rating. In fact, in the analyzed period, the total number of points increased by 2099 (with the initial total being 254680) and this was due to 24 games with the shootout rule (but not the knockout), 90 games with the knockout rule (but not the shootout), and 30 games where both rules were applied.

In the absence of known mathematical principles from which the knockout/shootout rules are derived, our initial hypothesis is that the knockout rule is a heuristics introduced to compensate for the increased value of I _c in the advanced stages of competitions. To test this hypothesis, we will proceed by removing the shootout or/and knockout rules from the algorithm and observe the impact of such removals on the MSE.

The results are shown in the last three columns of Table 2 and we conclude that the prediction capacity of the algorithm is negligibly affected by the shootout rule and it slightly but still notably deteriorates if the knockout rule is removed. 8 Thus, our hypothesis is not supported by the results, even if the argument in favor of using the knockout rule is very weak, as we will also see later in Section 5.2.

To obtain an intuitive understanding of how the removal of shootout/knockout rules affects the results, Table 3 compares the ranking obtained using the FIFA algorithm (first column) with the rating resulting from the modified algorithm in which we (i) eliminate the shootout rule (second column), (ii) eliminate the knockout rule (third column), as well as (iii) eliminate both rules (fourth column).

The Spearman correlation coefficient, ρ (Myers & Well, 2003, Ch. 18.5.3), which quantifies the difference in the rankings of all teams (perfect agreement yields ρ = 1.0) indicates that the changes in the ranking are similar to what was obtained by observing the MSE: comparing the shootout and the knockout rules, the latter affect the results more significantly.

We also show the rankings of the top teams where the changes are not major and the most notable is the switch of ranks between Belgium (BEL) and France (FRA), which can be attributed to a different number of times the teams benefitted from the knockout rules. Indeed, by analyzing the results of the games, we observed that in the original ranking, BEL benefited four times from the knockout rule for a total of 85 points (which would be lost without the rule (6)), while FRA benefited only once, gaining 14 points. 9

Although the knockout rule provides a slight but notable improvement from the prediction point of view, we may still debate whether this heuristics is fair and desirable.

In particular, we note that the points-preserving knockout rule partially ignores the direct comparison between the teams. For example, games in which BEL was not penalized (for losing in the knockout stages) were played against FRA (twice). Thus, despite direct evidence indicating that FRA was able to beat BEL, the knockout rule preserved the points earned by BEL in other games.

In fact, such situations are not surprising and, indeed, the top teams are likely to make it to the final stages of the important competitions and then play against each other in the games where knockout rules are applicable (in case of BEL’s games: World Cup 2018, Euro 2020, and UEFA Nations League 2021). Although these games will provide direct comparison results, the current knockout rule will preserve the points of the losing team.

The rating now depends on the choice of the likelihood function L (z ; y) and here we opt for the Davidson model, (Davidson, 1970), being a particular case of the multinomial model used also in Egidi and Torelli (2021). L ( z t ; H ) = 10 0.5 ( z t + η b t ) 10 0.5 ( z t + η b t ) + κ + 10 - 0.5 ( z t + η b t ) ,(21) L ( z t ; A ) = 10 - 0.5 ( z t + η b t ) 10 0.5 ( z t + η b t ) + κ + 10 - 0.5 ( z t + η b t ) ,(22) L ( z t ; D ) = κ L ( z t ; H ) L ( z t ; A ) ,(23) where the home-field advantage (HFA) is modeled as an apparent increase of the skills of the local team by the value η, the indicator b t = 𝕀 [game t is played in the home-team country] allows us to distinguish between the games played on the home or the neutral venues, 11 and κ is used to adjust to the presence of the draws.

Using (21)–(23) in (19), with straightforward algebra we obtain (see Appendix A and (Szczecinski & Djebbi, 2020, Sec. 3.1)) g ( z ; y t ) = d d z ℓ ( z ; y t )(24) = - ln 10 ( y ˇ t - F κ ( z ) ) ,(25) where y ˇ t is the “score” of the game which we already defined, and

F κ ( z t ) = 1 2 κ + 10 0.5 ( z t + η b t ) 10 0.5 ( z t + η b t ) + κ + 10 - 0.5 ( z t + η b t ) ;(26) by using (21)–(23) we immediately see that F _κ  (z _t ) has the meaning of the conditional expected score, F κ ( z t ) = 𝔼 [ y ˇ t | z t ] = ∑ y ∈ Y y ˇ L ( z t ; y ) .

Therefore, the SG algorithm (20) becomes

θ t + 1 ← θ t + K ξ c t x t ( y ˇ t - F κ ( z t / s ) ) .(27) This Davidson algorithm obviously has the form similar to the Elo/FIFA rating algorithm, see (1)–(3), except that we use F _κ  (z) while the former use F (z). Note that the step K in (27) absorbs the term ln 10 from (25).

It is easy to see that for η = 0 and κ = 0 (i.e., when L (z ; D) =0 and the draws are ignored) we have F₀ (z) = F (z) which is simply a logistic function as in the Elo algorithm. Furthermore, for η = 0 and κ = 2 we obtain F₂ (z) = F (z/2) and thus (27) is again equivalent to the Elo rating algorithm, but with the doubled scale value. These observations explain our preference for the model: it leads to a simple algorithm which generalizes the Elo/FIFA rating algorithm, (Szczecinski & Djebbi, 2020).

Although we conclude that the FIFA rating algorithm may be seen as the instance of the maximum weighted likelihood estimation, this is, of course, a “reverse-engineered” hypothesis because the FIFA document, (FIFA, 2018), does not mention any remotely similar concept.

While our goal is to obtain the on-line rating algorithms where the skills at time t + 1 are calculated from the observations up to time t, we will, for a moment, ignore this on-line rating aspect and rather focus on the evaluation of the model and the optimization criterion that underlie the algorithms.

We thus concentrate on the original problem defined in (16) for the entire set of data, and, in this way, we (i) will not need to remove a significant portion of the data (meant to eliminate the initialization effects during evaluation, see (9)) and (ii) eliminate the limitation of the SG optimization where, by fine-tuning the adaptation step K, the estimation error is traded off against the convergence speed.

We start by noting that the problem (16) is, in general, ill-posed: since the solution depends only on the differences between the skills, z _t , all solutions θ ˆ and θ ˆ + θ o 1 are equivalent because the differences z _t are independent from “origin” value θ_o. To remove this ambiguity, we may regularize the problem as θ ˆ = argmin θ J ( θ )(28) J ( θ ) = ∑ t ∈ T ξ c t ℓ ( z t / s ; y t ) + α 2 s 2 ∥ θ ∥ 2 ,(29) where α is the regularization parameter and we have opted for a so-called ridge regularization (Hastie, Tibshirani, & Friedman, 2009, [Ch. 3.4.1]).

Under the model (21)–(23), the regularized batch-optimization problem (28) is useful to resolve another difficulty. Namely, if there is a team m having registered only wins, i.e., when ∀i _t  = m,  y _t  = H and ∀j _t  = m,  y _t  = A, then (16) cannot be solved (or rather, θ ˆ m → ∞ ) because J ( θ ) does not limit the value of θ _m . Such a solution not only is unattainable numerically but is, in fact, meaningless, and the regularization (28) settles this issue. 12 footnote:ranking.info.

The estimated skills θ ˆ now depend on the weights ξ _c , on the regularization parameter α, and on the model parameters η and κ. If not known, all of these parameters must be optimized.

Regarding the optimization criterion, we recall that the FIFA algorithm only specified the expected score, so the quadratic error (8) allowed us to evaluate the algorithm and stay within the boundaries of its definitions. Now, however, with the explicit skills-outcome model, we may go beyond this limitation and will use the prediction metrics known in machine learning such as the (negated) log-score, (Gelman et al., 2014)

m ls ( z t ; y t ) = ℓ ( z t / s ; y t ) ,(30) often preferred due to its compatibility with the log-likelihood used as the optimization criterion, or the accuracy score, (Lasek & Gagolewski, 2020)

m acc ( z t ; y t ) = 𝕀 [ y t = argmax y L ( z t / s ; y ) ] ,(31) which equals one if the event with the largest predicted probability was actually observed; otherwise it is zero.

Furthermore, thanks to the batch-rating, we are able to consider the entire data set in the performance evaluation by averaging the scoring functions (30) or (31) over all games LS = 1 T ∑ t ∈ T m ls ( x t ⊤ θ ˆ ∖ t , y t ) ,(32) ACC = 1 T ∑ t ∈ T m acc ( x t ⊤ θ ˆ ∖ t , y t ) ,(33) where θ ˆ ∖ t = argmin θ J ∖ t ( θ ) ,(34) J \ t ( θ ) = ∑ l ∈ T l ≠ t ξ c l ℓ ( x l ⊤ θ / s ; y l ) + α 2 s 2 ‖ θ ‖ 2 .(35)

In simple words, for given parameters (α, κ, η, ξ _c ), we find the skills θ ˆ ∖ t from all, but the t-th game [this is (34)–(35)], and then use them to predict the results y _t ; we repeat it for all t ∈ T , summing the scores obtained. This is the well-known leave-one-out (LOO) cross-validation strategy (Hastie et al., 2009, Sec. 2.9), (Duda et al., 2001, Ch. 9.6.2): no data is discarded when calculating metrics (32)–(33) and this comes with the price of having to find θ ˆ ∖ t for all t ∈ T . To reduce the computational load, we opt here for the approximate leave-one-out (ALO) cross-validation (Rad & Maleki, 2020) based on the local quadratic approximation of the optimization function defined for all data. Details are given in Appendix B.

Although both the average log-score (32) and the accuracy (33) can now be optimized with respect to α, κ, η, and/or ξ _c , we only optimize the log-score whose optimal value is denoted as LS _opt ; the resulting accuracy, ACC will also be shown. 13 . It is, of course, possible to optimize the log-score with respect to any subset of parameters while keeping the rest constant;

we will do to this to determine how useful it is to optimize only some parameters.

Again, we use alternate minimization similar to the one shown in (11)–(12): LS was minimized with respect to one parameter at a time: α, κ, η, or ξ _c , until no improvement was observed. This simple strategy led to the minimum LS _opt which turned out to be independent of various starting points we used and, although we cannot prove the solution to be global, in all our observations the log-score functions seemed to be unimodal.

Optimized α, κ, η, ξ _c are shown in Table 4 and indicate that

The data does not provide evidence for using category-dependent weights ξ _c . In fact, the results obtained using the FIFA weights ξ _c are worse than those obtained using constant weights ξ _c  = 1 (i.e., essentially ignoring the possibility of weighting). Although it may be argued that the results are affected by a small number of games in some categories (such as a World Cup), it is very unlikely that observing more games will speak in favor of variable weights and almost surely not in favor of the highly disproportionate weights used in the FIFA algorithm.

Note that the optimal weights ξ₁ (Friendlies within the IMC) and the weights ξ₂ (Group phase of Nations Leagues) are smaller than those of the regular Friendlies. This result stands in contrast with the FIFA algorithm which doubles the weight ξ₁ of the Friendlies played in the IMC and triples the weight of ξ₂. But, of course, we should note a shallow minimum of the objective function which attains the same values LS _opt  = 0.942 (for η = 0 and κ = 2) and LS _opt  = 0.856 (η = 0.3 and κ = 0.9) whether we use the equal ξ _c  = 1 or the optimized ξ _c .

Here, it is interesting to compare the parameters we found by optimization with the simplified formulas proposed in Szczecinski and Djebbi (2020, [Sec. 3.2]) η = log 10 f H f A(36) κ = f D f H f A ≈ 2 f D 1 - f D .(37) where f y , y ∈ Y are empirical frequencies of outcomes. We can consider separately the games played on the neutral venues; the frequencies are obtained from the data, f D neut . = 0.24 , and those played on home venues as f A hfa = 0.26 , f D hfa = 0.23 , f H hfa = 0.51 , which yields κ hfa = 0.61 η hfa = 0.29(38) κ neut . = 0.63(39)

The parameter η^hfa predicted by (36) is practically equal to the one obtained by optimization. And while the parameters κ^hfa and κ^neut. are slightly different from the one predicted by (37), using them in the rating, we obtained LS _opt  = 0.864 (see last line in Table 4), which is still notably better than using the conventional FIFA rating. This is interesting because finding the parameters η and κ from the frequencies of the games not only avoids optimization, but also provides a simple-to-verify origin of the values used in the algorithm.

4.1 MOV via weighting

For context, in Table 5 we show the number of games depending on the value of the MOV variable d. While, in principle, it is possible to use directly d, it is customary to consider their absolute value, |d|.

The Elo/FIFA algorithms (27) can be easily modified as follows, to take into account the MOV variable:

K c , d = K ξ c ζ d ,(40) where, as before, K is the common step, ξ _c is the weight associated with the game-category c, and ζ _d is the function of the MOV-variable d.

Integrating the MOV-weight into the online rating defined in (27) (by replacing, therein Kξ _{c t} with Kξ _{c t} ζ _{v t} ) yields the Davidson-MOV algorithm.

For example, (eloratings.net, 2020) uses

ζ d = { 1 | d | ≤ 1 1.5 | d | = 2 1.75 + 0.125 ( | d | - 3 ) | d | ≥ 3 .(41) Similar propositions can be found in Hvattum and Arntzen (2010) (in the context of association football), in Kovalchik (2020) (to rate tennis players), or in Silver (2014) (for the rating of teams in American football).

To elucidate how useful such heuristics are, we note that the problem is very similar to the importance weighting we analyzed before; the difference lies in the fact that the weighting now depends on the product ξ _c ζ _d . Therefore, we may reuse our optimization strategy to find the optimal weights for games with different values of |d|.

To this end, we discretize |d| into V + 1 MOV-categories, v = 0, …, V and we use a very simple mapping v = |d| for v < V and v = V ⇔ |d| ≥ V. For example, with V = 2, ζ₀ weights the draws (|d|=0), ζ₁ weights the games with one goal difference (|d|=1) and ζ₂ weights the games with more than one goal difference (|d|≥2).

Breaking with the predefined functional relationship as shown in (41) we are more general than the latter, e.g., treating the cases |d|=0 and |d|=1 separately. This makes sense since these events are not only the most frequent ones (covering, respectively, 23% and 36% of the total; see Table 5), but also correspond to the events of draw and win/loss treated differently by the algorithm.

On the other hand, we are also less general due to the merging of events |d| ≥ V, although this effect will decrease with V, simply because there will be very few observations, as may be understood from Table 5. For example, with V = 4, the weighting ζ₄ will be the same for the events with |d|=4 and |d|>4 but the latter make only about 5% of the total.

We again consider the game categories defined in Table 1 and thus we now solve the following problem:

θ ˆ = argmin θ ∑ t ∈ T ξ c t ζ v t ℓ ( z t / s ; y t ) + α 2 s 2 ∥ θ ∥ 2 ,(42) where v _t is the index of the MOV variable d _t . Again, to remove ambiguity of the solution, we set ξ₀ = 1 and ζ₀ = 1.

Parameters ξ _c , ζ _v , η, κ, and α will be optimized again using the ALO approach we described in Section 3.2, that is, by minimizing the log-score criterion (32) using an alternate optimization similar to that defined in (11)–(12). The results shown in Table 6 allow us to conclude that:

The weighting of the MOV-categories is more beneficial than the weighting of the game-categories: by optimizing the MOV-weights ζ _v (and keeping ξ _c  = 1) yields LS _opt  = 0.937 while the optimization of ξ _c (keeping ζ _v  = 1) yields LS _opt  = 0.948 (see Table 4).

The MOV modelling consists in defining a formal relationship between the skills θ _t and the observed MOV variable d _t , where a simple approach proposed in Karlis and Ntzoufras (2008) relies on a modeling of the goal difference using the Skellam distribution Pr { d t = d | θ t } = L ( z t ; d )(43) = e - ( μ h , t + μ a , t ) ( μ h , t μ a , t ) d / 2 I | d | ( 2 μ h , t μ a , t ) ,(44) where I _v  (·) is the modified Bessel function of order v and μ_h,t and μ_a,t are means of the Poisson variables modeling the home- and away- goals. The latter are functions of the skills difference z _t , (Karlis & Ntzoufras, 2008, [Sec. 2.2])

μ h , t = e c + z t + b t η , μ a , t = e c - z t - b t η ,(45) where c is a constant and, as before, η is the HFA coefficient. 15

The model (43) is a particular case of a more general form shown in Karlis and Ntzoufras (2008), which models the offensive and the defensive skills. Here, however, we are interested in rating and thus one skill per team should be used. As noted in Ley et al. (2019) and in Lasek and Gagolewski (2020) this offers a sufficient prediction capacity avoiding the problem of overparameterization due to the doubled number of skills.

Using (45) in (43), the following log-likelihood is obtained: ℓ ( z t ; d t ) = - log L ( z t ; d t )(46) = ( μ h , t + μ a , t ) - d t ( z t + b t η ) - 2 e c - log I ˜ | d t | ( 2 e c )(47) where, for numerical stability, it is convenient to use an exponentially modified form of the Bessel function, I ˜ v ( u ) = I v ( u ) e - u , available in many computation packages.

The derivative of (46) is given by g ( z ; d ) = d d z ℓ ( z ; d ) = - ( d - F ¯ ( z ) ) ,(48) F ¯ ( z ) = μ h - μ a = e c ( e z + b η - e - z - b η ) .(49)

The batch rating then consists in solving the following problem:

θ ˆ = argmin θ ∑ t ∈ T ℓ ( z t / s ; d t ) + α 2 s 2 ∥ θ ∥ 2(50) and the SG implementation of the ML principle will produce the Skellam algorithm

θ t + 1 ← θ t + K x t ( d t - F ¯ ( z t / s ) ) ,(51) which is again written in a form similar to the FIFA rating algorithm, where the goal difference d _t plays the role of the “score”, and F ¯ ( z t / s ) = 𝔼 [ d t | z t ] is the expected score. The Skellam algorithm (51) can also be obtained by applying the Poisson model to the goals scored by each of the teams (Lasek & Gagolewski, 2020).

To calculate the log-score, we have to calculate the probabilities m ^ls  (z _t  ; A) = - log Pr{d _t  < 0} (away-win) and m ^ls  (z _t  ; H) = Pr{d _t  > 0} (home-win). Since the closed-form formulas do not exist, we use truncated sums m ls ( z ; A ) = - log ∑ d = - D - 1 L ( z ; d ) ,(52) m ls ( z ; D ) = - log L ( z ; 0 ) ,(53) m ls ( z ; H ) = - log ∑ d = 1 D L ( z ; d ) ,(54) where we applied D = 50 which we empirically verified to satisfy 1 - ∑ d = - D D L ( z ; d ) < 10 - 4 .

The results shown in Table 7 indicate that, with this very simple approach (only two parameters of the model which must be optimized), we are able to improve over the MOV-weighting strategy and this should be attributed to the use of a formal skills-outcome model. The price to pay for the improvement lies in abandoning the legacy of the Elo algorithm.

Moreover, possible implementation issues may arise since the expected score (49) is theoretically unbounded. Thus, whether the improvement of the log-score from LS = 0.849 (in the MOV-weighting, see Table 6) to LS = 0.843 in the Skellam MOV model is worth the change and the implementation risks is at least debatable.

5.1 Scale adjustment

The scale is obviously irrelevant in batch optimization, and the on-line update can also be written in a scale-invariant manner by dividing (20) by s: θ t + 1 ′ ← θ t ′ - K ′ ξ c t x t g ( z t ′ ; y t )(55) z t ′ = z t / s(56) θ t ′ = θ t / s(57) K ′ = K / s ;(58) in other words, we will obtain the same results θ t ′ as long as we use the same initial θ 0 ′ and the same step K′. In particular, with an all-zero initialization of the skills, i.e., θ _t  = 0 and using K = sK′ we will obtain the same scaled results θ t = s θ t ′ .

However, in the FIFA ranking, a non-zero initialization θ ₀ was determined in advance (see footnote 3) so θ 0 ′ is not scale-invariant. Thus, given the initialization at hand, the question is how to determine the scale. We do not know any clear answer, but an insight into finding the useful scale value may be gained assuming that the initialization corresponds to the “optimal” solution, e.g., θ ˆ obtained in batch optimization with a given scale s₀.

It is easy to see that using s > s₀ will force the algorithm to change significantly θ _t (attainable with large values of the adaptation step, K); the same will happen for s < s₀ because the optimal estimates θ _t will have to be scaled down.

Since scaling the skills up/down changes their empirical moments, we suggest choosing the scale s in a moment-preserving manner. To this end, we define the empirical standard deviation of the skills

σ t = ∥ θ ˆ t - θ ˆ ¯ t ∥ 2 / M(59) where θ ˆ ¯ t = ( ∑ m = 1 M θ ˆ t , m ) / M is the empirical mean and postulate that, at the initialization and at the final step, we have σ₀ ≈ σ _T .

In fact, the initialization used by FIFA yields σ₀ = 220 and, after running the original FIFA algorithm, we obtain σ _T  = 252.

Changing the scale s, we will obtain different σ _T so the idea is to run the algorithms for different values of the scale s (e.g., for multiples of 50) and to choose the one that produces a standard deviation σ _T  ≈ σ₀. In practice we might do it using historical data before the new rating is deployed.

In this manner we found s = 150 to be suitable for the Davidson algorithm: we obtained σ _T  ≈ 220 when κ = 2 and σ _T  ≈ 210 for κ = 1.

This indicates that the scale s = 600 was too large for the FIFA rating. This can be seen by comparing the result of the FIFA rating with ξ _c  = 1 (in Table 8a) to the results of the Davidson algorithm (with η = 0 and κ = 2). Both algorithms are essentially the same (although FIFA uses the shootout/knockout rules, which have rather small impact on performance) and the main difference resides in the scale. Since the Davidson algorithm (η = 0, κ = 2) with the scale s = 150 is equivalent to the FIFA algorithm with the scale s = 300, this latter scale value would ensure a better performance of the FIFA rating. However, this effect appears only due to the limited observation time we have at our disposal and will vanish after a sufficiently large number of games.

Similarly, for the Davidson-MOV algorithm, using s = 200, and for different values of V we obtained σ _T  ≈ 220, while using the scale s = 300 in the Skellam algorithms yields σ _T  = 225.

To evaluate the SG algorithms, we used the same methodology we applied to make a preliminary evaluation of the FIFA rating in Section 2.1. That is, we used the first (approximate) half of the observation period for initialization and the second half is used to calculate the performance metrics. The difference from Section 2.1 is that now we use the log-score and the accuracy metrics LS ′ = 1 T - T ′ ∑ t = T ′ + 1 T m ls ( z t , y t )(60) ACC ′ = 1 T - T ′ ∑ t = T ′ + 1 T m acc ( z t , y t ) ,(61) where T′ is defined after (9).

We consider the original and modified FIFA algorithm (Table 8a), the Davidson algorithm (Table 8b), the Davidson-MOV algorithm (Table 8c), and the Skellam algorithm (Table 8d).

In all cases, but in the original FIFA algorithm, we ignore the game-category weighting (i.e., we use ξ _c  ≡ 1) because, as we have already shown, its effect is negligible. This is clearly shown in the first row of Table 8a where we see that, using the FIFA weighting, we obtain worse results than when the weighting is ignored. This is essentially the same result as the one we have shown in Table 2 but we repeat it here to show the log-score metric which we could not calculate without first introducing the Davidson model underlying the FIFA algorithm.

In Table 8a we also show the log-score LS_∖so&ko obtained applying the FIFA algorithm but removing the shootout and knockout rules. We observe that the knockout rule improves the prediction in terms of the log-score which confirms our preliminary evaluation in Section 2.2 based on the MSE. Nevertheless, the improvement is minor, and a much more important decrease of the log-score may be obtained by changing the model as indicated by the remaining results. In particular

The most notable improvements are due to, in similar measures, two elements: the introduction of the HFA coefficient η and the explicit use of the Davidson model (and thus the optimization of the coefficient κ).

We present in Table 9 the rating obtained for the top teams through new rating algorithms. Of course, due to the different scales that we used, the skills obtained with different algorithms cannot be compared directly.

We emphasize that the quality of these rankings (i.e., ordered skills) cannot be assessed because there is no reference order of the teams to which the shown rankings can be compared. The only tool we have to assess their validity is to calculate the performance criteria as we did in Table 8 using the log-score.

Noting that even rather mild differences between the Davidson and the Davidson-MOV algorithms alter the final order/ranking, Table 9 should be treated as a cautionary illustration that the ranking/order of the team can be very easily changed by relatively benign modifications of the rating algorithm. With that caveat, the different algorithms based on different models consistently put the same group at the top of the list. In fact, the algorithms are rather consensual regarding the current (as of March 31, 2002) official top team, BRA which, or remains on the top of the list, or has skills within fraction of percentage of top team’s rating. On the other hand, BEL’s second position in the official ranking (see Table 3) is much more questionable. While the second spot is preserved with the optimized FIFA-like algorithm (the first column of Table 9 where we use K = 55 and ξ _c  ≡ 1, but the knockout and shootout rules are kept), the new algorithms consistently demote BEL to the fifth and lower position.

6.1 Recommendations

Given the analysis and the observations we made, if the FIFA rating is to be changed, the following steps are recommended:

Add the home-field advantage (HFA) parameter to the model because playing at the home venue is a strong predictor of victory. This well-known fact is already exploited in Women’s FIFA ranking, and such a modification is most likely the simplest and the least debatable element. In our opinion, it is surprising that the current rating adopted in 2018 does not include the HFA. The HFA can be obtained through optimization, or it can be calculated using the simple formula (36).

6.2 Further work

The analysis we carried out in this work should not be considered exhaustive by any means and was meant (i) to provide an understanding of the current FIFA rating and (ii) to propose the simplest and yet meaningful modifications of the current algorithm.

Our recommendations regarding further work on the improvement of the rating are the following:

Beside the simple weighting strategies we analyzed here, to deal with the MOV we should consider alternative solutions similar to those already considered in Women’s teams FIFA ranking. Again, the latter should be studied, e.g., using the methodology we used in this work and basing the results on a formal probabilistic model.