Testing and ranking on round-robin design for data sport analytics with application to basketball

Abstract

By modelling results of sport matches as a set of paired fixed effect linear models, the goal of the present article is showing that traditional scoring outputs can be used to do inference on parameters related to the net relative strength or weakness of teams within a league. As hypothesis testing method, we propose either a normal-based and a non-parametric permutation-based approach. As an extension to round-robin of the ranking methodology recently proposed by Arboretti Giancristofaro et al. (2014) and Corain et al. (2016), results of pairwise testing are then exploited to provide a ranking of teams within a league. Through an extensive Monte Carlo simulation study, we investigated the properties of the proposed testing and ranking methodology where we proved its validity under different random distributions. In its simplest univariate version, the proposed methodology allows us to infer on the teams average net scoring within a league, while in its more intriguing multivariate layout it is suitable for looking for any team-related global dominance using a wide set of performance indicators. Finally, by using traditional basketball box scores, we present an application to the Italian Basket League.

Keywords

permutation tests union-intersection principle nonparametric combination sport performances

1 Introduction and motivation

A round-robin tournament refers to a set of sport competitions in which each competitor challenges, once at time, all other competitors. Most sports leagues play a double round-robin tournament where teams meet twice, but single, triple and quadruple round-robin tournaments do also occur (Rasmussen and Trick, 2008). Besides sports, round-robin design is intensively used by psychologists to investigate on social relations models and social networks issues (Kenny et al., 2006). With the goal of comparing several foods, beverages, etc., the round-robin design is also considered in the field of sensory sciences by exploiting sensorial evaluations provided by a set of panelists towards all possible pairs of investigated items (Meilgaard et al., 2006).

Statistical models on round-robin design are distinctive by being suitable of handling pairs of dependent responses, in other words they are specifically designed to deal with the analysis of paired comparison data (Cattelan, 2012). By referring to mixed effect models, psychometricians deeply investigated social networks using the variance components approach where subjects within a network were handled as a random factor (Nestler, 2016). It is worth noting that variance components approach is not actually suitable for data sport analytics where team effects must act as fixed parameters in order to find out possible significant differences among competitors. In spite of the great assonance between round-robin design and the running of team- and individual-sports tournaments, the interest around inference on parameters of round-robin design was not deeply investigated in data sport analytics literature.

Since scores of the sport teams fluctuate over time and are obviously partially affected by some natural random variability, suitable modelling and inferential-based methods on performance indicators are demanded either for descriptive and explicative purposes. After modelling results of sport matches as a set of paired fixed effect linear models, based on the concept of multivariate stochastic dominance, the aim of this work is to propose a reference framework for modelling, testing and ranking on multivariate scoring sport data.

The present article is organized as follows: Section 2 provides a literature review on modelling for data sport outcomes; Section 3 outlines the adopted modelling for round-robin design and provides the description of two proposed procedures aimed at doing inference on a set of multivariate parameters of interest. In Section 4, the main results of a simulation study are shown and discussed, while Section 5 presents an application to basketball and finally Section 6 deals with conclusions, final remarks and some suggestions for future perspectives. In the Appendix, some additional details on the real case study application are provided.

2 Modelling for data Sport Outcomes

There is a considerable existing literature on modelling the scores of the two opposing teams (or players) for data sport analytics purposes. Harville (2003) proposed a linear model for the expected difference in score in each game of basketball or football as a difference in team effects plus or minus a home court/field advantage. In order to fit the related linear model, the author proposed a modified least squares estimation method and used the teams estimates either to rank the team within a league (for use in the selection or seeding procedures) and to effectively predict the outcomes of postseason games. Such a ranking system was considered also by Stefani (1980), Stern (1992), Stern (1995) and Harville and Smith (1994). An alternative approach suggests trying to separately model the scores of two opponents. Karlis and Ntzoufras (2003) proposed applying a bivariate Poisson distribution with a dependence parameter between the number of goals scored by the two teams and then accommodated the model in order to inflate the probabilities in case of draw. Adam (2016) tried to extend the bivariate Poisson approach by mean of a generalized linear model where the scores were modelled as the joint probability of a Poisson distribution, representing the total number of goals, and a binomial distribution, representing the goals of one team given that total number of goals.

By mainly focusing on the probability of winning or losing the sports matches, a second relevant piece of research in data sport outcome modelling has been inspired by the Bradley–Terry model (Bradley and Terry, 1952). As pointed out by Chan (2011), linear models for paired comparisons, the Bradley–Terry model and the Thurstone–Mosteller model in particular, have been widely used in sports for ranking and rating purposes. Cattelan et al. (2013) introduced a dynamic extension of the Bradley–Terry model for paired comparison data to model the outcomes of sporting contests, allowing for time varying abilities. By combining the Bradley–Terry model and a non-linear model that utilizes an exponential distribution to describe longitudinal changes in scale values, Usami (2017) proposed a procedure for Bayesian longitudinal paired comparison data analysis to rank sumo players. An alternative approach to model the outcomes of the matches in terms of win–draw–loss has been proposed by Goddard and Asimakopoulos (2004) via an ordered probit model. By specifying the probability of the match outcome as a function of the difference of abilities of the two teams, an ordered probit model was adopted also by Koning (2000).

Finally, thanks to the availability of more insightful sports data, also provided by emerging player tracking systems enabling a richer quantitative characterization of performance, a third research area for data sport outcome modelling has been developed around the concepts of stochastic processes and machine learning techniques. Franks et al. (2015) attempted to combine spatial and spatio-temporal processes, matrix factorization techniques and hierarchical regression models with player tracking data to model and advance the state of defensive analytics in basketball. Cervone et al. (2016) proposed a framework for using optical player tracking data to estimate, in real time, the expected number of points obtained by the end of a possession (EPV) that was derived from a stochastic process model for the evolution of a basketball possession. They model this process at multiple levels of resolution, differentiating between continuous, infinitesimal movements of players and discrete events such as shot attempts and turnovers. Vracar et al. (2016) presented a methodology for modelling and forecasting basketball match between two distinct teams as a sequence of team-level play-by-play in-game events where authors assumed a Markov property and model state transitions with a logistic regression model. By exploiting play-by-play basketball logs, Chen and Fan (2016) proposed using a functional data analysis approach to model the observed score difference viewed as the realization of the latent continuous intensity process. This approach has several nice features such as the ability of defining and numerically characterizing momentum in basketball games.

3 Testing and ranking on round-robin design for data sport analytics

Let us consider a tournament designed as a round-robin championship where a set of competitors are supposed to pairwise challenge each with one another. More formally, suppose that a round-robin design involves a sports tournament with $C$ competitors where competitor $j$ and $h$ meet $n_{jh}$ times, $j \neq h$ . If some $n_{jh}$ are zero, the design is said to be incomplete. If all $n_{jh}$ are equal, we say that the design is balanced. For unequal $n_{jh}$ , the design is unbalanced. Even if the approach we are going to present is flexible enough to handle unbalanced data, we focus our attention on the double round-robin design which is the most common case for team-based sports tournament, such as soccer, football, basketball and many others, Formally, we consider the case $n_{jh} = 2$ , $\forall j, h = 1, \dots, C, j \neq h$ (so-called double round-robin with one at home and one away rounds).

The competition between two opponents produces a pair of outcomes, in form of scalars but most often vectors, one for each competitor. Since generally a tournament takes place during a given time span, outcomes for each competitor can be viewed also as longitudinal multivariate observations of performance indicators.

By referring to the literature of social relations models for round-robin design often known as dyadic data (Kenny et al., 2006), when opponents $j$ and $h$ meet on the $i$ th occasion, we obtain a pair of observations $y_{ijh}$ and $y_{ihj}$ as a realization of a $p$ -variate random variables. Here $y_{ijh}$ represents the response of team $j$ as an actor towards opponent $h$ as a partner on the $i$ th occasion; and in $y_{ihj}$ the roles are reversed.

Without loss of generality, let us assume that for each $y_{ijhk}$ univariate component, $k = 1, \dots, p$ , the rule ‘the higher the better’ takes place, meaning that large values of $y$ correspond to a ‘better’ performance. In the simplest univariate version, $y_{ijh} = {pts}_{ijh}$ , that is, it might represent the scoring made by the $j$ th team facing the $h$ th opponent on the $i$ th occasion. Note that, ${pts}_{ihj}$ can be also viewed as the scoring granted by the $j$ th opponent facing the $h$ th on the same $i$ th occasion.

In order to model the performances of the two $j$ th and $h$ th competitors facing on the $i$ th occasion, we propose formalizing the round-robin design pair of $p$ -variate outcomes from a team-based sport championship as

\begin{matrix} y_{ijh} = μ + δ_{ij} + τ_{j} + β_{h} + (δ τ)_{ij} + (δ β)_{ih} + γ x_{ijh} + ε_{ijh}, \\ y_{ihj} = μ + δ_{ih} + τ_{h} + β_{j} + (δ τ)_{ih} + (δ β)_{ij} + γ x_{ihj} + ε_{ihj}, \\ i = 1, 2, \forall j, h = 1, \dots, C, j \neq h, \end{matrix}

(1)

where $μ$ is the league-related global mean across all competitors, $δ$ is the home-court effects (Harville and Smith, 1994; Harville, 2003), that is, $δ_{ij} = + δ$ and $δ_{ij} = - δ$ when on the $i$ th, occasion the $j$ th and $h$ th teams play respectively at home and away (the signs of $δ$ are reversed on the second occasion when home and away are inverted), $τ_{j}$ and $β_{j}$ are the active (attack) and passive (defense) effects, $(δ τ)_{ij}$ and $(δ β)_{ij}$ are the interactions between the home–away and the active and passive effects, and finally $ε$ are $p$ -variate $IID (0; Σ)$ random errors. Moreover, $γ$ are coefficients intended to model any possible covariate effect that in the spirit of (1) should not be referred to any related-performance information but instead to different issues such as tactic-related variable (e.g., the number of attempted goals) or to players state of fitness. Note that there is no interaction term between $τ_{j}$ and $β_{h}$ . This is because in the case of double round-robin design (just two time occasions), this additional interaction effect is confused with the main effects we set up in the model.

In order to focus our attention on a new parametrization highlighting the net effect due to attack plus defense, let us consider the net performance ${Δ y}_{ihj}$ of competitor $j$ towards opponent $h$ on the $i$ th occasion, that is,

\begin{matrix} {Δ y}_{ihj} = 2 δ + [τ_{j} + (δ τ)_{ij} - β_{j} - (δ β)_{ij}] - [τ_{h} + (δ τ)_{ih} - β_{h} - (δ β)_{ih}] + \\ + γ (x_{ijh} - x_{ihj}) + (ε_{ijh} - ε_{ihj}) = 2 δ + α_{ij} - α_{ih} + γ Δ x_{ijh} + u_{ijh}, \end{matrix}

(2)

where

α_{ij}

represents the total abilit, in the form of ‘attack’ + ‘defense’, of team

j

on the

i

th occasion and

u

are

p

-variate

IID (0; \tilde{Σ})

random errors.

Note that expressions (1) and (2) are actually fixed effect multivariate multi-way ANOVA model that, in each univariate component and using an R-like coding, can be also expressed as

Y ∼ Home+Team+Opponent+Home*Team+Home*Opponent+Covar_1+…+Covar_q

and

ΔY∼ Home+Team+Opponent+Home*Team+Home*Opponent+ΔCovar_1+…+ΔCovar_q.

When the univariate response represents just the points made and granted, $Δ y_{ihj}$ is nothing but the so-called $margin$ of basketball boxscore whose sign defines the winner and the loser. In this connection, by assuming independently and normally distributed random errors, it is worth noting that model (2) is actually equivalent, apart from the inclusion of covariates and the interaction between team and home-court effect, to the linear model proposed by Harville (2003).

It is worth noting also that the inference on the net relative strength or weakness of each teams within a league is nothing but that testing on the $p$ -variate $α$ s parameters whose related set of hypotheses of interest may by expressed either in terms of significant testing, as in (3), and in terms of pairwise comparisons, as in (4)

\{\begin{matrix} H_{0 j} : ⋂_{i = 1}^{2} α_{ij} = 0 \equiv ⋂_{i = 1}^{2} ⋂_{k = 1}^{p} α_{ijk} = 0 \equiv ⋂_{i = 1}^{2} ⋂_{k = 1}^{p} H_{0 ijk} \\ H_{1 j} : ⋃_{i = 1}^{2} α_{ij} \neq 0 \equiv ⋃_{i = 1}^{2} ⋃_{k = 1}^{p} α_{ijk} \neq 0 \equiv ⋃_{i = 1}^{2} [H_{1 ij}^{+} ⋃ H_{1 ij}^{-}] \equiv \\ \equiv ⋃_{i = 1}^{2} [(α_{ij} < 0) ⋃ (α_{ij} > 0)] \equiv ⋃_{i = 1}^{2} ⋃_{k = 1}^{p} [(α_{ijk} < 0) ⋃ (α_{ijk} > 0)] \equiv \\ \equiv ⋃_{i = 1}^{2} ⋃_{k = 1}^{p} [H_{1 ijk}^{+} ⋃ H_{1 ijk}^{-}] \end{matrix}

(3)

j = 1, \dots, C

\{\begin{matrix} H_{0 (jh)} : ⋂_{i = 1}^{2} α_{ij} = α_{ih} \equiv ⋂_{i = 1}^{2} ⋂_{k = 1}^{p} α_{ijk} = α_{ihk} \equiv ⋂_{i = 1}^{2} ⋂_{k = 1}^{p} H_{0 i (jh) k} \\ H_{1 (jh)} : ⋃_{i = 1}^{2} α_{ij} \neq α_{ih} \equiv ⋃_{i = 1}^{2} ⋃_{k = 1}^{p} α_{ijk} \neq α_{ihk} \equiv ⋃_{i = 1}^{2} [H_{1 i (jh)}^{+} ⋃ H_{1 i (jh)}^{-}] \equiv \\ \equiv ⋃_{i = 1}^{2} [(α_{ij} < α_{ih}) ⋃ (α_{ij} > α_{ih})] \equiv ⋃_{i = 1}^{2} ⋃_{k = 1}^{p} [(α_{ijk} < α_{ihk}) ⋃ (α_{ijk} > α_{ihk})] \equiv \\ \equiv ⋃_{i = 1}^{2} ⋃_{k = 1}^{p} [H_{1 i (jh) k}^{+} ⋃ H_{1 i (jh) k}^{-}] \end{matrix}

(4)

\forall j, h = 1, \dots, C, j \neq h

We expressed the multivariate null and alternative hypotheses using the Union-Intersection Roys principle (Roy, 1953; Pesarin and Salmaso, 2010), where we take into account for the possible interaction between team and home–away effect (by using the $i$ index) and by decomposing the multivariate hypothesis into $k$ univariate hypotheses ( $k$ index). This kind of representation of the global hypotheses is actually very useful when applying the combination methodology on univariate (and not independent) $p$ -values to eventually combine them into multivariate tests (Corain and Salmaso, 2004, 2015).

Moreover, the alternative hypothesis was also broke down into two separated hypotheses, denoted by an upper-right ‘ $-$ ’ and ‘+’ with the goal of emphasizing in which direction the possible difference actually takes place (Arboretti Giancristofaro et al., 2014; Corain et al., 2016; Corain and Salmaso, 2007b). Note that by using the Union-Intersection approach, we emphasize that under the alternative hypothesis just one of the possible one-sided directions must be true.

Anyway, although this should rarely happen in real problems, since it could happen that for some univariate performances a team is either above and at the same time below the mean of the league, it is worth noting that $H_{1 j}^{-}$ and $H_{1 j}^{+}$ (and $H_{1 jh}^{-}$ and $H_{1 jh}^{+}$ as well) may by $jointly$ be true.

By exploiting the multivariate one-sided alternatives in expression (4), we may provide a league's ranking using the ranking methodology proposed by Arboretti Giancristofaro et al. (2014). In fact, by suitable combining information from directional multivariate $p$ -values, the underline possible latent ordering among ï' ¡s parameters can be properly estimated. In a nutshell, the rationale behind the ranking within a multivariate setting is the following: if not all $H_{0 jh}$ in (4) are true, it must exist an ordering [1],[2],…,[C] among $α$ s, such that

\begin{matrix} α_{[1]} \leq α_{[2]} \leq \dots \leq α_{[C]}, \end{matrix}

where ‘

\leq

’ should be intended as ‘

<

’ when there exists at least one univariate

α

for which the strict inequality holds and, at the same time, there is not any univariate

α

for which the opposite strict inequality holds. When the last condition is not true, ‘

\leq

’ is intended as ‘=’, meaning that the two

α

s are tied in the ranking. Note that within a multivariate setting, we state that two multivariate parameters are tied not only when all univariate

α

are equal but also when that

H_{1 jhk}^{-}

and

H_{1 jhk}^{+}

are

jointly

true. For more details on the ranking methodology within a multivariate setting, we refer the reader to Arboretti Giancristofaro et al. (2014) and Corain et al. (2016).

Let $p_{ijk}^{-}$ and $p_{ijk}^{+}$ , and equivalently $p_{i (jh) k}^{-}$ and $p_{i (jh) k}^{+}$ , be the univariate $p$ -values useful to infer on the possible equality versus directional alternative respectively in expressions (3) or (4). Depending on the assumptions on the $p$ -variate random errors, we propose two set of suitable univariate test statistics for testing

$H_{0 ijk}$ vs. $H_{1 ijk}^{-}$ or $H_{1 ijk}^{+}$ in (3),

$H_{0 i (jh) k}$ vs. $H_{1 i (jh) k}^{-}$ or $H_{1 i (jh) k}^{+}$ in (4),

either using (a) a normal-based and a (b) non-parametric permutation-based approach, that is,

(a) Assuming the normality of random errors in (1), that is, $ε$ $\sim$ $IIN (0; Σ)$ ,

(1.a). $t_{ijk} = \frac{{\hat{α}}_{ijk}}{se ({\hat{α}}_{ijk})} \sim t_{df (error)}$ ,

(2.a). $t_{i (jh) k} = \frac{{\hat{α}}_{ijk} - {\hat{α}}_{ihk}}{se ({\hat{α}}_{ijk}) + se ({\hat{α}}_{ihk}) - 2 cov ({\hat{α}}_{ijk}, {\hat{α}}_{ihk})} \sim t_{df (error) - 2}$ ,

where $df (error)$ are the errors degrees of freedom in the linear model (2);

(b) Without assuming any specific random distribution, within a non-parametric framework,

(1.a). $T_{ijk} = sign (Δ {\tilde{y}}_{ijk})$ ,

(2.b). $T_{i (jh) k} = \bar{Δ} {\tilde{y}}_{ijck} - \bar{Δ} {\tilde{y}}_{ihck}$ ,

where ${\tilde{y}}_{ijk}$ are the residuals from the covariate effects calculated by model (2) and $sign (\cdot)$ is the permutation-based sign-test for symmetry (Pesarin and Salmaso, 2010; Bonnini et al., 2014); $T_{i (jh) k}$ is the difference between sample means of suitable kind of paired $j$ and $h$ residuals (for more details see expression (5)).

As about the multivariate $p$ -values $p_{ij \cdot}^{-}$ and $p_{ij \cdot}^{+}$ , and equivalently $p_{i (jh) \cdot}^{-}$ and $p_{i (jh) \cdot}^{+}$ , suitable for testing $H_{0 j}$ vs. $H_{1 j}^{-}$ or $H_{1 j}^{+}$ (and $H_{0 (jh)}$ vs. $H_{1 (jh)}^{-}$ or $H_{1 (jh)}^{+}$ as well), we propose a combination procedure which, in the light of the random error assumptions, can be

(1). The Kosts method for combining dependent $p$ -values Poole et al. (2015), implemented in R by Poole (2017); univariate normal-based p-values for $t_{i (jh) k}$ statistic was calculated by using the R package provided by Bretz et al. (2010);

(2). One suitable non-parametric combining function, such as the Fishers and the Tippets combining function (Pesarin and Salmaso, 2010; Bonnini et al., 2014).

Depending on the fact that the covariate effects are truly active or null, the combined multivariate permutation tests can be considered respectively as an approximated or an exact testing solution for problems (3) and (4). The normal-based solution has to be always viewed as approximated and its behaviour under finite samples should be evaluated via simulation study, as well as its robustness against violations of the normality assumption (see the next section).

As a final remark, we highlight that under the non-parametric combination setting, the testing problems (3) and (4) can be actually viewed respectively as a multivariate sign one-sample problem and a two-sample multivariate paired data problem (Pesarin and Salmaso, 2010; Bonnini et al., 2014) with constrained permutations under two-way layout (Corain and Salmaso, 2007a). In fact, in the first case, we seek for an evidence against the symmetric distribution around 0 of all univariate responses, while in the second problem, once the $Δ y$ of two teams are properly matched under the null hypothesis $H_{0 (jh)}$ the exchangeability principle holds and we can apply the permutation strategy in order to find out if there is evidence against the null hypothesis.

More formally, let us consider all the two occasions where both opponents $j$ and $h$ are facing the set of all $(C - 2)$ remaining competitors, and the related ${Δ y}_{ijc}$ and ${Δ y}_{ihc}$ observations, that is,

\begin{matrix} {Δ y}_{ijc} = α_{ij} - α_{ic} + γ Δ x_{ijc} + u_{ijc}, c = 1, \dots, C, j \neq c \neq h, \\ {Δ y}_{ihc} = α_{ih} - α_{ic} + γ Δ x_{ihc} + u_{ihc}, c = 1, \dots, C, h \neq c \neq j . \end{matrix}

(5)

Once we pair and take the difference

{Δ y}_{ijc}

vs.

{Δ y}_{ihc}

at each given fixed third part opponent

c

, it results that

\begin{matrix} Δ y (c)_{jh} = {Δ y}_{{ii}^{'} jc} - {Δ y}_{{ii}^{'} hc} = α_{ij} - α_{i^{'} h} + u_{{ii}^{'} jhc}, \\ i, i^{'} = 1, 2 ​ (i \neq i^{'}), j, h, c = 1, \dots, C, j \neq h \neq c . \end{matrix}

It turns out that there are actually a number of

(C - 2) \cdot 2

observations to infer on

(α_{ij} - α_{i^{'} h})

. Note that

Δ y (c)

is a kind of indirect ‘net’ ability of two opponents when they are facing the same third-part opponent. Note that

(α_{1 j} - α_{1 h}) = (α_{2 j} - α_{2 h})

if and only if both interactions between the home-court and the active and passive effects are null. Under this assumption, the comparison between the pair of opponents should not take into account the relation between time occasion and home-court information.

4 Simulation study

In order to investigate the properties of the proposed testing and ranking methods under finite samples of sizes comparable to real situations in data sport analytics, we performed a Monte Carlo simulation study where we independently simulated the outcomes of a tournament consisting of 112 simulated matches as a result of a double round-robin with $C = 8$ teams. The rationale of the simulation study was focused on investigating how power of either the combined normal-based and the permutation-based significant tests, as in (3), and pairwise comparisons, as in (4), are affected by the type of multivariate random distribution. More specifically, the simulation study considered 1 000 bivariate simulated performances, where variance $σ^{2}$ and correlation $ρ$ of the simulated performances are set up as $σ_{jk}^{2} = 1$ and $ρ_{jhk} = 0.5, : \forall k = 1, 2$ , and $j, h = 1, \dots, C, j \neq h$ .

The simulation study was designed to take into account for six different settings, defined as combinations of the following configurations:

Two possible scenarios of bivariate mean vectors, as graphically displayed in Figure 1; note that each scenario has some competitors that are equal in performance, that is, in the simulated tournament we are jointly either under the null and the alternative hypothesis;

Three type of multivariate distributions for random errors: (a) Normal, (b) Studentst3 (with degree of freedom equal to 3), as an example of an heavy-tailed distribution, (c) g-and-h distributions with g = (0.5,0.5) and h = (0,0), as an example of a moderate skewed distribution (Kowalchuk and Headrick, 2000). Figure 2 provides a graphical representation of each bivariate distribution, along with the simulated performances under scenario 1.

In order to keep our simulation more realistic, we also set up an home-court effect equal to 0.5 and two covariate effect with an average size effect equal to 0.25.

Figure 1:

Bivariate mean alpha values by competitor and scenario

Figure 2:

Simulated errors and simulated team performance from scenario 1 by type of error

Despite the team performance was simulated according to expression (1) with $ε$ component-wise variance set up to the value of 1, it is worth noting that the u random terms of ${Δ y}_{ihj}$ in expression (2), labelled here as D_Errors, has variance larger than 1. More specifically, thanks to the error independence assumption, it turns out that it is equal to 2 for normal errors, while for Students t and g-and-h errors it is respectively equal to 6 and about 3. Figure 2 should provide a graphical picture to easily understand the relative size and shape of the simulated team performances (from scenario 1) under each type of random error.

As far as the specific combination method we used to calculate the combined-multivariate $p$ -values, as mentioned in the previous section, we considered the Kosts method and the Fisher's combining function respectively for the normal-based and the permutation-based tests. Finally, the null permutation distribution was estimated by using $B$ = 2 000 conditional Monte Carlo iterations. The performance of both multivariate testing methods we proposed, that is, the combined normal-based and the combined permutation-based, was evaluated in terms of achieved nominal levels and of power with respect to either significant testing (Figure 4), as in (3), and pairwise comparisons (Figure 5), as in (4). Note that due to the relative large number of team-related alphas ( $C = 8$ ), we selected only a relevant subset of alpha vectors whose reference is presented in Figure 3.

Figure 3:

Legend for rejection rates results in Figures 4 and 5

Note that the solid blue and orange lines represent the rejection rates under the true null hypothesis. This null hypothesis, along with each one of the remaining four selected alternatives, are all tested versus both multivariate one-sided alternatives.

Figure 4:

Significant testing on alpha: Rejection rates test by method and type of multivariate distribution

First of all, we note that under the null hypothesis of equality to 0, both procedures properly respect the nominal levels, while under the alternatives, as far as alpha moves apart from 0, the rejection rates approach faster to 1. The normal-based method looks like somewhat more powerful than the permutation-based procedure, showing also a robust behaviour with respect to the type of random error.

Figure 5:

Pairwise testing on alpha: Rejection rates test by method and type of multivariate distribution

As far as the pairwise testing, we note that under the null hypothesis of pairwise equality both procedures look like a bit biased but, for the most used significant levels, they are able to properly respect the nominal levels. Under the alternatives and as expected, as the distance between pairs of alpha gets larger, the rejection rates approach faster to 1. The permutation-based method looks like somewhat more powerful than the normal-based procedure, and both methods have a robust behaviour with respect to the type of random error. By setting the significant $α$ -level at 5%, Figure 6 provides a power comparison summary that basically confirms all previous remarks. Note also that under heavy-tailed errors, the permutation approach is somewhat more powerful.

Figure 6:

Power comparison, at 5% significant $α$ -level, on significant and pairwise combined tests by testing method and type of multivariate distribution

Finally, it is interesting to investigate how both pairwise testing methods behave when applied to the ranking methodology, that is, when they are used to estimate the true ordering among teams (it is actually reported in Figure 1, for both scenarios). In this regards, for each one simulated tournament we calculated the Spearman correlation index between the estimated and the true ranking: The larger the correlation the better is the ranking method. As benchmark, after assuming that the first bivariate simulated response do represent the match score, we also considered

A kind of simulated Win/Loss W/L criterion, whose ranks, at the end of the simulated tournament, may be viewed as a benchmark of the proposed two ranking methods;

The least squares ranking of $α$ estimates Estim. Alphas (Harville, 2003), that is, the ranking obtained by ordering the teams from first possibly to last in accordance with model (2).

Figure 7:

Ranking performance comparison, at 5% significant $α$ -level, by scenario, ranking method and type of multivariate distribution

As suggested by Figure 7, the comparison among Spearmans rho values by ranking method suggest some interesting hints: Both testing methods (dark blue and red box) appear to be as a reliable tool, either in mean and in variance, to estimate the true underlying ranking among teams. As expected, note that the W/L criterion (yellow box), which is a kind of descriptive in nature ranking approach, is relatively much less reliable, especially under scenario 1 when some teams are supposed to perform equally. Under scenario 1, the ranking results provided by the least squares ï' ¡ estimates method (orange box) are slightly more reliable than the W/L criterion but certainly not better than all two pairwise testing rankings. Note also that the performance of the last method is getting better under scenario 2. The normal-based testing approach show a slightly better performance than the permutation-based one. This kind of bit surprisingly result is probably explained by the intrinsic ability of t-test of getting significant values much more close to zero: since the ranking method requires a suitable adjustment correction, getting closer to zero do provide to the normal-based approach an overall larger power.

5 Application to basketball

In order to highlight the practical use of the proposed methodology, we present its application to a real case study in basketball, namely to the Italian Basket A League (first division, www.legabasket.it). We used box scores data from regular season 2016–2017 where a set of 16 teams challenged within a typical double round-robin design, so that dataset consists into 480 rows (2 outcomes from each of 240 matches, $480 = 16 \times 15 \times 2$ ). As response variables, we used three performances, that is, PTS (scored points), OE (Offensive Efficiency) and Floor.Per (Floor Percentage). Further classical basketball performance indicators (Oliver, 2004), such as Off.Rtg (Offensive Rating) and eFG.Per (Effective Field Goal Percentage), were discharged because too much correlated with the scored points (i.e., Pearson correlation(PTS,Off.Rtg): 0.87). As tactical-related covariates, we considered FTA (free throw attempts; it refers to the offensive aggressiveness and so it must be included in the indicators of attacks strategy), FGA and 3PA (field goal and 3 points attempts, they refer to the attack strategy). It is worth noting that the choice of appropriate covariates is a relevant problem as it has a potential large impact on the actual explanatory capability of the model and also in the interpretation of results. For both reasons, a different choice of which covariates deserve to be included in the model can lead to different results. Note also that the parameters of interest (i.e., the $α$ s in expression 2) do represent the actual mean team effects which are, by definition, independent by the covariate effects. As a result, one hypothetical team with a low value for $α$ but which is also used to vice versa take large values on any positively-to-performance related covariate tend to be underestimated, despite, as a matter of facts, it should be ‘better’ ranked when considering also the additional ‘benefit’ from covariate effects. To overcome this issue, it would need a more advanced definition of team performance conditioned on covariates. Since this goal goes far beyond the purposes of this work, we refer the readers to future developments. More details on team label reference and on notation/formulae of performance indicators we used, along with some descriptive statistics, are provided in the Appendix. In Figure 8, summary table results of univariate ANOVA analysis by performance indicator are presented, where $p$ -values are related to classical ANOVA F-test. Not significant (n.s.) terms were removed by using a step-wise procedure (by setting $α$ to enter/remove equal to 15%; this value is typically greater than the usual 5% level so that it is not too difficult to enter predictors into the model. Anyway, this is a kind of accepted value that many commercial software and R packages set up as default). The negative sign observed on $Δ$ (FTA) covariate is presumably due to its possible indirect and inverse relation with the attack performance indicators (see the D_Floor.Per*D_FTA scatter plot in Appendix): The larger $Δ$ (FTA), the more effective seems to be the defence strategy. Note that just only one significant interaction between Home-court effect and Team/Opponent was observed: This means that in two cases out of three, the alpha coefficients (net effect of attack plus defense) are independent from playing at home or away. This result is consistent with the remarks in Harville and Smith (1994), where the authors noted that the effect of team-to-team differences in the home-court advantage was found to be relatively small. Only in the case of $Δ$ (OE), the teams alpha differ when playing home or away but for the sake of simplicity in Figure 8, we reported their mean estimates also for $Δ$ (OE).

Figure 8:

Univariate ANOVA table summary results by performance indicator

Analysis of residuals did not highlight any kind of issue. Residuals were specifically detected against violation of time independence assumption (see the analysis of residuals plots in Appendix). In Figure 9, table results of both normal-based and permutation-based univariate and multivariate $p$ -values from hypothesis testing as in (3) are presented. At univariate level, each team displays the estimated alpha coefficient and the related greater-than significance $p$ -value ( $H_{1 ijk}^{+} : α_{ijk} > 0$ ) in both normal-based (above in the cell) and permutation-based (below in the cell) versions. Multivariate combined $p$ -values are shown for each less-than and greater-than alternative. In the last right columns, the main finale outcome of the regular season is shown, that is, the number of wins and losses and the final official ranking. Interestingly and in confirmation of simulation results, parametric and non-parametric $p$ -values do provide very similar univariate and multivariate $p$ -values that are also in tune with the regular season finale outcome as reported in the last right columns of Figure 9. In fact, the stronger/weaker teams are associated with larger/lower positive/negative estimated coefficients and lower combined greater-than/lower-than $p$ -values. The general agreement between our analysis and the regular season finale outcome is definitely proved by calculating the Spearman rho correlation coefficients between the number of wins and (a) normal combined less-than $p$ -values: $ρ = + 0.89$ , (b) normal combined greater-than $p$ -values: $ρ = - 0.82$ , (c) permutation combined less-than $p$ -values: $ρ = + 0.88$ , (d). permutation combined greater-than $p$ -values: $ρ = - 0.81$ ; all four rho are significant at 1% level. By following the suggestion of Harville (2003), the estimated alpha coefficients prove to provide a pretty reliable ranking method: In fact, when calculating the Spearman rho correlation coefficients between the number of wins and the univariate estimated alpha by using as response (a) $Δ$ (PTS), (b) $Δ$ (OE) and (c) $Δ$ (Floor.Per), we obtained, respectively, +0.89, +0.78 and +0.80 (all three rho are significant at 1% level).

Figure 9:

Notes: In bold are highlighted less than 10% combined p-values

Using an heat-map-like representation, Figures 10 and 11 display the multivariate normal- and permutation-based $p$ -values from team-pairwise comparisons as stated in (4). As the multivariate combined $p$ -values are less the 5% and approach to 0, the more intense becomes the red in the related cells. In a given row, each team is compared with the others versus the greater-than alternative. Vice versa, along the column each team is analysed versus the opposite less-than direction. In order to make easier and more intuitive the interpretation, note that the team labels have been reordered according to the final outcome of the regular season (Figure 9, last column). As a result, if the proposed methodology worked fine, we expect to find out the red highlighted cells mostly in the upper-right area of the table, and this is just what happens.

Figure 10:

Heat-map of multivariate pairwise one-sided normal $p$ -values (from hypothesis testing as in (4))

Figure 11:

Heat-map of multivariate pairwise one-sided permutation $p$ -values (from hypothesis testing as in (4))

Accordingly to simulation results, both pairwise normal-based and permutation-based combined directional $p$ -values do provide similar results. The normal-based $p$ -values are in general smaller by taking advantage of the relative large dataset. In Figure 12, results of ranking analysis by pairwise testing method are reported. Note that the normal-based ranking looks like somewhat more reliable and this confirms the previously mentioned clues. In fact, since the significant combined normal-based $p$ -values are generally much more closer to zero than their permutation-based counterparts, they are more likely to keep their significance also after applying a multiplicity correction (Arboretti Giancristofaro et al., 2012a), which in this case is severe because of the relative large number of teams.

Figure 12:

Ranking analysis results (10% significant $α$ level) by pairwise testing

As a final remark, it is interesting to underline that the final outcomes of play-off and play-out, not reported nor analysed here, are basically in tune with our findings. In fact, semi-finals were reached by MIO, VE, AV and TN, and eventually VE won the championship by defeating TN in the finals.

6 Conclusion and final remarks

The main goal of the article was proposing a new approach to infer on parameters related to the net relative strength or weakness of teams within a league based on modelling results of sport matches within a round-robin setting with a set of paired fixed effect linear models. Even if our reference modelling for data sport performance has been used widely in the literature (Harville, 2003), the novelty of our approach relays on how to do inference in the set of parameters of interest; therefore, we take into account two main issues:

The team ranking is obtained by hypothesis testing and not simply by ordering any point estimate of suitable parameters;

Our testing and ranking approach is multivariate in nature, so that we may jointly consider not just one but a (possible large) set of outcomes/performances.

As a result of point 1, our approach is able to handle with tied teams, that is, truly ex-aequo condition, which seem to be a realistic situation happening within data sport analytics. As an advantage from point 2, we are allowed to extract from sport outcomes more information than what is usually done by univariate models. Both the hypothesis testing methods we proposed appeared as reliable tools, in particular with good behaviour irrespective of the shape of the multivariate underlying random distribution. In general, since we are not assuming normality of random errors as done by the traditional testing approaches for linear modelling, the methodology we proposed provides a flexible and less demanding in terms of underlying assumptions tool to infer on the presence of possible multivariate-related dominances that may take place among a set of several multivariate populations, that is, teams or, as forthcoming future extension, among set of players or just single players within a team.

From the practical point of view in its simplest univariate version, the proposed methodology allows us to infer on the teams average net scoring within a league, while in its more intriguing multivariate layout it is suitable for looking for any team-related global dominance using a wide set of performance indicators. The proposed approach can be in future extended to not only numerical but also ordered categorical performance indicators (Arboretti Giancristofaro et al., 2012b,c).

Finally, we presented a real application to basketball in order to highlight that the proposed methodology can be effective to face some real problems in sport performance analytics. In this connection, we suggest that some directions for future research should be addressed to a more in-depth pre-analysis to carefully select the relevant list of sport performances and covariates with the goal of making our analysis more predictive in terms of consistency with sport, outcomes and the related ranking rules and rationales. Finally, by taking inspiration of what is done in statistical process control (Corain and Salmaso, 2013, 2014), a kind of innovative multivariate control chart could be proposed as a way to monitor the teams performance time-by-time during the tournament.

Appendix

Notation and formulae of performance indicators

$PTS = scored points$

$OE = Offensive Efficiency = (FGM + AST) / (FGA - ORB + AST + TO)$

where

$FGM : field goals made, AST : assists, FGA : field goal attempts, ORB : offensive rebounds, AST : assists, TO : turnovers$

$Floor . Per = Floor Percentage = Sc . Poss / Poss$

where

$Sc . Poss = (Scoring Possessions) = FGM - [1 - (1 - FT . Per)^{2}] \times 0.4 \times FTA$ \\ $FTA : free throw attempts, FT . Per : free throw percentage$

Table 1:

Team label reference

Team label	Team name	Team label	Team name
AV	Sidigas Avellino	PS	Consultinvest Pesaro
BR	Enel Brindisi	PT	The Flexx Pistoia
BS	Germani Basket Brescia	RE	Grissin Bon Reggio Emilia
CDO	Betaland Capo d'Orlando	SS	Banco di Sardegna Sassari
CE	Pasta Reggia Caserta	TN	Dolomiti Energia Trento
CNT	MIA-Red October Cantù	TO	Fiat Torino
CR	Vanoli Cremona	VA	Openjobmetis Varese
MIO	EA7 Emporio Armani Milano	VE	Umana Reyer Venezia

Figure A1

Performance and covariate scatter plots

Figure A2

Analysis of residuals

Acknowledgments

The authors would like to thank the anonymous reviewers for their helpful and constructive comments that greatly contributed to improving the quality of the manuscript. It has to be acknowledged that authors contributed equally likely to this manuscript.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The authors received no financial support for the research, authorship and/or publication of this article.

References

Adam

(2016) Generalised linear model for football matches prediction. In CEUR Workshop Proceedings, Proceedings of the Workshop on Machine Learning and Data Mining for Sports Analytics, edited by Jan Van Haaren, Mehdi Kaytoue and Jesse Davis, Riva del Garda, Italy, 19 September 2016, page 1842.

Arboretti Giancristofaro

Bolzan

Bonnini

Corain

Solmi

(2012a) Advantages of closed testing method for multiple comparison procedures. Communications in Statistics Simulation and Computation , 41, 746–763.

Arboretti Giancristofaro

Bonnini

Corain

Salmoso

(2014) A permutation approach for ranking of multivariate populations. Journal of Multivariate Analysis , 132, 39–57.

Arboretti Giancristofaro

Corain

Ragazzi

(2012b) The multivariate randomized complete block design: A novel permutation solution in case of ordered categorical variables. Communications in Statistics Theory and Methods , 41, 3094–3109.

Arboretti Giancristofaro

Corain

Ragazzi

(2012c) A comparison among combination-based permutation statistics for randomized complete block design. Communications in Statistics Simulation and Computation , 41, 964–979.

Bonnini

Corain

Marozzi

Salmaso

(2014) Nonparametric Hypothesis Testing: Rank and Permutation Methods with Applications in R . Chichester: Wiley.

Bradley

Terry

(1952) Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 39, 324–345.

Bretz

Hothorn

Westfall

(2010) Multiple Comparisons Using R . Boca Raton, FL: Chapman & Hall/CRC.

Cattelan

(2012) Paired comparison data: A review with emphasis on dependent data. Statistical Science , 27, 412–423.

10.

Cattelan

Varin

Firth

(2013) Dynamic Bradley—Terry modelling of sports tournaments. Journal of the Royal Statistical Society Series C Applied Statistics , 62, 135–150.

11.

Cervone

DAmour

Bornn

Goldsberry

(2016) A multiresolution stochastic process model for predicting basketball possession outcomes. Journal of the American Statistical Association , 111, 585–599.

12.

Chan

(2011) Prediction accuracy of linear models for paired comparisons in sports. Journal of Quantitative Analysis in Sports , 7, 1–35.

13.

Chen

Fan

(2016) A functional data approach to model score difference process in professional basketball games. Journal of Applied Statistics , 45, 112–127.

14.

Corain

Arboretti

Bonnini

(2016) Ranking of Multivariate Populations: A Permutation Approach with Applications . Boca Raton, FL: Chapman & Hall/CRC.

15.

Corain

Salmaso

(2004) Multivariate and multistrata nonparametric tests: The NPC method. Journal of Modern Applied Statistical Methods , 3, 443–461.

16.

Corain

Salmaso

(2007a) A critical review and a compara- tive study on conditional permutation tests for two-way ANOVA. Communications in Statistics Simulation and Computation , 36, 791–805.

17.

Corain

Salmaso

(2007b) A nonparametric method for defining a global preference ranking of industrial products. Journal of Applied Statistics , 34, 203–216.

18.

Corain

Salmaso

(2013) Nonparametric permutation and combination-based multivariate control charts with applications in microelectronics. Applied Stochastic Models in Business and Industry , 29, 334–349.

19.

Corain

Salmaso

(2014) Nonparametric permutation-based control charts for ordinal data. In Springer Proceedings in Mathematics & Statistics: Topics in Nonparametric Statistics—Proceedings of the First Conference of the International Society for Nonparametric Statistics, edited by Michael G Akritas, S N Lahiri and Dimitris N Politis, Vol. 74, pages 309–21. New York, NY: Springer.

20.

Corain

Salmaso

(2015) Improving power of multivariate combination-based permutation tests. Statistics and Computing , 25, 203–14.

21.

Franks

Miller

Bornn

Goldsberry

(2015) Characterizing the spatial structure of defensive skill in professional basketball. The Annals of Applied Statistics , 9, 94–121.

22.

Goddard

Asimakopoulos

(2004) Forecasting football results and the efficiency of fixed-odds betting. Journal of Forecasting , 23, 51–66.

23.

Harville

(2003) The selection or seeding of college basketball or football teams for postseason competition. Journal of the American Statistical Association , 98, 17–27.

24.

Harville

Smith

(1994) The home-court advantage: How large is it and does it vary from team to team. The American Statistician , 48, 22–28.

25.

Karlis

Ntzoufras

(2003) Analysis of sports data by using bivariate Poisson models. Statistician , 52, 381–393.

26.

Kenny

Kashy

Cook

(2006) The Analysis of Dyadic Data . New York, NY: Guilford Press.

27.

Koning

(2000) Balance in competitions in Dutch soccer. Statistician , 49, 419–431.

28.

Kowalchuk

Headrick

(2000) Simulating multivariate g-and-h distributions. British Journal of Mathematical and Statistical Psychology , 63, 63–74.

29.

Meilgaard

Carr

Civille

(2006) Sensory Evaluation Techniques, 4th Edition . Boca Raton, FL: CRC Press.

30.

Nestler

(2016) Restricted maximum likelihood estimation for parameters of the social relations model. Psychometrika , 81, 1098–1117.

31.

Oliver

(2004) Basketball on Paper: Rules and Tools for Performance Analysis . Dulles, VA: Potomac Books.

32.

Pesarin

Salmaso

(2010) Permutation Tests for Complex Data: Theory, Applications and Software . Chichester: Wiley.

33.

Poole

(2017) EmpiricalBrownsmethod: Uses brown's method to combine p-values from dependent tests. R package version 1.6.0 .

34.

Poole

Gibbs

Shmulevich

Bernard

Knijnenburg

(2015) Combining dependent p-values with an empirical adaptation of Browns method. Bioinformatics , 32, 430–436.

35.

Rasmussen

Trick

(2008) Round robin scheduling: A survey. European Journal of Operational Research , 188, 617–636.

36.

Roy

(1953) On a heuristic method of test cons- truction and its use in multivariate analysis. Annals of Mathematical Statistics , 24, 220–38.

37.

Stefani

(1980) Improved least squares football, basketball, and soccer predictions. IEEE Transactions on Systems, Man, and Cybernetics , SMC-10, 116–23.

38.

Stern

(1992) Whos number one? Rating football teams. In Proceedings of the Section on Statistics in Sports, Vol. SMC-10, pages 1—6. Alexandria, VA: American Statistical Association.

39.

Stern

(1995) Whos number one in college football?... and how might we decide? Chance , 8, 7–14.

40.

Usami

(2017) Bayesian longitudinal paired comparison model and its application to sports data using weighted likelihood bootstrap. Communications in Statistics: Simulation and Computation , 46, 1974–1990.

41.

Vracar

Trumbelj

Kononenko

(2016) Modeling basketball play-by-play data. Expert Systems With Applications , 44, 58–66.

Testing and ranking on round-robin design for data sport analytics with application to basketball

Abstract

Keywords

1 Introduction and motivation

2 Modelling for data Sport Outcomes

3 Testing and ranking on round-robin design for data sport analytics

Figure 1:

Bivariate mean alpha values by competitor and scenario

Simulated errors and simulated team performance from scenario 1 by type of error

Legend for rejection rates results in Figures 4 and 5

Significant testing on alpha: Rejection rates test by method and type of multivariate distribution

Pairwise testing on alpha: Rejection rates test by method and type of multivariate distribution

Power comparison, at 5% significant α -level, on significant and pairwise combined tests by testing method and type of multivariate distribution

Ranking performance comparison, at 5% significant α -level, by scenario, ranking method and type of multivariate distribution

Figure 8:

Univariate ANOVA table summary results by performance indicator

Heat-map of multivariate pairwise one-sided normal p -values (from hypothesis testing as in (4))

Heat-map of multivariate pairwise one-sided permutation p -values (from hypothesis testing as in (4))

Ranking analysis results (10% significant α level) by pairwise testing

Appendix

Team label reference

Performance and covariate scatter plots

Analysis of residuals

Acknowledgments

Declaration of conflicting interests

Funding

References

Power comparison, at 5% significant $α$ -level, on significant and pairwise combined tests by testing method and type of multivariate distribution

Ranking performance comparison, at 5% significant $α$ -level, by scenario, ranking method and type of multivariate distribution

Heat-map of multivariate pairwise one-sided normal $p$ -values (from hypothesis testing as in (4))

Heat-map of multivariate pairwise one-sided permutation $p$ -values (from hypothesis testing as in (4))

Ranking analysis results (10% significant $α$ level) by pairwise testing