Statistical modelling of gained university credits to evaluate the role of pre-enrolment assessment tests: An approach based on quantile regression for counts

Abstract

Considering the case of the School of Economics of the University of Florence, the paper investigates whether the pre-enrolment assessment test is an effective tool to predict student performance. The analysis is tailored to evaluate the additional information yielded by the test beyond the background characteristics of the candidates already available from administrative records, such as the high school type and final grade. The student performance is measured by the number of gained credits after one year, which is a count variable with an irregular distribution and a peak in zero. These features pose a challenge in statistical modelling, which is solved by a two-part model with a logit specification for the zeros, while positive values are analyzed by quantile regression for counts. To disentangle direct and indirect effects of background variables, the result of the pre-enrolment assessment test is treated as an intermediate variable in a regression chain graph. The results show that the pre-enrolment test adds some information to predict student performance, which can be exploited for tutoring.

Keywords

Hurdle model quantile regression regression chain graph two-part model zero-inflated counts

1 Introduction

Predicting student performance is a key step in order to improve the efficiency of university systems. Indeed, delays or failures are costly for both the students and the administration. Therefore, it is of primary importance to determine the factors associated with the performance in order to plan actions such as restrictions to the access and tutoring. To this end, universities can typically rely on information about students’ high school career, such as the type of school and various measures of proficiency. However, the results at high school are not fully appropriate to predict the academic performance due to several limitations, including the possible mismatch between the competencies evaluated at high school and those required for a given degree programme, and the heterogeneity in the criteria for awarding marks (usually, there is substantial variability across types of schools and across geographical regions). To overcome these limitations, several universities devise a pre-enrolment assessment test tailored on the needs of each degree programme. However, a quick look at pre-enrolment tests around the world reveals a lack of commonly accepted guidelines and a shortage of empirical evidence about their predictive ability.

The literature on the empirical research about predicting student performance at university is scattered in several journals, ranging from Psychology to Economics. Noteworthy papers include Murray-Harvey (1993), Wedman (1994), Hoefer (2000), Murphy et al. (2001), Maree et al. (2003), Dancer (2004), Win and Miller (2005), Smith and Naylor (2005), Birch and Miller (2006); Birch and Miller (2007), Mills et al. (2009), Mallik and Lodewijks (2010), Bianconcini and Cagnone (2012), Chowdhury (2012) and Adelfio et al. (2014).

The statistical modelling of student performance at university is challenging due to the complexity of the process. The pre-enrolment test is an instrument to measure student competencies in addition to already known characteristics, such as the high school final grade. Therefore, it is important to assess the value added by the test and to disentangle the effect of the available characteristics on student performance into a direct effect and an indirect effect mediated by the test. To this end, we use regression chain graphs (Wermuth and Sadeghi 2012).

Italian universities are characterized by a high dropout rate and a large proportion of slowly progressing students; thus, it is of primary interest to monitor the progression of students, which is conventionally measured by credits. The Italian credit system is consistent with the European Credit Transfer and Accumulation System (ECTS): one academic year corresponds to 60 credits, with one credit representing 25 hours of study. In this paper, we focus on gained credits at the end of the first year. Indeed, a good performance during the first year is crucial to graduate in time, moreover most dropouts happen during the first year.

A complication for the statistical modelling of gained credits is that the observed distribution is typically quite irregular; in fact, exams yield different credits and the sequence of exams varies across students; moreover, the distribution usually has peaks at zero and at the maximum. In particular, in our data, there is no peak at the maximum, but there is a huge peak at the minimum ( $23 %$ ).

The main approaches to deal with excess zeros are zero-inflated models and two-part (hurdle) models (Mullahy 1986). We rely on a two-part model since it lets the zeros and the positives to arise from different data-generating processes, entailing a separate model for the probability of gaining no credits. Indeed, students failing to gain any credit deserve special attention in order to plan interventions aimed at preventing dropout.

Two-part models usually rely on a parametric distribution for the positives, for example a Poisson distribution. However, in our application it is difficult to devise a suitable parametric specification due to the multi-modality of the distribution of gained credits. Thus, a more flexible approach is needed, such as a mixture model (Böhning and Kuhnert 2006). In this vein, Grilli et al. (2015) analyzed gained credits using a concomitant variable binomial mixture model (Dayton and Macready 1988). The mixture approach not only yields a satisfactory fit, but it also entails difficulties in implementation, for example the selection of the number of mixture components, and difficulties in interpreting the parameters, in particular the coefficients of the multinomial logit model for the probabilities of the mixture components. Such difficulties can be overcome by quantile regression for counts (Machado and Santos Silva 2005), which is even more flexible as it does not require to specify any parametric distribution for the outcome.

Quantile regression (Koenker 2005; Kneib 2013; Davino et al. 2014) is a methodology to analyze the relationships between the quantiles of the outcome and a set of explanatory variables. Most of the theoretical developments and empirical applications concern the linear specification, which is suitable for a continuous outcome. Noteworthy applications to the analysis of the performance of university students are Birch and Miller (2006) and Adelfio et al. (2014). The extension of quantile regression to count data raises several issues that can be solved by the approach of Machado and Santos Silva (2005), which has been recently applied to the analysis of fertility data (Miranda 2008; Booth and Kee 2009), frequency of individual doctor visits (Winkelmann 2006; Moreira and Barros 2010) and traffic accidents (Qin and Reyes 2011). A Bayesian implementation to analyze respiratory hospital admissions has been developed by Lee and Neocleous (2010). To the best of our knowledge, quantile regression for counts has not yet been applied in education, though it is well suited for the analysis of data such as gained credits.

In this paper, we analyze credits gained by students enrolled in the School of Economics of the University of Florence in the academic year 2008/2009. The novelty of our approach lies in embedding quantile regression for counts into a two-part model. This raises an issue about how to make predictions which is solved by a procedure to compute marginal quantiles. The proposed modelling approach is valuable beyond the specific application to gained credits, as it is a flexible and easy to interpret approach to the analysis of zero-inflated count data.

The rest of the paper is organized as follows. Section 2 presents the case study concerning the pre-enrolment test at the University of Florence, describing the test and reporting summary statistics. Section 3 outlines the statistical methods, namely, regression chain graphs and two-part modelling with a logit specification for the zeros and a quantile regression for positive counts. Section 4 illustrates the results. Finally, Section 5 discusses the main findings.

2 Data and preliminary analysis

In the academic year 2008/2009, the School of Economics of the University of Florence introduced a compulsory test to evaluate the background of the candidates wishing to enrol in one of the degree programmes. The test has three editions (September, November and December) and is based on 40 multiple-choice items covering three areas: Logic (12 items, 30%), Reading (10 items, 25%) and Mathematics (18 items, 45%). For each item, one out of five alternatives is correct, with the following scoring system: 1 if correct, 0 if blank, $- 0.25$ if wrong. Thus, the total score ranges from $- 10$ to 40, and the threshold for passing the test is fixed at 9; candidates with a lower total score are advised against enrolment. These students could still enrol in a degree programme of the School of Economics, but they were allowed to take exams only after passing the test during one of the later editions.

We consider the participants to the main edition of the test (September), which covers almost all the enrolled students. The dataset is obtained by merging data collected at the test with the administrative data of the School of Economics. After deleting 68 foreign students (due to missing values), the dataset has 1057 observations. For these candidates, Figure 1 reports a path tree showing the frequencies of test results, enrolment decisions and credits gained during the first academic year.

Figure 1

Source: Authors' own.

Our aim is to evaluate if the pre-enrolment test is an effective predictor of gained credits in addition to background characteristics of the students already available from administrative records. It is worth to note that our analysis is restricted to students who actually enrolled; thus, it does not provide evidence on the ability of the pre-enrolment test to effectively select students.

The analysis exploits the following student variables available in the administrative records:

Pre-test variables: Female, Far-away resident (indicator for residence in the provinces of Massa-Carrara and Grosseto or in a province out of Tuscany), Type of high school (HS type: Scientific, Humanities, Technical, Other), High school irregular career (indicator for age at high school diploma $> 19$ ), High school grade (from $60$ to $100$ , centred at $80$ );

Test variables: partial test scores on Logic, Reading and Mathematics;

University performance variables: Credits gained during the first year (from 0 to 60).

Other pre-test variables are available, but they are not listed above as they are not statistically significant in the fitted models.

Table 1 reports the test results and the enrolment rates for students participating to the pre-enrolment assessment test in September 2008 (students with complete data). Females are slightly more numerous than males; 112 out of 1057 (11%) of the candidates are far-away residents. Most participants come from a scientific or technical high school and 136 ( $13 %$ ) of the candidates have an irregular high school career. As for the high school grade, 405 candidates ( $38$ %) are above the mid-point ( $80$ ).

Table 1

Test results and enrolment decisions for students participating to the pre-enrolment assessment test in September 2008 (students with complete data). School of Economics, University of Florence.

,		Average test score				%Test	%
	N	Total	Logic	Reading	Math	passed	enrolled
		max 40	max 12	max 10	max 18
All	1057	15.0	5.1	4.8	5.1	80.7	65.3
Gender
Male	518	16.3	5.4	5.2	5.7	84.6	68.9
Female	539	13.8	4.8	4.5	4.5	77.0	61.8
Far-away resident
No	945	15.2	5.1	4.8	5.3	81.6	68.0
Yes	112	13.8	4.8	4.9	4.1	73.2	42.0
HS type
Scientific	379	18.4	5.9	5.1	7.4	93.9	67.0
Humanities	106	16.1	5.9	5.8	4.4	89.6	50.0
Technical	382	13.2	4.5	4.5	4.2	74.1	71.7
Other	190	11.2	4.1	4.3	2.9	62.6	57.4
HS irreg. career
No	921	15.4	5.2	4.8	5.4	81.5	65.7
Yes	136	12.5	4.6	4.5	3.4	75.0	62.5
HS grade
$\leq 80$	652	13.9	4.8	4.4	4.7	77.1	66.6
$> 80$	405	16.9	5.5	5.5	5.8	86.4	63.2

Source: Authors' own.

The test is passed by $853$ candidates ( $80.7 %$ ). The average total score is $15.0$ out of 40. The partial average scores are $5.1$ out of $12$ in Logic, $4.8$ out of $10$ in Reading and $5.1$ out of $18$ in Mathematics. These results are quite unsatisfactory, especially for Mathematics. As expected, the pre-test career affects the test results: the high school type is very relevant, for example, the test was passed by 94% of candidates from scientific high school, as compared to $74 %$ of candidates from technical education; the results are better for candidates with a regular career and a school grade over $80$ .

The test is designed as a tool for student pre-enrolment assessment, thus passing the test is not mandatory for enrolment. However, the test result influences the probability of enrolment: the enrolment rate was $67.9 %$ for candidates who passed the test and $54.4 %$ for candidates who did not pass the test (see Figure 1).

In the following, we restrict the analysis to $690$ students who took the test and enrolled at the School of Economics in academic year 2008/2009 in order to evaluate their performance after one year. As discussed in the introduction, a relevant performance indicator is the number of credits gained during the first academic year, namely, by the end of year 2009. Table 2 summarizes the relationship between gained credits and total test score: students with a higher score have a better performance in terms of both the probability of acquiring at least one credit and the average number of credits. It is worth to note that students with poor test results have remarkable chances of not acquiring any credit (e.g., $41.4 %$ for those not passing the test), posing serious problems for the university management. Indeed, acquiring no credit is strongly related to dropout: the dropout rate is $77 %$ among students not gaining credits in the first year, as compared to $11 %$ among students gaining at least one credit. Therefore, in our analysis, we use the number of gained credits after one year as a single outcome which is related to both dropout and speed of progression.

Table 2

Gained credits after one year by pre-enrolment test result ’ academic year 2008/2009, School of Economics, University of Florence

,		Credits			Avg. Credits
Test result		N	$= 0$	$> 0$	(Credits $> 0$ )
Not passed ( $< 9.00$ )	111	41.4%	58.6%	23.4
Passed with score
$9.00 - 12.75$	( $1^{st}$ quarter)	147	25.9%	74.1%	25.7
$13.00 - 16.25$	( $2^{nd}$ quarter)	150	23.3%	76.7%	29.0
$16.50 - 20.25$	( $3^{rd}$ quarter)	143	21.0%	79.0%	32.8
$\geq 20.50$	( $4^{th}$ quarter)	139	7.2%	92.8%	34.1
All		690	23.0%	77.0%	29.8

Source: Authors' own.

3 Methods

The analysis aims at evaluating whether the university pre-enrolment assessment test is a good predictor of a freshman performance in terms of gained credits. In particular, it is crucial to assess the value added by the test over background characteristics, which are easy to collect and include some ability measures such as the high school grade. To this end, we exploit a regression chain graph model where the background variables affect the test score and then both the background variables and the test score affect the freshmen performance in terms of gained credits. This approach allows us to evaluate the ability of the pre-enrolment test to predict the number of gained credits controlling for background variables. At the same time, the effects of background variables are decomposed into direct and indirect components, giving insight into the abilities measured by the pre-enrolment test.

3.1 Regression chain graph

Regression chain graphs are a class of graphical models for description of conditional independence structures. In this framework, graphical models are associated with a chain graph (Cox and Wermuth 1996; Wermuth and Sadeghi 2012), namely, a graph which may have both directed and undirected edges, but without directed cycles. These graphs are useful when the variables of interest admit a partial ordering on the basis of subject matter considerations, such as timing. Variables are partitioned into blocks and the variables belonging to the same block are considered to be of equal standing. By convention, blocks are ordered from right to left: the right-most block includes pure explanatory variables, the left-most block includes pure response variables, whereas the middle blocks hold intermediate variables, which are responses with respect to variables in blocks on the right and explanatory variables for the variables in blocks on the left. Variables belonging to different blocks are either disconnected or joined by an arrow representing a conditional dependence; an arrow between two variables is traced when the corresponding regression coefficient is statistically significant at a given level, such as $5 %$ .

As later shown in Figure 3, in our application the variables are partitioned into three blocks according to time ordering: (i) pre-test characteristics (pure explanatory variables); (ii) standardized test scores (intermediate variables); and (iii) gained credits after one year (pure response variable). The pre-test variables in the right-most block are regarded as fixed for analysis; thus, their relationships with each other are not represented and the block is enclosed in double lines, as in Cox and Wermuth (1996).

The middle box in Figure 3 includes the three partial scores on Logic, Reading and Math. The scores have been standardized to make them comparable. The use of the standardized partial scores instead of the raw total score has the merit of giving insight into the role of the three subject areas and avoiding the implicit weighting due to the different number of items of the areas. The standardized test scores in the middle block are connected by dashed lines to denote that they are associated conditionally on pre-test variables (Wermuth and Sadeghi 2012). Accordingly, the standardized scores are jointly regressed on pre-test variables via a multivariate regression model.

The number of gained credits is regressed on test and pre-test variables using the two-part logit-quantile regression model defined in the following subsection.

3.2 Two-part model: Logit specification for the zeros and quantile regression for positive counts

The model for the pure response variable (left-most block in Figure 3) has to account for the peculiar pattern of gained credits after one year reported in Figure 2. In particular, the distribution of gained credits shows an excess of zeros (23% of freshmen did not gain any credit). Moreover, exams have different credits, usually 6, 9 or 12, thus the distribution of positive credits is quite irregular: the main peaks are at 6, 15, 24, 36 and 45 credits, depending on the path followed by the student. Considering the 531 students gaining at least one credit, the median is $30$ credits, while only $0.75$ % gained the maximum value $60$ . Since each exam gives a number of credits multiple of $3$ , we define the response variable as $Y = credits / 3$ , where $credits$ is the number of gained credits at the end of the first year, therefore the response $Y$ is a count ranging from $0$ to $20$ .

Figure 2

Source: Authors' own.

In order to allow the zeros and the positives to come from two different data-generating processes, we specify a two-part or hurdle model (Mullahy 1986) with two components: a model for the probability of gaining at least one credit, $P (Y > 0 ∣ x)$ , and a model for the distribution of the positive part, $f (y ∣ x, Y > 0)$ , where $x$ is the vector of covariates including the variables in the middle and right-hand boxes of Figure 3. The two parts of the model are functionally independent, thus their parameters can be estimated separately. The first sub-model is fitted on the whole sample using a logit specification:

logit P (Y > 0 ∣ x) = x^{'} α .

(3.1)

Model (3.1) allows us to analyze how student background characteristics and test scores affect the probability of gaining at least one credit.

Figure 3

Source: Authors' own.

As for the second sub-model, $f (y ∣ x, Y > 0)$ , no parametric distribution appropriately describes the pattern shown by the credits (see Figure 2). Therefore, in order to avoid distributional assumptions and to account for the discrete nature of credits, we use quantile regression for counts. Quantile regression (Koenker 2005; Kneib 2013; Davino et al. 2014) is a methodology to study the relationships between the quantiles of the outcome and a set of covariates, without any distributional assumption. This approach is very flexible since regression equations at different quantiles are fitted separately, for example, it is possible that a given covariate has a negligible effect on the 0.5 $th$ quantile (i.e., the median) and a large effect on the 0.9 $th$ quantile (i.e., the right tail of the distribution). Therefore, in quantile regression the dependence of the outcome on the covariates is described in a finer way as compared to a standard regression model, which is based on a single equation for the mean of the outcome.

The methodology of quantile regression is well established for continuous outcomes, whereas the extension to count data raises several issues. The main difficulty is that the conditional quantile function of a discrete random variable cannot be a continuous function of the regression parameters. Here, we rely on the proposal of Machado and Santos Silva (2005), which is based on smoothing the counts through jittering in order to obtain a continuous working variable. Jittered data can be generated by adding a uniform random variable $U$ , defined on the interval $[0, 1)$ , to the count variable $Y$ , thus yielding a continuous variable $Z = Y + U$ . This approach also requires to choose a monotone transformation of the conditional quantile function of $Z$ . As suggested by Machado and Santos Silva (2005), we use the following specification similar to Generalized Linear Models for counts:

Q_{Z} (τ ∣ x, Y > 0) = τ + exp (x^{'} β (τ)),

(3.2)

where

Q_{Z}

denotes the conditional quantile function of the working variable

Z

. Specifically,

Q_{Z} (τ ∣ x, Y > 0)

is the

τ

th quantile,

τ \in (0, 1)

, of the distribution of

Z

conditional on the covariates

x

. In our case, the distribution of

Z

is also conditional on

Y > 0

, since quantile regression is applied to

f (y ∣ x, Y > 0)

, i.e., the positive component of the two-part model.

It is worth to note that quantile regression for counts refers to a standard count variable with support ${0, 1, 2 . . .}$ , whereas the conditional distribution of $Y$ given $Y > 0$ has support ${1, 2, 3 . . .}$ . This shift is expected to have little impact on model fit because quantile regression does not assume a parametric distribution for the response. We checked that fitting quantile regression on $W = Y - 1$ yields similar results.

The conditional quantile function of the count variable $Y$ is obtained from (3.2) as:

Q_{Y} (τ ∣ x, Y > 0) = ⌈ Q_{Z} (τ ∣ x, Y > 0) - 1 ⌉,

(3.3)

where

⌈ t ⌉

denotes the ceiling function that returns the smallest integer greater than, or equal to,

t

. The approach entails estimating the regression parameters

β (τ)

through the linear quantile estimation algorithm applied to the transformed jittered variable

Z

(see formula (7) in Machado and Santos Silva (2005). Under mild conditions on the covariates, it can be proved that the resulting estimators are consistent and asymptotically normal. Furthermore, in order to average out the uniform noise, the authors propose to generate

m

‘jittered’ samples and average the

m

estimates. The ‘average-jittering’ estimator

{\hat{β}}_{m}^{A} (τ) = \frac{1}{m} \sum_{l = 1}^{m} {\hat{β}}^{(l)} (τ)

(3.4)

is proved to be more efficient than the estimator obtained with a single sample. The simulation study in Machado and Santos Silva (2005) shows that the proposed estimator (3.4) has good properties in finite samples of size

n = 500

, similar to our application.

In linear quantile regression, local model fit for each quantile $τ$ can be evaluated through the $R^{1} (τ)$ measure defined by Koenker and Machado (1999):

R^{1} (τ) = 1 - \frac{V_{full} (τ)}{V_{null} (τ)}

(3.5)

where

V_{model} (τ) = \sum_{i : y_{i} \geq {\hat{y}}_{i} (τ)} τ ∣ y_{i} - {\hat{y}}_{i} (τ) ∣ + \sum_{i : y_{i} < {\hat{y}}_{i} (τ)} (1 - τ) ∣ y_{i} - {\hat{y}}_{i} (τ) ∣,

with

{\hat{y}}_{i} (τ)

representing the predicted quantile for the

i

th subject. The quantity

V_{full} (τ)

is obtained plugging in

{\hat{y}}_{i} (τ)

predicted from the current model, while

V_{null} (τ)

is the analogous for the null model. Since

V_{model} (τ)

is the objective function of the estimation algorithm for linear quantile regression,

R^{1} (τ)

has an interpretation similar to the conventional

R^{2}

. Even if

V_{model} (τ)

is not the objective function in quantile regression for counts,

R^{1} (τ)

still has an

R^{2}

-like interpretation, thus we propose to use this measure to the count data setting with

{\hat{y}}_{i} (τ) = ⌈ τ + exp (x_{i}^{'} \hat{β} (τ)) - 1 ⌉

, obtained from formulas (3.2) and (3.3).

The results of the quantile regression for counts can be summarized in many ways. Following Machado and Santos Silva (2005), we will report the partial effects of the covariates on the quantiles of the continuous jittered variable $Z$ , instead of the count variable $Y$ , thus allowing a finer evaluation of the effects. Specifically, we will consider the partial effects on $Q_{Z} (τ ∣ x_{*}, Y > 0)$ , where $x_{*}$ is a vector containing the mean value of the continuous covariates and zeros for the dummy covariates. The partial effect for a continuous covariate $x_{k}$ is the derivative $\partial Q_{Z} (τ) / \partial x_{k} = β_{k} (τ) exp (x_{*}^{'} β (τ))$ , while for a binary covariate $x_{s}$ with coefficient $β_{s}$ the partial effect is the discrete change in $Q_{Z} (τ)$ , i.e., $exp (x_{*}^{'} β (τ) + β_{s} (τ)) - exp (x_{*}^{'} β (τ))$ .

In our two-part specification, the quantile regression model for counts (3.2) is fitted on the subset of students who gained at least one credit ( $Y > 0$ ). Quantile regression could be applied directly to the whole sample (Machado and Santos Silva 2005), i.e., including students with zero credits. However, we have a special interest in students who do not gain any credit, thus we prefer the two-part specification that allows us to explicitly model the probability of gaining zero credits. On the contrary, a quantile regression model on the whole sample would yield results at a fixed grid (e.g. $τ = 0.10, 0.25, \dots$ ) which do not allow to directly compute $P (Y = 0 ∣ x)$ . Indeed, computing $P (Y = 0 ∣ x)$ for a given value of $x$ would require searching the maximum $τ$ such that $Q_{y} (τ ∣ x) = 0$ .

A relevant use of the proposed two-part model is in making predictions, for example predicting the outcome for a hypothetical new unit with covariate vector $x_{*}$ . In a standard two-part model, this prediction is usually obtained through the marginal expectation. However, since the proposed two-part model has a component based on quantile regression, we suggest to predict the outcome through marginal quantiles.

Let us denote with $Q_{Y} (κ ∣ x_{*})$ the quantile of interest for prediction, for a given $κ \in (0, 1)$ , and with $p_{*}$ the probability of a zero count, i.e., $p_{*} = P (Y = 0 ∣ x_{*}) = 1 - P (Y > 0 ∣ x_{*}) = {(1 + exp (x_{*}^{'} α))}^{- 1}$ from Equation (3.1). If $p_{*} \geq κ$ , the predicted $κ$ -quantile of $Y$ is $Q_{Y} (κ ∣ x_{*}) = 0$ . Otherwise, if $p_{*} < κ$ , the predicted $κ$ -quantile of $Y$ is equal to a specific quantile of the conditional distribution $f (y ∣ x_{*}, Y > 0)$ :

Q_{Y} (κ ∣ x_{*}) = Q_{Y} (τ_{κ, p_{*}} ∣ x_{*}, Y > 0)

(3.6)

where

τ_{κ, p_{*}} = (κ - p_{*}) / (1 - p_{*}),

(3.7)

which is obtained by solving

p_{*} + (1 - p_{*}) τ_{κ, p_{*}} = κ

The marginal quantiles (3.6) can be used to make predictions after plugging in parameter estimates. For example, a point-wise prediction could be based on the median ( $κ = 0.5$ ), while a prediction interval could be based on a couple of quantiles, such as $κ = 0.25$ and $κ = 0.75$ .

4 Results

The analysis is based on $690$ students enrolled at the School of Economics in academic year 2008/2009 who attended the main edition of the pre-enrolment test (see Section 2). Estimation is performed with the statistical software Stata (StataCorp 2013). In particular, quantile regression for counts is fitted through the user-written qcount command of Miranda (2008) implementing the average-jittering estimator (3.4). The reported results are based on $m = 1000$ replicates.

The regression chain graph depicted in Figure 3 is obtained by fitting a multivariate linear regression model for the three test scores and the two-part model of Section 3.2 for gained credits. An arrow between two variables is traced when the corresponding regression coefficient is statistically significant at $5 %$ level. For the two-part model, an arrow is traced when there is at least one statistically significant effect, which could be a coefficient of the logit model or a coefficient of the quantile regression model for the selected quantiles $Q (τ)$ , $τ = 0.10, 0.25, 0.50, 0.75, 0.90$ . The fitted chain graph shows that the effect of far-away resident is fully mediated by the test scores, while the other pre-test covariates have both direct and indirect effects.

Let us consider in detail the results of the regression models for pre-enrolment assessment test scores (Section 4.1) and for gained credits (Section 4.2).

4.1 Pre-enrolment assessment test scores

The results of the multivariate linear regression of pre-enrolment assessment test scores on background characteristics are reported in Table 3. The pre-test variables explain a small part of the variability of the standardized test scores, with $R^{2}$ ranging from $0.129$ for Logic to $0.312$ for Math; thus, the test measures abilities only partially captured by background characteristics. Candidates with a higher school grade have better results in all three areas. All things been equal, females have worse results. The type of high school has a significant effect; compared to candidates from a scientific high school, candidates from humanities have a worse performance in Math and a better performance in Reading, whereas candidates from technical and other high schools have a worse performance in all three areas. Candidates with an irregular career, as well as candidates residing far away, have worse results in Math.

Table 3

Multivariate linear regression of pre-enrolment assessment test scores (middle block of Figure 3) on background characteristics (right-most block). Parameter estimates (standard errors in parenthesis, coefficients with $p$ -value $< 0.05$ printed in italics).

,Variable	Logic	Reading	Math
Intercept	0.500	0.432	0.864
	(0.070)	(0.069)	(0.062)
Female	$-$ 0.230	$-$ 0.442	$-$ 0.328
	(0.077)	(0.075)	(0.068)
Far-away resident	$-$ 0.221	$-$ 0.186	$-$ 0.412
	(0.143)	(0.140)	(0.127)
HS type (ref: Scientific)
Humanities	0.008	0.482	$-$ 0.672
	(0.146)	(0.143)	(0.130)
Technical	$-$ 0.565	$-$ 0.341	$-$ 0.876
	(0.086)	(0.084)	(0.076)
Other	$-$ 0.769	$-$ 0.331	$-$ 1.138
	(0.113)	(0.111)	(0.101)
HS irregular career	0.059	0.030	$-$ 0.404
	(0.111)	(0.109)	(0.099)
HS grade	0.017	0.027	0.023
	(0.003)	(0.003)	(0.003)
Fit measures
Residual std deviation	2.369	2.219	3.003
$R^{2}$	0.129	0.163	0.312

Source: Authors' own.

4.2 Gained credits

The results of the two-part model for gained credits defined by equations (3.1) and (3.2) are shown in Table 4. Since the model parameters are not directly interpretable, we report the partial effects defined in Section 3.2. The partial effects have been evaluated at $x_{*} = 0$ , corresponding to the baseline student, i.e., male, resident in Florence or surrounding provinces, scientific high school, regular career, high school grade at mid-point (80), test scores at mean values (0). The first column of Table 4, concerning the logit model (3.1), shows the partial effects of the covariates on the probability of gaining at least one credit. The following five columns of Table 4, concerning the quantile regression model (3.2), show the partial effects of the covariates on the quantiles of the positive part of the distribution of gained credits; specifically, the partial effects refer to the quantiles $Q (τ) = 3 \times Q_{Z} (τ)$ , namely, the quantiles of the jittered variable $Z$ are multiplied by $3$ in order to recover the original scale of the credits (ranging from $0$ to $60$ ).

Table 4

Two-part model with quantile regression for counts, regressing gained credits (left-most block of Figure 3) on pre-enrolment assessment test scores (middle block) and background characteristics (right-most block). Predictions and partial effects for the baseline student $†$ (delta-method standard errors in parenthesis, coefficients with $p$ -value $< 0.05$ printed in italics).

P (credits > 0)	f (credits ∣ credits > 0)
		$Q (0.10)$	$Q (0.25)$	$Q (0.50)$	$Q (0.75)$	$Q (0.90)$
Prediction
Baseline student $†$	0.849	12.963	19.854	31.874	44.252	48.971
Partial effect
Female	0.016	3.129	3.377	2.182	0.035	0.939
	(0.026)	(1.241)	(1.568)	(2.278)	(1.514)	(1.121)
Far-away resident	$-$ 0.026	0.094	$-$ 1.482	0.888	0.210	$-$ 0.053
	(0.054)	(1.688)	(2.377)	(3.632)	(3.086)	(2.025)
HS type (ref: science)
Humanities	$-$ 0.019	$-$ 3.512	$-$ 0.939	$-$ 1.135	$-$ 0.618	$-$ 0.435
	(0.062)	(2.182)	(9.170)	(2.741)	(2.958)	(2.622)
Technical	$-$ 0.048	$-$ 0.060	1.030	$-$ 2.475	$-$ 3.619	$-$ 2.118
	(0.036)	(1.152)	(1.671)	(2.483)	(2.609)	(2.186)
Other	$-$ 0.047	$-$ 2.386	$-$ 3.749	$-$ 5.682	$-$ 8.179	$-$ 6.749
	(0.048)	(1.663)	(1.715)	(4.179)	(3.797)	(2.587)
HS irregular career	$-$ 0.177	$-$ 2.550	$-$ 6.936	$-$ 10.222	$-$ 11.710	$-$ 3.589
	(0.056)	(0.924)	(1.609)	(2.083)	(7.658)	(4.852)
HS grade (+10)	0.035	4.739	11.246	10.667	9.420	3.020
	(0.008)	(0.934)	(0.918)	(1.258)	(1.469)	(0.785)
Std test scores
Logic	0.016	0.278	1.397	0.443	$-$ 0.557	$-$ 0.279
	(0.014)	(0.718)	(0.771)	(0.939)	(0.733)	(0.637)
Reading	0.050	$-$ 0.018	$-$ 0.238	0.155	0.378	0.555
	(0.016)	(0.649)	(0.754)	(1.034)	(1.070)	(0.806)
Mathematics	0.021	0.686	1.928	3.045	2.909	2.361
	(0.018)	(0.786)	(0.893)	(1.190)	(1.091)	(1.058)
Fit measures
Wald test statistic	67.37	326.9	234.7	226.9	156.67	232.44
Pseudo- $R^{2}$ $♯$	0.106	0.124	0.167	0.161	0.149	0.170

Source: Authors' own.

Notes: $†$ Baseline student: male, resident in Florence or surrounding provinces, HS scientific,

HS regular career, mid-point HS grade (80), test scores at mean values (0).

Wald test for the nullity of all the regression coefficients.

♯ McFadden's pseudo-R² for the logit model, R¹(τ) of equation (3.5) for quantile regression.

The Wald test statistics for the nullity of all the regression coefficients are high for both components of the two-part model. In particular, the covariates jointly have a significant effect at all the considered quantiles. Moreover, the pseudo- $R^{2}$ index, i.e., the $R^{1} (τ)$ index defined in Equation (3.5), shows that the fit is similar for all the considered values of $τ$ ( $0.10, 0.25, 0.50, 0.90, 0.75$ ). The values of $R^{1} (τ)$ are in line with those usually found in applications of linear quantile regression; see, e.g., Koenker and Machado (1999) and Section 4.3.1 in Davino et al. (2014).

The baseline student has a predicted probability of gaining at least one credit equal to $0.849$ . The significant background characteristics are the irregular career and the high school grade. Specifically, the partial effect of an irregular career is $- 0.177$ , yielding a predicted probability of $0.672$ , whereas the effect of a ten-point increase in the high school grade is $0.035$ . Among test scores, Reading is the only significant one, with an effect of $0.050$ for a one-point increase in the standardized score. Figure 4 shows how the probability to gain at least one credit varies with the three test scores for the baseline student.

Figure 4

Source: Authors' own.

Let us now consider the positive part of the distribution of gained credits, namely, columns $Q (0.10)$ to $Q (0.90)$ of Table 4. The first line reports the predicted quantiles for the baseline student, showing the approximate symmetry of the distribution. Subsequent lines report the partial effects of the covariates for the baseline student.

The only pre-test characteristic having an effect on all the considered quantiles is the high school grade; the effect of a ten-point increase is about $10$ credits in the central part of the distribution, and lower in the tails. Gender has a significant positive effect on the left tail of the distribution; females gain about three credits more than males. On the contrary, the high school type has an effect mainly on the right tail of the distribution; students coming from other types of school gain about $7$ credits less than students from a scientific high school. Students with an irregular career have a lower performance; the effect at the median is about $- 10$ credits. The finding that several pre-test covariates are significant is an indication that some abilities and attitudes of the students are not captured by the assessment test.

The bottom part of Table 4 shows the partial effects of the three test scores. The standardized test score on Math has a significant effect, even controlling for pre-test covariates. Perhaps surprisingly, the scores on Reading and Logic do not contribute to predict gained credits when adjusting for Math and pre-test covariates. Estimated quantiles of the conditional distribution $f (y ∣ x, Y > 0)$ for the baseline student as a function of the Math score are computed through formula (3.3) and depicted in Figure 5 (after multiplying the values by 3 to recover the original scale). The quantile functions are stepwise, reflecting the discreteness of the response variable. Note that the spread of the distribution increases with the Math score (the length of the central $80$ % interval is $33$ credits for $Math = - 3$ and $39$ for $Math = + 3$ ).

Figure 5

Source: Authors' own.

In order to predict the number of gained credits for a hypothetical student with covariate vector $x_{*}$ , we apply the procedure outlined in Section 3.2 to compute the marginal quantiles, namely, the quantiles of the distribution $f (y ∣ x_{*})$ . For example, let us consider the baseline student ( $x_{*} = 0$ ) and the median ( $κ = 0.5$ ): plugging in the parameter estimates of the logit model (3.1), the predicted probability to achieve zero credits is $p_{*} = 1 - 0.849 = 0.151$ , thus the value of $τ_{κ, p_{*}}$ in Equation (3.7) is $(κ - p_{*}) / (1 - p_{*}) = (0.5 - 0.151) / (1 - 0.151) = 0.41$ . Therefore, from Equation (3.6); the predicted marginal median is $Q_{Y} (0.5 ∣ x_{*}) = Q_{Y} (0.41 ∣ x_{*}, Y > 0)$ . The computation entails the following steps: (i) fit model (3.2) for $τ = 0.41$ ; (ii) apply equation (3.3); and (iii) multiply by 3 to recover the original scale. It turns out that the predicted marginal median of gained credits for the baseline student is $27$ . The same procedure can be used to compute the interquartile prediction interval ( $κ = 0.25$ and $κ = 0.75$ ). Given that $τ_{0.25, p_{*}} = 0.12$ and $τ_{0.75, p_{*}} = 0.71$ , it turns out that the interquartile prediction interval of gained credits for the baseline student is $[12, 42]$ .

In summary, the two-part model shows that the pre-enrolment test adds some information to predict student performance. In particular, students with a higher score on Reading have a higher probability of gaining at least one credit, whereas students with a higher score on Math on average gain a higher number of credits during the first year. Thus, students with difficulties in Reading have lower chances to start up their university career, while students with difficulties in Math tend to proceed slowly in the first year, likely for problems encountered in quantitative disciplines such as Math, Microeconomics and Statistics.

5 Concluding remarks

The paper presented an analysis of university freshmen performance based on a regression chain graph with the main purpose of evaluating the ability of the pre-enrolment assessment test to predict gained credits at the end of the first year.

The regression chain graph approach allowed us to visualize the assumptions on the ordering of the variables and to easily communicate the results of the analysis. Variables have been collected in three blocks: pre-test student characteristics, test scores, and gained credits. The estimated chain graph showed the decomposition of the effects of pre-test variables into direct effects and indirect effects mediated by test scores.

The number of gained credits has been modelled by a two-part model with a logit specification for the zeros and a quantile regression for positive counts. This specification allowed us to separately model the probability of zero credits and the positive part of the distribution of credits, explicitly taking into account the discrete nature of gained credits and avoiding distributional assumptions. The modelling strategy proved to be simple and effective, thus it should be considered also for other types of applications involving zero-inflated count data.

Our analysis exploited the approach of Machado and Santos Silva (2005) to quantile regression for counts, which involves adding uniform random noise (jittering). This noise induces a perturbation, which is proportionally larger for small counts. However, in our application the effect on the estimated quantiles is likely to be negligible, since the estimator is averaged over $1000$ replicates, and the lowest considered quantile is the $10$ th of the distribution of positive counts. Nonetheless, future research should investigate the performances of alternative approaches avoiding jittering, such as Bayesian density regression (Dunson et al. 2007; Chanialidis et al. 2014).

The analysis on gained university credits confirmed the predictive role of background characteristics such as the high school grade and the regularity of the school career. Moreover, the analysis showed that the pre-enrolment assessment test designed by the School of Economics of the University of Florence gives additional information. Thus, the test results can be effectively added to background characteristics to yield valuable indications for student tutoring: in particular, a low Reading score is related to a difficult start-up, while a low Math score is related to a slow progression.

Footnotes

Acknowledgments

The research has been supported by the Italian government's project Futuro in Ricerca 2012 entitled Mixture and latent variable models for causal inference and analysis of socio-economic data (RBFR12SHVV).

References

Adelfio

Boscaino

Capursi

(2014) A new indicator for higher education student performance. Higher Education , 68, 653–668.

Bianconcini

Cagnone

(2012) A general multivariate latent growth model with applications to student achievement. Journal of Educational and Behavioral Statistics , 37, 339–364.

Birch

Miller

(2006) Student Outcomes At University In Australia: A Quantile Regression Approach. Australian Economic Papers , 45, 1–17.

Birch

Miller

(2007) The influence of type of high school attended on university performance. Australian Economic Papers , 46, 1–17.

Böhning

Kuhnert

(2006) Equivalence of truncated count mixture distributions and mixtures of truncated count distributions. Biometrics , 62, 1207–1215.

Booth

Kee

(2009) Intergenerational transmission of fertility patterns, Oxford Bulletin of Economics and Statistics , 71, 183–208.

Chanialidis

Evers

Neocleous

(2014) Bayesian density regression for count data. arXiv :1406.1882. Available at http://arxiv.org/pdf/1406.1882v1.pdf

Chowdhury

Mallik

(2012) How important are introductory subjects in advanced economics studies? Economic Papers: A journal of Applied Economics and Policy , 31, 255–264.

Cox

Wermuth

(1996) Multivariate Dependencies: Models, Analysis and Interpretation . London: Chapman & Hall.

10.

Dayton

Macready

(1988) Concomitant-variable latent-class models. Journal of the American Statistical Association , 83, 173–178.

11.

Dancer

Fiebig

(2004) Modelling students at risk. Australian Economic Papers , 43, 158–173.

12.

Davino

Furno

Vistocco

(2014) Quantile Regression. Theory and Applications. Chichester, UK: John Wiley & Sons Ltd.

13.

Dunson

Pillai

Park

J-H

(2007) Bayesian density regression. Journal of the Royal Statistical Society: Series B , 69, 163–183.

14.

Grilli

Rampichini

Varriale

(2015) Binomial mixture modelling of university credits. Communications in Statistics—Theory and Methods . Advanced online publication. DOI: 10.1080/03610926.2013.804565

15.

Hoefer

Gould

(2000) Assessment of admission criteria for predicting students’ academic performance in graduate business programs. Journal of Education for Business , 75, 225–229.

16.

Kneib

(2013) Beyond mean regression (with discussion). Statistical Modelling , 13, 275–303.

17.

Koenker

(2005) Quantile Regression . Cambridge: Cambridge University Press.

18.

Koenker

Machado

(1999) Goodness of fit and related inference processes for quantile regression. Journal of the American Statistical Association , 94, 1296–1310.

19.

Lee

Neocleous

(2010) Bayesian quantile regression for count data with application to environmental epidemiology. Applied Statistics , 59, 905–920.

20.

Machado

JAF

Santos

Silva JMC

(2005) Quantiles for counts. Journal of the American Statistical Association , 100, 1226–1237.

21.

Mallik

Lodewijks

(2010) Student performance in a large first year economics subject: Which variables are significant? Economic Papers: A Journal of Applied Economics and Policy , 29, 80–86.

22.

Maree

Pretorius

Eiselen

(2003) Predicting success among first-year engineering students at the rand afrikaans university. Psychological Reports , 93, 399–409.

23.

Mills

Heyworth

Rosenwax

Carr

Rosenberg

(2009) Factors associated with the academic success of first year Health Science students. Advances in Health Science Education , 14, 205–217.

24.

Miranda

(2008) Planned fertility and family background: A quantile regression for counts analysis. Journal of Population Economics , 21, 67–81.

25.

Moreira

Barros

(2010) Double health insurance coverage and health care utilisation: Evidence from quantile regression. Health Economics , 19, 1075–1092.

26.

Mullahy

(1986) Specification and testing of some modified count data models. Journal of Econometrics , 3, 341–365.

27.

Murphy

Papanicolaou

McDowell

(2001) Entrance score and performance: A three year study of success. Journal of Institutional Research , 10, 32–49.

28.

Murray-Harvey

(1993) Identifying characteristics of successful tertiary students using path analysis. Australian Educational Researcher , 20, 63–81.

29.

Qin

Reyes

(2011) Conditional quantile analysis for crash count data. Journal of Transportation Engineering , 137, 601–607.

30.

Saffari

Adnan

Greene

(2013) Investigating the impact of excess zeros on hurdle-generalized Poisson regression model with right censored count data. Statistica Neerlandica , 67, 67–80.

31.

Smith

Naylor

(2005) Schooling effects on subsequent university performance: Evidence for the UK University Population. Economics of Education Review , 24, 549–562.

32.

StataCorp (2013) Stata: Release 13. Statistical Software. College Station, TX: StataCorp LP.

33.

Wedman

(1994) The Swedish Scholastic Aptitude Test: Development, use, and research. Educational Measurement: Issues and Practice , 13, 5–11.

34.

Wermuth

Sadeghi

(2012) Sequences of regressions and their independences. Test , 21, 215–252.

35.

Win

Miller

(2005) The effects of individual and school factors on university students' academic performance. The Australian Economic Review , 38, 1–18.

36.

Winkelmann

(2006) Reforming health care: Evidence from quantile regressions for counts. Journal of Health Economics , 25, 131–145.

Statistical modelling of gained university credits to evaluate the role of pre-enrolment assessment tests: An approach based on quantile regression for counts

Abstract

Keywords

1 Introduction

2 Data and preliminary analysis

Test results and enrolment decisions for students participating to the pre-enrolment assessment test in September 2008 (students with complete data). School of Economics, University of Florence.

Gained credits after one year by pre-enrolment test result ’ academic year 2008/2009, School of Economics, University of Florence

3.1 Regression chain graph

3.2 Two-part model: Logit specification for the zeros and quantile regression for positive counts

4.1 Pre-enrolment assessment test scores

Table 3

Multivariate linear regression of pre-enrolment assessment test scores (middle block of Figure 3) on background characteristics (right-most block). Parameter estimates (standard errors in parenthesis, coefficients with p -value < 0.05 printed in italics).

Table 4

Footnotes

Acknowledgments

References

Multivariate linear regression of pre-enrolment assessment test scores (middle block of Figure 3) on background characteristics (right-most block). Parameter estimates (standard errors in parenthesis, coefficients with $p$ -value $< 0.05$ printed in italics).