Eta- and partial eta-squared in L2 research: A cautionary review and guide to more appropriate usage

Abstract

Eta-squared (η²) and partial eta-squared (η_p²) are effect sizes that express the amount of variance accounted for by one or more independent variables. These indices are generally used in conjunction with ANOVA, the most commonly used statistical test in second language (L2) research (Plonsky, 2013). Consequently, it is critical that these effect sizes are applied and interpreted appropriately. The present study examined the use of these two effect sizes in L2 research. We begin by outlining the statistical and conceptual foundation of and distinction between η² and η_p². We then review the use of these indices in a sample of published L2 research (K = 156). Among other results, we show that η_p² values are frequently being mislabeled as η². We interpret and discuss potential causes and consequences related to the confusion surrounding these related but distinct indices. Within the context of reform efforts in quantitative L2 research, the current study seeks to respond to the recent, pointed calls for improving study quality (Plonsky, 2014) and statistical literacy (Loewen et al., 2014) in the field.

Keywords

effect size eta squared partial eta squared statistics quantitative L2 research

I Introduction

It has been almost three decades since Cohen (1988) wisely noted that ‘a moment’s thought suggests that it [effect size] is, after all, what science is all about’ (p. 532). With this position in mind, some have gone as far as to argue that failing to appropriately report estimates of effect sizes amounts to ‘a kind of withholding of evidence’ (Grissom and Kim, 2012: 9). In the case of second language (L2) research, however, effect sizes are still a relatively novel concept. Historically, the field has relied very heavily on statistical significance and null hypothesis significance testing (p values; see Norris, 2015; Plonsky 2015). It is only in the last decade or so that we have seen a shift in favor of effect sizes and practical significance, which can be attributed both to influential advocates (e.g. Norris and Ortega, 2000; Plonsky and Oswald, 2014) and journal editors. We know of at least eight L2 journals that now require effect sizes to be included in reports of quantitative research: Foreign Language Annals, Language Learning, Language Learning and Technology, Language Testing, Modern Language Journal, Second Language Research, Studies in Second Language Acquisition, and TESOL Quarterly.

Two of the most commonly employed effect sizes are eta-squared (η²), and partial eta-squared (η_p²), which are used in conjunction with ANOVA and its variants. We have chosen, therefore, to examine these two effect sizes in terms of how they are reported and interpreted in L2 research. We are concerned that, in an era of point-and-click analyses (see discussion in Mizumoto and Plonsky, 2016), choices regarding effect sizes and other statistical results may be made based on program defaults rather than on an accurate understanding of the data. This is particularly likely to occur in the case of effect sizes, which, despite their presence in published L2 research, are not generally well understood. The result of an overreliance on statistical packages together with the relative lack of detailed knowledge about effect sizes carries the risk of erroneous reporting, mislabeling, and faulty interpretations.

The present study builds on the momentum surrounding methodological reform in applied linguistics, including concerns expressed in recent years over, for example, study quality (Plonsky, 2013, 2014) and statistical literacy (Loewen et al., 2014). Such discourse responds to repeated calls for examining how ‘APPROPRIATELY’ (e.g. Lazaraton, 2009: 415; emphasis in original) different statistical concepts are being employed.

II Eta squared and partial eta squared as applied in ANOVA models

Although L2 researchers often report effect sizes such as eta-squared (η²), such values are rarely accompanied by much in the way of interpretation (Plonsky and Oswald, 2014). One reason for this is that there appears to be a good deal of confusion surrounding the terminology of ‘proportion of variance’ (Grissom and Kim, 2012: 181) effect sizes. Therefore, for the purposes of clarity, some conceptual explanation of what these indices express is warranted.

Consider, for example, a study wherein the researcher was interested in analysing the effect of an experimental treatment across conditions (e.g. condition 1, condition 2, control). Conceptually, the focus of analysis is on group differences as regards the dependent variable (e.g. usually a measure of L2 knowledge or learning). Statistically speaking, scores on the dependent variable scores also contain the amount and source of variance caused by treatment effects (Thompson, 2006).

Proportion of variance effect sizes in the η² family partition the amount of total variation in the dependent variable (DV; e.g. knowledge as measured) to determine how much of the variation is separately accounted for or explained by each independent variable (i.e. explained sum of squares or SOS). Also taken into account by the η² family is how much of the DV variation is left unexplained (i.e. unexplained or error SOS). Thus, total variation in the DV can be described in terms of explained and unexplained variance.

We suspect that, despite the now-widespread reporting of eta-squared (η²), many L2 researchers may not be aware of the differences among its variants, most notably partial eta-squared (η_p²). Further complicating this matter is the mislabeling of η² and η_p² by certain early versions of SPSS, the most frequently used statistical software package in L2 research (Loewen et al., 2014). The likelihood of this error in the context of L2 research is supported by evidence presented in other fields. Levine and Hullett (2002) and Pierce et al. (2004) found widespread misreporting and misinterpretation of η² and η_p² in published studies in communication and psychology, respectively. Both studies also cite the mislabeling of η_p² values as η² in early versions of SPSS.

In the two sections that follow, we provide a brief overview of these two important effect size indices. We also illustrate the points being made with heuristic examples.

1 Classical eta-squared

Imagine an intervention study in which four treatment conditions are compared on a single dependent variable. To examine the relationship of interest here, we would likely use a one-way ANOVA. The effect size in this case, η² (also called the squared correlation ratio), is computed using Kerlinger’s (1964) classical formula (p. 203) as:

η^{2} = \frac{{SOS}_{A}}{{SOS}_{TOTAL}} .

(1)

Note that in one-way designs, there is only one independent source (SOS_A; treatment) of variance to explain some portion of the total variation in the dependent variable (SOS_TOTAL; L2 knowledge). The numerator of the effect size estimator then represents variability that is attributable to the only independent variable we have (e.g. treatment condition). Therefore, an η² of, say, 0.35 (or 35%), indicates that we can account for 35% of the total variation in L2 knowledge as measured. The rest of the total SOS remains unexplained (i.e. SOS_Error = 65%), and may be due to individual differences, measurement error, or any number of other factors.

2 Partial eta-squared

One-way designs can certainly be found in L2 research. However, designs with multiple independent variables are likely much more common due to the multivariate nature of L2 learning, knowledge, use, and so forth (Brown, 2015). In such cases, the conceptual approach embodied by η² can be extended to apply to multi-way or factorial ANOVA. However, we now may have multiple sources of independent effects leading to a distinction that must be drawn between the classical η² and partial η² (e.g. Bakeman, 2005; Richardson, 2011).

Building on the example from above, imagine a 3 × 4 design in which proficiency level (with 3 levels) and treatment condition (with 4 levels) are jointly examined to explain the variation in learners’ (N = 120) scores on a subsequent grammar test (i.e. dependent variable). Table 1 presents the hypothetical results for this two-way design.

Table 1.

Hypothetical results of a fixed-effects 3 × 4 ANOVA (n = 120).

Source	SOS	df	MS	F _obtained	p _obtained	η²	η_p²	Inflation%
Treatment	80	3	26.67	57.60	2.57E-22	0.39	0.62	57.69%
Proficiency	70	2	35.00	75.60	2.94E-21	0.34	0.58	70.83%
Treatment × Proficiency	5	6	0.83	1.80	0.106	0.02	0.09	272.73%
Error	50	108	0.46
Total	205	119	1.72

Note. Eta-squared values and their corresponding partial eta-squared values appear in bold. Inflation% = (η_p² _– η²) / η² × 100, and this shows how different η_p² and η² can be in this two-way design.

If we want to quantify any of the independent variables’ contributions to the variation observed in post-test scores, we can do so by invoking the classical η² in each case. But a different form of η² may be computed as well. Cohen (1965) implicitly introduced a new variant of η² (now often denoted by η_p²) in multi-way designs which was similar to the classical η² formula with ‘other nonerror sources of variance being partialled out [from the denominator]’ (p. 105). Later, Cohen (1973) emphasized that this new variant is distinct from the classical η² and may be called ‘partial η²’ (p. 108; italics in original). Thus, in multi-way designs, the term partial refers to removing all other possible sources of effect in the design except the one of interest in the denominator of equation (1) and the error/unexplained variance.

In our two-way design, which includes two main effects and one interaction effect, partial eta-squared (η_p²) for treatment condition (A) can be computed as:

η_{p}^{2} = \frac{{SOS}_{A}}{{SOS}_{A} {+ SOS}_{Error}} .

(2)

Thus, η_p² = 80 / (80 + 50) = .62 [90% CI: .494, .659]. Likewise, η_p² for the effect of proficiency level (SOS_B) can be computed in a similar fashion with other independent sources (i.e. treatment, and the treatment × proficiency interaction) removed from the denominator:

η_{p}^{2} = \frac{{SOS}_{B}}{{SOS}_{B} {+ SOS}_{Error}} .

(3)

Therefore, for proficiency, η_p² = 70 / (70 + 50) = .58 [90% CI: .458, .632]. And for the interaction effect (SOS_A*B), we will have:

η_{p}^{2} = \frac{{SOS}_{A * B}}{{SOS}_{A * B} {+ SOS}_{Error}} .

(4)

Thus, regarding the interaction effect, η_p² = 5 / (5 + 50) = .09 [90% CI: .000, .133]. Note that because in one-way designs there is only one source of effect, no difference in the denominator of the classical and partial eta-squared formulas exists. In other words, because there are no other effects to be partialled out, eta-squared and partial eta squared are identical in one-way designs. However, as shown in Table 1, for our two-way design, η_p² values are invariably larger – often much larger – than their η² counterparts. This occurs because the partial η² formula is partialling out the other nonerror terms (i.e. proficiency: SOS_B and proficiency × treatment: SOS_A*B) from the denominator for each effect, thus augmenting the outcome (see Grissom and Kim, 2012; Pedhazur, 1997). It is therefore critical that care be taken to report and interpret these indices appropriately.

III Assumptions and rationale of the present study

Having laid out the conceptual and statistical reasoning behind η² and η_p², the present study seeks to examine the use and interpretation of these two indices. The study is motivated by several factors that, in coordination, may create conditions that are counterproductive for the field’s progress. First, although effect sizes are regularly reported, they are not often interpreted and even less often are they interpreted meaningfully (see Larson-Hall and Plonsky, 2015; Plonsky and Oswald, 2014). Second, ANOVA designs are exceedingly common and therefore highly influential in L2 research. The family of effect sizes for this set of techniques is particularly prone to error, however, due to very similar and often ambiguous or even misleading labels, as described in Section II. This problem, observed in other social sciences, is only compounded by a lack of general statistical literacy in the field (Loewen et al., 2014). With these issues in mind, we anticipate that erroneous reporting of these frequently used effect sizes is likely to occur in L2 research. Therefore, in this study we examine the use of η² and η_p² as a means to improve future research practices in the field. With these concerns in mind, the present study sought to answer the following question: To what extent does published L2 research demonstrate erroneous reporting of η_p² as representing η²?

IV Method

1 Journal selection and search criteria

In order to collect a representative sample of L2 research, we first consulted previous surveys of L2 research practices (e.g. Egbert, 2007; Gass, 2009; Lazaraton, 2005; Plonsky, 2013) as well as L2 research methods textbooks providing various L2 journals’ descriptions (e.g. Perry, 2011: Appendix C) and other documents discussing L2 journals (VanPatten and Williams, 2002). There is, of course, no consensus as to which journals are most prominent or influential in the field. In the end, we decided to survey the following five journals: Applied Linguistics, Language Learning, Language Teaching Research, Modern Language Journal, and System. This sample is by no means exhaustive but we would argue that it does provide generally representative view of quantitative L2 research.

In order to gain a current view of this domain, we limited our search to studies published from 2005 to 2015. In line with previous reviews (e.g. Gass, 2009), we excluded from consideration forums, short reports, commentaries, review articles, and book reviews. We then examined all studies that included variants of multi-way ANOVA (repeated measures, factorial, ANCOVA- henceforth, multi-way ANOVA studies). The total sample consisted of 156 studies. Our goal to include multi-way designs was because, as discussed in Section III, in these studies η² and η_p² lead to different results. Thus in these designs, mistakenly reporting η_p² as η² presents a distorted view of the results. Figure 1 shows the distribution of the sampled studies across the period 2005 through 2015.

Figure 1.

Distribution of multi-way ANOVA studies over time.

2 Procedures and analyses

In order to address our research question, following best practices in synthetic research (see Plonsky and Oswald, 2015), each study in the sample was systematically coded for the design type (repeated measures, factorial, ANCOVA), model (fixed-, random-, mixed-effects), and sampling unit distribution (balanced, unbalanced) applied. We also extracted from each study F values, degrees of freedom, and descriptive statistics (Mean, SD). We then conducted secondary analyses by using any or a combination of the following three methods, as appropriate.

First, in line with previous studies that have examined the reporting and interpreting of η² effect sizes (Levine and Hullett, 2002; Pierce et al., 2004), we computed the sum of the η² values for every multi-way design in papers that reported them (i.e. ∑η²_max limit check). When the sum for a multi-way design exceeded 1 (or equivalently 100%), the values were assumed to be representing instances of η_p² labeled erroneously as η². This method was applied to all 156 multi-way studies we collected. Using this technique, we found 17 studies with this type of erroneous reporting.

Second, we applied Cohen’s (1973) partial eta-squared meta-analytic equation which is computed as:

η_{p}^{2} = \frac{d f_{A} (F_{A})}{d f_{A} (F_{A}) + d f_{Error}} .

(5)

Equation 5 was used to evaluate if the values reported and labeled as η² in reality were η_p². Being ‘purely algebraic [i.e. insensitive to the design type and model]’ (Cohen, 1973: 107), this equation was applied to all designs types (e.g. repeated measures, ANCOVA) and models of multi-way analysis (i.e. fixed-, mixed-, and random-effects). If the answer from the manual calculations matched (within rounding error) those in the primary published report, we concluded that η_p² values were mistakenly presented as η². Also, when possible (i.e. when the design was fixed-effects with all relevant error and effect terms reported), we used Haase’s (1983) meta-analytic equation which for a two-way design is computed as:

η^{2} = \frac{d f_{A} (F_{A})}{d f_{A} (F_{A}) + d f_{B} (F_{B}) + d f_{A * B} (F_{A * B}) + d f_{Error}} .

(6)

Equation 6 was used to correctly compute η² values in fixed-effects multi-way designs. For the second method, when no match was found between our calculation of η² or η_p² and those reported in the original paper, the analysis in question was excluded from our study. The second method was also uniformly applied to all 156 multi-way studies and resulted in the identification of an additional 12 studies with erroneous reporting of η², which also confirmed and extended the results of method 1.

A third method for identifying erroneous reporting was applied when full summary tables (i.e. with all sum of squares, dfs, F values made available) were reported. Using these data, we separately computed the η² and η_p² values to compare them with the values appearing in the original published studies. The third method led to the identification of 5 additional studies which had inaccurately presented η_p² effect size as representing η².

V Results

The present study was intended to determine the extent of erroneous reporting of η_p² as representing η² in quantitative L2 research published between 2005 and 2015. Previous studies in other fields (Levine and Hullett, 2002; Pierce et al., 2004) were only able to show that η_p² values were mistakenly reported as η² if the sum of η² values in a multi-way design exceeded 1 or 100% (method 1). As noted in the Section IV, we have sought to gain a more comprehensive view of this practice by employing additional equations (method 2) and in some cases directly computing η_p² and η² effect sizes from summary tables (method 3).

Table 2 presents the sum of η² values (i.e. ∑η²; method 1) for the studies in our sample along with relevant data retrieved from these studies, and other features specific to the methods used for our secondary calculations. All 34 studies in Table 2, which we have anonymized, have incorrectly reported η_p² as representing η² values.

Table 2.

Summary of 34 studies erroneously reporting η_p² as representing η².

Number	Study ID (year)	Design	Statistics reported	Software used	Main effect η² values reported	Interactioneffect η² values reported	∑η² valuesComputed %
1	Study 87 (2011) Main Analysis	2*2 ANOVA	F, dfs, SD, Mean	n.r.	n.r.	.12	n.a.^a
2	Study 101 (2012) Main Analysis	2*3 ANCOVA	full summary table	SPSSversion 18	.452.17	n.r.	n.a.
3	Study 147 (2015) Main Analysis	4*4 RM^b ANOVA	F, dfs, SD, Mean	n.r.	.93	.90	183%
4	Study 39 (2007) Analysis 1	2*3 RM ANOVA	F, dfs, SD, Mean	n.r.	.754	.193	n.a.
5	Study 43 (2007) Analysis 2	2*3 RM ANOVA	F, dfs, SD, Mean	n.r.	.820.123	n.r.	n.a.
6	Study 41 (2007) Analysis 3	2*3 RM ANOVA	F, dfs, SD, Mean	n.r.	.831.654	n.r.	148.5%
7	Study 127 (2014) Analysis 1	2*3 RM ANOVA	F, dfs, SD, Mean	SPSS version 16.0	.73.40	.50	163%
8	Study 128 (2014)Analysis 2	2*3 RM ANOVA	F, dfs, SD, Mean	n.r.	.72.29	.54	155%
9	Study 14 (2005) Analysis 1	2*3 RM ANOVA	full summary table	n.r.	.29.66	.59	154%
10	Study 16 (2005) Analysis 2	2*3 RM ANOVA	full summary table	n.r.	.30.46	.56	132%
11	Study 11 (2009) Main Analysis	2*2 ANOVA	full summary table	SPSS	.06.24	.00	n.a.
12	Study 9 (2005) Main Analysis	2*6 ANOVA	full summary table	n.r.	.013.105	.009	n.a.
13	Study 31 (2007) Main Analysis	One-Way^c ANCOVA	F, dfs, SD, Mean	n.r.	.69.52	.12	133%
14	Study 114 (2013) Analysis 1	2*3 RM ANOVA	F, dfs, SD, Mean	n.r.	.63.54	.62	179%
15	Study 117 (2013) Analysis 2	2*3 RM ANOVA	F, dfs, SD, Mean	n.r.	.67.55	.52	174%
16	Study 126 (2014) Experiment 3	2*2 RM ANOVA	F, dfs, SD, Mean	n.r.	.19.37	n.r.	n.a.
17	Study 41 (2007) Main Analysis	2*3 ANOVA	F, dfs, SD, Mean	n.r.	.28.10	.06	n.a.
18	Study 30 (2007) Main Analysis	2*3 ANOVA	F, dfs, SD, Mean	n.r.	.048.024	n.r.	n.a.
19	Study 21 (2006) Analysis 1	3*2 RM ANOVA	F, dfs	n.r.	.99	.76	175%
20	Study 18 (2006) Analysis 2	2*2 RM ANOVA	F, dfs	n.r.	.68.43	.40	151%
21	Study 23 (2006) Analysis 3	2*3 RM ANOVA	F, dfs	n.r.	.97	.38	135%
22	Study 12 (2005) Experiment 1	3*4 RM ANOVA	F, dfs, SD, Mean	n.r.	n.r.	.373	n.a.
23	Study 15 (2005) Main Analysis	4*5 RM ANOVA	F, dfs, SD, Mean	n.r.	n.r.	.396	n.a.
24	Study 47 (2008) Main Analysis	2*2 ANOVA	F, dfs, SD, Mean	n.r.	.620.473	n.r.	109.3%
25	Study 51 (2008) Main Analysis	223 RM ANOVA	F, dfs, SD, Mean	n.r.	.332.803	n.r.	113.5%
26	Study 56 (2008) Main Analysis	2*2 RM ANOVA	F, dfs, SD, Mean	n.r.	.448.798	n.r.	124.6%
27	Study 44 (2008) Main Analysis	2*3 RM ANOVA	F, dfs, SD, Mean	n.r.	.31.21	n.r.	n.a.
28	Study 41 (2008) Main Analysis	2*3 RM ANOVA	F, dfs, SD, Mean	n.r.	.25.05	.13	n.a.
29	Study 39 (2008) Main Analysis	2*3 RM ANOVA	F, dfs, SD, Mean	n.r.	.13	.08	n.a.
30	Study 76 (2010) Main Analysis	2*2 ANCOVA	F, dfs, SD, Mean	SPSS GLM^d	.30 .44.34	.01	109%
31	Study 84 (2010) Main Analysis	2*4 RM ANOVA	F, dfs, SD, Mean	n.r.	.62	n.r	n.a.
32	Study 137 (2014) Main Analysis	3*3 RM ANOVA	F, dfs, SD, Mean	n.r.	.346	.244	n.a.
33	Study 152 (2015) Main Analysis	222 RM ANOVA	F, dfs, SD, Mean	n.r.	.15 .13.01	.01 .13.03	n.a.
34	Study 143 (2015) Main Analysis	4*2 ANOVA	F, dfs, SD, Mean	n.r.	.97	.83	180%

Notes. ‘n.r.’ = not reported; ‘n.a.’ = not applicable; ^a Not applicable: either summary table was presented (method 3) or equation’s 5 outcome matched that in the original report (method 2). ^b RM = Repeated measures. ^c One-way ANCOVA’s summary table terms are algebraically similar to those of two-way ANOVA. ^d GLM = General linear model.

As can be seen in Table 2, mistakenly reporting partial eta squared as representing eta squared is not uncommon in published quantitative L2 research. More precisely, this error occurred in 34 of the 156 studies in our sample, or 22%. Figure 2 shows the breakdown of the misreported studies in the 156 multi-way ANOVA studies published between 2005 and 2015. In Figure 2 the proportion of studies that misreported eta squared to the total multi-way ANOVAs in each year is represented by the cross-hatched columns. One important observation is that inaccurately presenting partial eta squared as representing eta squared is still present in recent L2 research. This might be due to that fact that multi-way ANOVA and its variants are frequently and increasingly employed to answer different substantive questions in L2 research (Plonsky, 2014).

Figure 2.

Multi-way ANOVA studies that presented partial eta squared as representing eta squared (crosshatched bars).

VI Discussion

The confusion between η² and η_p², found in the present study to be fairly widespread in quantitative L2 research, can lead to, at least four problems that affect interpretations of findings to varying degrees. Some actual examples here may convey the general tenor of these problems.

First, while reporting η_p² values in place of η² values does not change the rank ordering of effects within a single study (see Table 1 for example), η_p² and η² values use different denominators in their formulas. Put succinctly, the base of each η_p² value differs in nature from others in the same design because η_p² values do not share a common base (i.e. denominator). Therefore, cross-effect comparison of η_p² values is not meaningful (see Olejnik and Algina, 2000: 268; Pedhazur, 1997: 507). Interpretations are especially problematic, however, when η_p² values (often expressed in percentages), either correctly labeled as η_p² or erroneously presented as η², are used to indicate that they have explained a certain amount of total variation in the dependent variable as exemplified in Table 3.

Table 3.

Misinterpreting η_p² as proportion of total variance.

Study ID	η_p² mislabeled as η²	Author(s) interpretation
Study 21 (2006)	7%69%19%	In all cases, η_p² values taken to account for total variation
Study 47 (2008)	.177	The effect of interaction explained 17.7% of variance in the dependent variable
Study 15 (2005)	.105	About 11% of variability explained in the dependent variable

Second, η² values are often upwardly biased (an issue not discussed here, but see Grissom and Kim 2012), and particularly so when the effects are based on small samples, which is often the case in L2 research (Plonsky, 2013). Therefore, erroneously reporting η_p² as η² in a multi-way study can inflate an already-biased η² effect size even further. Pedhazur (1997) warned that ‘[b]ecause partial η² tends to be larger than η², I am afraid that novices will be [more] inclined to use it’ (p. 509). Thus, it is critical not to look at the effect sizes in a single study ‘from a bigger-is-better standpoint’ (Bakeman, 2005: 380). We were able to estimate the inflation due to mistaken reporting in some of the studies by using summary table information or by applying equation 5 for η_p² and equation 6, when applicable, for η². The inflation percentage may vary from one study to the next. Table 4 presents some examples along with the amount of inflation observed.

Table 4.

Inflation resulting from mistakenly presenting η_p² for η².

Study ID	Effect	η_p² mislabeled as η²	η²	Inflation%^a
Study 101 (2012)	main	.452^b	.127	72%
Study 13 (2005)	interaction	.59	.32	46%
Study 7 (2005)	main	.46	.27	41%
Study 64 (2009)	main	.06	.04	33%

Notes. ^a Inflation% = (η_p² _– η²) / η_p² × 100; ^b As reported by the original authors to 3 decimal places.

Third, for one-way ANOVA designs, reporting η_p² instead of η² seems to be following a logic of redundancy. And perhaps it further ‘indicates that researchers are either very passionate about unnecessary subscript letters [i.e. p for partial], or rely too much on the effect sizes as they are provided by statistical software packages’ (Lakens, 2013: 1). Curiously, we also found several instances of reporting η_p² values in one-way ANOVA. As discussed, the case of presenting η_p² in one-way ANOVA represents at least the nominal confusion of regarding η_p² as η²; in one-way designs there are no other effects to be partialled out.

Fourth, Cohen’s (1988) benchmarks for interpreting effect sizes are arbitrary and should not be applied in L2 research or elsewhere (Cohen, 1988). Even so, the frequently used Cohen’s (1988: 283) proportion of variance effect size cut-off points (i.e. small = .0099; medium = .0588; large = .1379) may only relate to ‘partial eta squared’ values and not to those of ‘eta squared’ in multi-way designs (see Richardson, 2011). Thus, employing Cohen’s benchmarks (error 1) and erroneously applying them to eta squared (error 2) creates a ‘double-error’ situation. For example, if η² is erroneously chosen to be benchmarked against Cohen’s (1988) cut-offs, one may interpret the magnitude of a given effect as ‘small’. However, for the same effect, if ‘η_p²’ is compared against Cohen’s benchmarks it may be interpreted as ‘large’.

It is useful at this point to recall that effect sizes are descriptive statistics that leave decision about the importance of an observed effect(s) to the community of researchers in any specific domain given (1) their understanding of the phenomenon they study, (2) prior studies in the same domain, (3) the predictions of theory, (4) practical implications, if any, (5) the design and instrumentation from which the effect was derived, and so forth. Looking ahead, we recommend that Cohen’s conventions be dropped in favor of researchers’ direct and explicit comparison of the effects in related literature as well as these and other considerations (see Plonsky and Oswald, 2014; Thompson, 2006).

The findings of this study, which reveal somewhat widespread misuse of a common statistic, prompt us to consider why this problem exists (and persists). One explanation might be the lack of appropriate reference material. In examining 14 texts on L2 research methods at our immediate disposal, the materials available to L2 researchers do not appear to address adequately the distinction between η² and η_p². For example, Larson-Hall (2012), a brief and generally quite useful chapter-length overview of statistics used in L2 research, briefly commented that ‘Effect sizes for ANOVA results are also of the same type as the correlation but use the Greek letter eta (η) and are called eta-squared or partial eta-squared’ (p. 249; italics added). However, no clear distinction is made between η² and η_p². In Phakiti (2014), another generally strong reference, no clear distinction between the use of eta-squared and partial eta-squared is made (see p. 205 and pp. 283–300).

Other L2 research methods textbooks that we reviewed likewise lacked sufficient discussion of the difference between the η² and η_p² effect size measures. Dörnyei (2007), for example, first provided a brief account of eta squared followed by presenting the formula for computing it. However, the only passing reference to η_p² was made later in the context of ANCOVA: ‘The good news about SPSS output is that next to the significance value we find the “partial eta squared” index, which is an effect size’ (p. 223; italics added). A discussion on η² and η_p², however, did appear in Larson-Hall (2016) where a number of the same considerations addressed here as regards these two variants were usefully and clearly explained (see p. 149).

We suggest that future texts that discuss ANOVA explain all the terms that appear in a full table of summary results (see Thompson, 2006). It would be particularly useful if such a rubric would detail how all terms in the ANOVA results, including but not limited to η² and η_p², are (1) computed and (2) related to each other. When a reader is able to ascertain the relationship between all the terms in an ANOVA summary table, the distinction between η² and η_p² becomes more meaningful. As a final note, we would add that in many studies using ANOVA and its variants, researchers will want to go beyond the initial analysis to often perform pair-wise comparisons of groups’ mean scores. In such cases, it is not sufficient to report only the effect sizes for the ANOVA main and interaction effects; rather, an eta-squared (equivalent to a point-biserial correlation; r_pb) or a standardized mean difference effect size such as Cohen’s d for each pair-wise comparison of interest should be reported and interpreted as well.

VII Conclusions

‘[A]ny effect size that is chosen from possible alternatives should be technically [and nominally] appropriate’ (Grissom and Kim 2012: 9). Evidence we provide in this article contains numerous and recent examples of erroneous reporting of often large η_p² effect sizes in multi-way designs being misinterpreted and mislabeled as η².

The distinction we draw in this article between η² and η_p² is in no way semantic or statistical nit-picking. These effect sizes are increasingly reported throughout quantitative L2 research. They also have immense potential to inform our understanding of L2 learning and use. Clarity about these indices, their reporting, and interpretation is, in fact, critical to arrive at appropriate conclusions regarding L2 theory and practice and, at the same time, to preventing misinterpretations that compromise work in this field.

We remind readers that, like many measures of effect size and in contrast to the dichotomous result embodied by p values, an η² value provides a continuously-expressed result within a single multi-way ANOVA. Thus, reporting η_p² values alone which (1) lack comparability advantages within a study and (2) are often larger than η² values (see Section VI) may lead to erroneous interpretation. Added to the above problems is that η_p² values depend on the model of analysis (i.e. fixed, random, and mixed). That is, for a given study, if we run the data analysis under fixed-, random-, or mixed-effects models, values of η_p² for some treatment effects can change. Presenting the reasoning behind this dynamic requires knowledge of ‘Mean Squares Expectation Rubric’, which falls outside the scope of the present article (but see Thompson, 2006).

Based on these considerations, we encourage L2 researchers to compute, report and interpret, by default, (classical) η² for all ANOVA-based analyses. This approach will provide an estimate of variance accounted for that is insensitive to the model of analysis as well as comparable within a single multi-way study. However, it is also useful for researchers to report η_p² along with η² to avoid the possibility of erroneous reporting and interpretation. In addition, in multi-way designs, reporting η_p² facilitates the calculation of power for an effect and thus using the size of that effect as the basis for planning the sample size for a relevant, future research. Thus, eta- and partial eta-squared serve different purposes that legitimize the presence of both estimates in published multi-way ANOVA studies.¹ Finally, it is critical to note that reporting confidence intervals for η_p² values is both highly recommended and possible via several statistical packages. For example, an easy-to-use program is the MBESS package in the R software program (Kelley, 2015). Unfortunately, confidence intervals for eta-squared are more complex (often roughly estimated) than those for partial eta-squared, and not currently widely available.

In closing, the results of this study do not present an ideal state of statistical proficiency in L2 research. Nevertheless, we are hopeful that the field’s momentum toward methodological reform – a movement to which the present study seeks to contribute – will continue to improve L2 research and reporting practices thereby leading to a clearer understanding of language learning and use.

Footnotes

Acknowledgements

We are deeply indebted to Professor Bruce Thompson for his insightful comments on the earlier drafts of this article.

Declaration of Conflicting Interest

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

References

Bakeman

(2005) Recommended effect size statistics for repeated measures designs. Behavior Research Methods 37: 379–84.

Brown

(2015) Why bother learning advanced quantitative methods in L2 research. In: L Plonsky (ed.) Advancing quantitative methods in second language research. New York: Routledge, pp. 9–20.

Cohen

(1965) Some statistical issues in psychological research. In B. B. Wolman (Ed.), Handbook of clinical psychology. New York: McGraw-Hill, pp. 95–121.

Cohen

(1973) Eta-squared and partial eta-squared in fixed factor ANOVA designs. Educational and Psychological Measurement 33: 107–12.

Cohen

(1988) Statistical power analysis for the behavioral sciences. 2nd edition. Hillsdale, NJ: Erlbaum.

Dörnyei

(2007) Research methods in applied linguistics: Quantitative, qualitative, and mixed methodologies. Oxford: Oxford University Press.

Egbert

(2007) Quality analysis of journals in TESOL and applied linguistics. TESOL Quarterly 41: 157–71.

Gass

(2009) A historical survey of SLA research. In: Bhatia

Ritchie

(eds) The new handbook of second language acquisition. Bingley: Emerald, pp. 3–27.

Grissom

Kim

(2012) Effect sizes for research: Univariate and multivariate applications. 2nd edition. New York: Routledge.

10.

Haase

(1983) Classical and partial eta square in multifactor ANOVA designs. Educational and Psychological Measurement 43: 35–39.

11.

Kelley

(2015) MBESS (Version 4.0.0 and higher) [computer software and manual]. Available from http://cran.r-project.org (accessed December 2016).

12.

Kerlinger

(1964) Foundations of behavioral research. New York: Holt, Rinehart and Winston.

13.

Lakens

(2013) Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Frontiers in Psychology 4: 1–12.

14.

Larson-Hall

(2012) How to run statistical analyses. In: Mackey

Gass

(eds) Research methods in second language acquisition: A practical guide. Oxford: Blackwell, pp. 245–74.

15.

Larson-Hall

(2016) A guide to doing statistics in second language research using SPSS and R. 2nd edition. New York: Routledge.

16.

Larson-Hall

Plonsky

(2015) Reporting and interpreting quantitative research findings: What gets reported and recommendations for the field. Language Learning 65: supplement 1: 127–59.

17.

Lazaraton

(2005) Quantitative research methods. In: Hinkel

(ed.) Handbook of research in second language teaching and learning. Mahwah, NJ: Erlbaum, pp. 209–24.

18.

Lazaraton

(2009) The use of statistics in SLA: A response to Loewen and Gass (2009). Language Teaching 42: 415–16.

19.

Lazaraton

Riggenbach

Ediger

(1987) Forming a discipline: Applied linguists’ literacy in research methodology and statistics. TESOL Quarterly 21: 263–77.

20.

Levine

Hullett

(2002) Eta squared, partial eta squared, and misreporting of effect size in communication research. Human Communication Research 28: 612–25.

21.

Loewen

Gass

(2009) The use of statistics in L2 acquisition research. Language Teaching 42: 181–96.

22.

Loewen

Lavolette

Spino

. (2014) Statistical literacy among applied linguists and second language acquisition researchers. TESOL Quarterly 48: 360–88.

23.

Mizumoto

Plonsky

(2016) R as a lingua franca: Advantages of using R for quantitative research in applied linguistics. Applied Linguistics 37: 284–91.

24.

Norris

(2015) Statistical significance testing in second language research: Basic problems and suggestions for reform. Language Learning 65: 97–126.

25.

Norris

Ortega

(2000) Effectiveness of L2 instruction: A research synthesis and quantitative meta-analysis. Language learning 50: 417–528.

26.

Olejnik

Algina

(2000) Measures of effect size for comparative studies: Applications, interpretations, and limitations. Contemporary Educational Psychology 25: 241–86.

27.

Pedhazur

(1997) Multiple regression in behavioral research: Explanation and prediction. 3rd edition. Fort Worth, TX: Harcourt Brace.

28.

Perry

(2011) Research in applied linguistics: Becoming a discerning consumer. 2nd edition. New York: Routledge.

29.

Pierce

Block

Aguinis

(2004) Cautionary note on reporting eta-squared values from multifactor ANOVA designs. Educational and Psychological Measurement 64: 916–24.

30.

Phakiti

(2014) Research methods in linguistics: Experimental research methods in language learning. London: Bloomsbury Academic.

31.

Plonsky

(2013) Study quality in SLA: An assessment of designs, analyses, and reporting practices in quantitative L2 research. Studies in Second Language Acquisition 35: 655–87.

32.

Plonsky

(2014) Study quality in quantitative L2 research (1990–2010): A methodological synthesis and call for reform. Modern Language Journal 98: 450–70.

33.

Plonsky

(2015) Statistical power, p values, descriptive statistics, and effect sizes: A ‘back-to-basics’ approach to advancing quantitative methods in L2 research. In: Plonsky

(ed.) Advancing quantitative methods in second language research. New York: Routledge, pp. 23–45.

34.

Plonsky

Oswald

(2014) How big is ‘big’? Interpreting effect sizes in L2 research. Language Learning 64: 878–912.

35.

Plonsky

Oswald

(2015) Meta-analyzing second language research. In: Plonsky

(ed.) Advancing quantitative methods in second language research. New York: Routledge, pp. 106–28.

36.

Richardson

(2011) Eta squared and partial eta squared as measures of effect size in educational research. Educational Research Review 6: 135–47.

37.

Thompson

(2006) Foundations of behavioral statistics: An insight-based approach. New York: Guilford Press.

38.

VanPatten

Williams

(2002) Research criteria for tenure in second language acquisition: Results from a survey of the field. Unpublished manuscript, The University of Illinois at Chicago, IL, USA.