A Systematic Review of Test Component Ordering with Implications for Language Assessment

Objective Performance

Objective performance, particularly test scores, is by far the most researched dependent variable category concerning test component arrangement. The vast majority of studies analyzed some form of test scores (82/88), sometimes as one of several dependent variables. The total score was almost always a number correct/sum score, that is, not scored with item response theory (IRT), which takes item difficulty into account when computing scores. It is important to keep this limitation in mind when considering the conclusions about how, and to what degree, test ordering impacts scores.

Some studies (27/88) have also investigated other measures related to objective performance, such as item/test characteristics (e.g., Carlson & Ostrosky, 1992) or time on test/task (e.g., Ling et al., 2017). How test component ordering impacts objective performance is perhaps of greatest importance to high-stakes assessments, as any impact could change score distributions, score interpretations, and reliability estimates, with potentially serious consequences for test takers.

Perceived Performance

Some studies (21/88) examined test-taker perceptions of their performance and/or test difficulty (e.g., Barcikowski & Olsen, 1975; Laffitte, 1984), which can be unrelated to quantitative measures of performance/difficulty. Perceived performance can be operationalized as predictions of test/item scores on upcoming parts of the test, predicted scores after completing the test (also called postdictions), or qualitative descriptions of performance (e.g., “well” and “good”). Perceived difficulty can be measured immediately following the test and/or at time points during the test. Perceived performance is important to test developers because perceptions of success or failure during task engagement are known to influence effort and persistence, which in turn can affect performance outcomes (Bandura, 1997). For instance, we might expect test takers who perceive they are doing poorly during the test to exert less effort. Beyond any potential impact on objective performance, perceived performance might plausibly impact factors such as face validity and repeat testing behavior.

Test-taker Affect

We use test-taker affect to refer to temporary, context-dependent psychological states that test takers experience during and immediately following the testing experience, such as self-efficacy (e.g., Habbert & Schroeder, 2020) or emotions (e.g., Bieleke et al., 2023). Of the 26 studies to report such variables, nine measured negative emotions/experience, including stress and test anxiety, defined as anxiety due to anticipation of failure and associated negative cognitive and physiological responses (Chen, 2012). Chen posits two competing views of how anxiety impacts test performance: the monotonic view (more anxiety leads to worse performance) and the non-linear view (moderate anxiety leads to optimal performance).

For language tests specifically, test anxiety is part of the broader construct of language anxiety. As first described in Horwitz et al. (1986), language anxiety consists of three components: communication apprehension, fear of negative evaluation, and test anxiety. In numerous studies, language (including test) anxiety has been shown to produce negative academic effects (e.g., lower grades), cognitive effects (e.g., slower language processing), and social effects (e.g., reduced linguistic self-confidence; MacIntyre, 2017).

Like perceived difficulty/performance, affect is a test taker’s subjective perception of something that is not captured by their responses to test items. Research in this area thus usually involves presenting test takers with additional, non-test items to measure affect (e.g., Ahmadi et al., 2012; Plake et al., 1981) before, during, and/or after the testing experience. The difference from perceived difficulty/performance is that the focus is on the test taker’s emotional/affective state. Test-taker affect is important for test developers to consider because it can impact performance and/or sentiment about the assessment. Of particular concern is the possibility of differential impact on affect (e.g., by proficiency level), leading to sources of unfairness.

Difficulty-Based Ordering

It is unclear from the public documentation of most high-stakes language proficiency tests whether items are intentionally and systematically ordered by difficulty. For CATs, item-difficulty ordering is dictated by test-taker responses, but for fixed-form tests, it may be that E–H ordering is implicitly assumed to be best (a tendency noted over 90 years ago by Capron, 1933). Item-difficulty ordering is by far the most researched topic within the literature on test ordering (50/88), ¹ with E–H ordering the most researched item arrangement (48/88).

Of the studies comparing E–H ordering to alternative orderings in terms of impact on test scores, most (29/38) found no consistent, overall significant difference (e.g., Marso, 1970; Perlini et al., 1998; Plake & Ansorge, 1984; Şad, 2020), while a few (9/38) found a consistent benefit of E–H ordering (e.g., Anaya et al., 2022; Hodson, 1984b; Holliday & Partridge, 1979; Sax & Carr, 1962). For instance, Hodson (1984b) reported significantly higher mean scores on an A-level chemistry test with E–H versus H–E sequencing, and Sax and Carr (1962) found that an aptitude test with different item types intermixed and in E–H order yielded significantly higher scores and more items attempted than when items were grouped into subtests by content. Of note, the studies finding significant score differences in favor of E–H ordering typically considered speeded tests (6/9). In a study of untimed math exams, Gruss and Clemons (2023) found that the difficulty of the first two items did not significantly predict the overall score, noting that lack of a time limit may offset any effect of initial item difficulty. In studies by Plake and colleagues (1981, 1982), prior knowledge of the ordering arrangement did not affect scores.

Aamodt and McShane (1992) found only a small effect size of E–H versus random order (d = .11, studies n = 26, participants n = 13,428) and a larger but still modest effect size of E–H versus H–E (d = .22, studies n = 11, participants n = 1564). However, one of the studies in the meta-analysis found a significant interaction of item order and anxiety types, despite no main effect of item order (Munz & Smouse, 1968); facilitators (people who benefit from test anxiety) performed significantly higher in the E–H and random order conditions, but there were no significant group differences in the H–E condition. Chen (2012; n = 250) also found an interaction between item order and anxiety levels; however, all anxiety-level groups performed significantly better on a CAT version of the test than a H–E fixed version. ²

The potential benefit of beginning with easy items is conceptually related to the common practice in language pedagogy of starting lessons with a warm-up activity. In our pool, only two studies empirically examined difficulty ordering effects for language tests. In Soureshjani (2011), intermediate EFL learners in Iran (n = 70) completed two forms of a 40-item multiple-choice English grammar test (covering tenses, modals, gerunds/infinitives). They first completed the E–H form and 2 weeks later the H–E form, with item difficulties based on pilot data. Scores on the E–H form were significantly higher (no effect size reported). This finding of the positive impact of E–H ordering is noteworthy, given that the fixed administration order (E–H then H–E) meant any practice effects would have presumably benefited the H–E form, potentially masking the full extent of the E–H advantage. Soureshjani (2011) acknowledged this potential for practice effect, though he pointed to the potentially mitigating 2-week interval between administrations. He proposed that the benefit of the E–H ordering might stem from reduced student anxiety and improved student confidence.

In Coniam (1993), 342 intermediate-level secondary-school EFL students in Hong Kong completed an MCQ listening comprehension test and an MCQ cloze test in either E–H (n = 163) or H–E (n = 179) item ordering conditions based on pilot data. A post-test questionnaire then assessed motivation through students’ perceived test difficulty, effort expended, confidence, and enjoyment. Unlike Soureshjani (2011), Coniam found no significant difference between groups in scores for either task type or for any aspect of motivation. He suggested that for global, integrative language tasks that require sustained comprehension, the ordering of individual items by difficulty might be less impactful than for discrete-point tests.

Research has also investigated how item ordering affects perceptions of test difficulty, with mixed results. Barcikowski and Olsen (1975) and Olsen and Barickowski (1976) found that students perceived items as easier in the hard-medium-easy condition, but that these perceptions did not typically correspond to differences in test scores. Bieleke et al. (2023) also found that students (n = 208) who started a math test with a section of easy items perceived the test as more challenging than students who started with a section of difficult items. Perlini et al. (1998) noticed that students (n = 140) perceived they had performed better on a H–E psychology exam than when ordered randomly, despite no significant difference in scores. Munz and Jacobs (1971) also sequenced items on the basis of student-perceived difficulty and found that although there was no effect on test scores or on situational test-taking anxiety, the E–H sequence was regarded by examinees as easier and fairer than the random and H–E test forms. Conversely, Aamodt and McShane (1992) did not find E–H and random orders to significantly impact perceived exam difficulty. Thus, there seems to be some, but not unanimous, evidence of an item-difficulty order effect when comparing E–H and H–E.

More recent studies have explored the mechanism of the item-difficulty order effect. Weinstein and Roediger (2010, 2012) found that participants in the E–H condition postdicted higher scores, and participants in the H–E condition perceived the test as more difficult, despite no significant group difference in scores. The authors concluded that the results supported the anchoring explanation (i.e., the earliest questions anchor test-taker perception of the entire test). This position aligns with earlier work suggesting that initial perception of low difficulty is better (Carlson & Ostrosky, 1992). Further supporting the anchoring hypothesis, Bard and Weinstein (2017) found that participants’ (n = 55) optimism about their performance on a general knowledge test did not change when they took the test in opposite difficulty orders; participants who took the E–H version first were still more optimistic about their performance than those who took the E–H version after already seeing the same items on the H–E version. Additional support comes from Habbert and Schroeder (2020), who observed that participants in a series of experiments tended to perform better under H–E ordering, even though a majority of participants expressed an a priori preference for the E–H order; these findings suggest that people over-emphasize the earliest tasks when imagining a series of tasks, and thus prefer delaying more difficult tasks.

Multiple studies (10/88) have also investigated the impact of item-difficulty ordering on test-taker affect, primarily state anxiety (i.e., the temporary experience of anxiety due to a particular context or stimulus). Based on five studies (total n = 378) comparing E–H to random order, Aamodt and McShane (1992) concluded that there was no significant difference in test-taker anxiety levels. For example, Smouse and Munz (1968) found no difference in state anxiety between E–H, H–E, and random orders, possibly due to high baseline test anxiety masking item order effects. However, three other studies (total n = 189) comparing E–H and H–E orders supported a significant moderate effect on anxiety (mean d = -.3), with E–H causing less anxiety. Recently, Bieleke et al. (2023) demonstrated that ending a math test on the most difficult items caused more negative emotions (e.g., boredom and anger) among the 8th-grade participants, concluding that it is best to avoid clear monotonic patterns in item-difficulty arrangements.

A few studies have investigated test-taker affect indirectly by analyzing completion rate (4/88) or time on test/task (9/88) as an outcome variable. Ling et al. (2017) found that middle-school students spent longer on an easy CAT (i.e., target probability of correct response > 50%) than on a regular CAT. Miller and Andrade (2020) found that alternating between easy and difficult items led to greater persistence on “impossible” items at the end of the test. And Anaya et al. (2022) found that participants in the H–E condition of a low-stakes online math test were 50% more likely to abandon the test than those in the E–H condition. Thus, while the studies cannot conclusively explain the affective mechanism at play, there is more recent evidence that alternating item difficulty can help maintain intrinsically motivated persistence, which may be less evident in high-stakes testing contexts with highly motivated test takers. One scenario where the link between difficulty and persistence may be especially relevant is in wide-range language assessments taken by lower-proficiency test takers. For example, productive skills tasks with linguistically complex prompts could conceivably function as an initial “hard” or “impossible” item; encountering such items earlier in the test could increase the chance of task abandonment, not due to productive skills, but because of a failure to comprehend the instructions, combined with lower persistence due to the H–E order.

These studies in this section provide empirical support that E–H item ordering slightly increases scores and decreases anxiety, although impacts may be mediated by trait anxiety and/or proficiency. However, these findings are by no means definitive; Hauck et al. (2017) identified 18 studies comparing scores under E–H, H–E and/or random order conditions. Across these studies, the most common finding was no significant difference, with only two consistently showing higher scores for E–H > H–E, one for H–E > E–H, and two for E–H > Random. Item ordering may also be moderated by test time limit and scoring mechanism (i.e., number correct vs. IRT).

On the contrary, ending a test with the most difficult items can decrease motivation/persistence and leave test takers in a more negative emotional state. Collective evidence suggests a potential benefit of starting with easy items and alternating between difficult and easy items over the course of the test, avoiding a monotonic increase in difficulty. However, this particular item arrangement has not been widely investigated. Notably, item-difficulty ordering studies have not looked specifically at language tests, presenting an opportunity for novel research. For example, we might hypothesize that the effect of variables could be exacerbated in a language testing context due to test takers’ foreign language anxiety.

Context Switching

The studies reviewed in this section involve manipulations at various levels of test structure. These include the ordering of items based on their content, and of task formats or characteristics. Broader context switching of test sections is then discussed in the subsequent section on Section/Skills-based Ordering.

On tests measuring multiple subskills or comprising multiple task types (i.e., most language tests), some context switching is inevitable. There are three primary ways in which test takers switch contexts: content, format, and skill/modality. Content-related context switching occurs when adjacent tasks use stimuli on different topics (e.g., two clozes on different topics). Format-related context switching occurs when adjacent tasks or items are of different formats (e.g., an MCQ item followed by a matching item). Skill/modality-related context switching occurs when adjacent tasks assess different skills (e.g., a listening task followed by a reading task). Skill/modality-related context switching is particularly relevant to language proficiency tests and will be addressed in more detail in the following section.

Two contradictory hypotheses exist about context switching’s effect on test performance. The debilitative hypothesis supposes that context switching will hurt test performance due to the additional time and mental effort required to adjust to new task requirements and/or activate different mental resources. The facilitative hypothesis supposes that context switching has a net positive effect by avoiding monotony and fatigue of specific mental processes. However, little research has directly addressed this issue, and thus, there is no clear consensus on whether tasks or items should be grouped by format and/or content. This topic is conceptually related to the notion of schemata, as applied in the field of language pedagogy, which suggests learners are better able to perform tasks for which their schemata have been activated (Carrell, 1987).

Studies investigating item grouping by content (31/88) are more numerous than those on grouping by format or skill/modality (15/88). The earliest and most direct study is Newburger (1942), investigating task-level characteristics in the context of homogeneity and potential fatigue. In this study, participants responded to spatial reasoning MCQs in easy or hard and homogeneous or heterogeneous item sets. The author concluded that sustained engagement with highly similar tasks can reduce performance due to fatigue from repeatedly activating the same cognitive processes (Newburger). This finding illustrates how the homogeneity of a sequence of cognitive demands at the task or item-block level, rather than just individual item characteristics, can impact performance.

Numerous subsequent studies (17/88) have found no significant group differences in mean scores due to syllabus/coursebook/topic-based ordering (e.g., Davis, 2017; Marso, 1970; Pettijohn & Sacco, 2007). For example, Marso (1970) investigated item ordering on a university-level psychology final exam (n1 = 122, n2 = 156), comparing random, chronological (by course coverage), and reverse chronological ordering, finding no significant impact on scores or test completion time. Similarly, in a series of three experiments, Neely et al. (1994) found no consistent evidence that sequential (by topic) item ordering in psychology exams resulted in better student performance compared to random ordering. Likewise, Cathey et al. (2018) reported no significant effects of sequential chapter-based versus random item ordering on psychology students’ exam performance, test anxiety, or confidence, even when chapter labels were used to make the sequential organization explicit. Klosner and Gellman (1973) simultaneously investigated both difficulty ordering and content ordering for an educational measurement exam (n = 54), arranging items either by subject matter, E–H, or E–H within subject matter. Overall interaction of ordering condition and performance was not significant, but poor-performing students (based on midterm grades) performed best when difficulty increased within subject matter areas, pointing to the importance of considering mediating variables.

In contrast, a number of studies (14/88) observed improved scores for sequential ordering, either forward sequencing (e.g., Balch, 1989; Togo, 2002) or reverse sequencing (e.g., Baldwin & Howard, 1983; Gruber, 1987). Conversely, Norman (1954) found a negative effect of reverse chronological ordering, but only on MCQs. Aamodt and McShane (1992) found a minuscule effect size (d = .04) of content-based item organization across 16 studies, and Hauck et al. (2017) likewise noted conceptual item ordering’s lack of impact. Of note, Carlson and Ostrosky (1992), in a study comparing multiple forms of a microeconomics exam, manipulated both difficulty ordering and content ordering of items. Although they found no difficulty ordering effect, they reported marginal significance (p = .06) that items ordered sequentially by chapter content led to higher performance than items scrambled by content.

As noted, less research has explored format-based context switching, for example comparing grouping/ordering of item or task formats. Monk and Stallings (1970) investigated this area by ordering geography exams in different configurations, with blocks of items grouped by format (e.g., MCQs, True/False, and matching). Under their primarily unspeeded conditions, they generally found no significant impact of these rearrangements on overall test scores, individual item difficulties, or test reliabilities; however, items were always grouped by format, so this study did not address grouping by format versus alternating item formats. In research on fatigue effects of the SAT, J. Liu et al. (2004) assert that other studies have indicated that more frequent switching between tasks could alleviate the necessity for breaks.

Further contributing to the understanding of format-related context switching at the task level, Carter and Prevost (2018) examined the ordering of two distinct constructed-response tasks, one requiring students to define a physiological principle and another requiring them to provide an example of it. Their findings revealed a conceptual priming effect: students generally produced better quality responses on whichever task was positioned second in the sequence. Specifically, definitions improved when they followed the example-generation task, and examples were better when they followed the definition task. These findings suggest that the sequence of tasks with differing cognitive demands or formats can meaningfully impact student performance through mechanisms like priming.

Overall, research on context switching in assessments is scant and does not focus on language testing. Based on the existing research, one takeaway for language test developers is that there is currently no solid empirical foundation for organizing tests by content or item/task format. However, there is weak evidence and speculation that monotonous task sets increase mental fatigue, which would support alternating task types during test administration if validated. Given the lack of evidence, it may be advisable for psychometric and item-bank considerations to carry more weight in determining item order.

Section/Skills-Based Ordering

This subsection centers on research that has manipulated the order of major test sections. As indicated above, a smaller but distinct group of seven studies investigated ordering effects at this macro-level, often focusing on sequences based on language skills.

At a minimum, context switching occurs between test sections. Traditionally, most language tests comprise speaking, writing, listening, and reading (SWRL) sections. This configuration intuitively aligns with common conceptualizations of language skills and is expedient for administering pen-and-paper tests (e.g., playing listening stimuli for all test takers simultaneously), as well as some group-administered computer-based testing (e.g., to avoid test takers speaking at different times). However, there is little empirical basis for this configuration, including both the grouping of test content into skills-based sections and the ordering of these sections. Furthermore, as tests are increasingly being designed (or retrofitted) for remote administration, practical considerations relevant to pen-and-paper tests are often moot, and the lines between test sections may be blurred or nonexistent.

Research regarding the impact of test section configurations is sparse (see Stout & Dellva, 1990). Smeding et al. (2013) showed that middle-school girls (n1 = 1127, n2 = 498) performed significantly worse on a standardized math test when it was administered before a verbal test, whereas this effect was not observed in boys. The authors attributed the results to stereotype threat, demonstrating how test component ordering can interact with test-taker characteristics to introduce construct-irrelevant score variance. More relevant to language testing, J. Liu et al. (2004) examined the effects on performance and test-taker affect when the SAT’s essay section was placed at either the start or end of the test. Essay placement did not affect performance, though starting with the essay was preferred by the participants and reportedly reduced anxiety. As a follow-up, Oh and Walker (2006) examined additional test variables in a larger sample (n = 5982). Unlike J. Liu et al. (2004), a difference in essay performance was found, with the “essay-first” group scoring significantly higher, though essay placement only accounted for 2% of essay score variance. It is noteworthy that in these studies examining essay placement on the SAT, an entire “section” effectively consisted of a single, substantial task (the essay). Consequently, while the manipulation occurs at the macro-level of test structure (section placement), the findings also offer insights into the sequencing of a longer productive skills task relative to other sections. These findings are relevant to language assessment decisions of where to place open-ended productive skills tasks, which are often perceived as inducing high anxiety.

Ahmadi et al. (2012) is the only study to explicitly examine skills-based language test section order (Reading, Listening, and Writing). Students in IELTS preparation classes (n = 60) were surveyed about their preferred section order, with the two most popular being Listening–Reading–Writing and Reading–Listening–Writing. Reasons given focused on the relative difficulty of different modules and the anxiety/confidence they induce. Practice tests arranged in these two orders were then administered to test takers (n = 120). For intermediate and advanced learners, ordering did not significantly affect performance. However, beginners performed better overall when listening came first. These results illustrate the potential interaction of proficiency and section order on performance. The preference for Listening first among these Iranian IELTS test takers, and the performance benefit for beginners when Listening was placed in initial position, may be partly understood in the context of national IELTS performance patterns. IELTS (2025) Academic test-taker performance data from 2023–2024 shows that Iranian candidates (and candidates in most reported countries) tend to score higher in Listening (M = 6.6) than in Writing (M = 6.0). However, such national performance trends, and thus perceived or actual relative skill difficulty, vary by country (e.g., in Oman, Listening M = 5.3 and Writing M = 5.4). These score trends suggest that the preferred or optimal skill sequencing may differ by first language or educational contexts.

In this section, we have seen that research into section ordering is limited, and sections may be ordered for historical or practical reasons. However, different language skills may be (or may be perceived) as more challenging for test takers. As such, further research into the effects of section ordering is needed. Table 3 presents a sample of high-stakes language tests and their section orders, as inferred from public documentation. This brief survey shows consistency amongst high-stakes language proficiency tests: typically, sections are clearly defined according to traditional SWRL language skills; receptive skills (reading and listening) are followed by productive skills (speaking and writing); and objective tasks are followed by subjective tasks.

OPEN IN VIEWER

Table 3.

Examples of language Test Section Ordering.

Test	DELE B2(Diploma de español nivel B2)	DELF B2(Diplôme d’Etudes en Langue Française B2)	DET(Duolingo English test)	Goethe-Zertifikat B2	IELTS(International English Language Testing Service)	JLPT(Japanese Language Proficiency Test)	TOCFL(Test of Chinese as a Foreign Language)	TOPIK
Language	Spanish	French	English	German	English	Japanese	Mandarin	Korean
Sections	1. Reading2. Listening3. Writing4. Speaking	1. Listening2. Reading3. Writing4. Speaking	1. Linguistic resources (Mostly CAT)2. Skills mastery (Mostly non-CAT)	1. Reading2. Listening3. Writing4. Speaking	1. Listening2. Reading3. Writing4. Speaking (typically)	1. Vocabulary2. Grammar & Reading3. Listening	1. Listening2. Reading	1. Listening2. Writing3. Reading
Task type grouping	1. Objective2. Subjective	1. (Mostly) Objective2. Subjective	1. Objective2. Subjective	1. (Mostly) Objective2. Subjective	1. Objective2. Subjective	1. Objective	1. Objective	1. Objective2. Subjective (short)3. Subjective (long)4. Objective
Skills grouping	1. Receptive2. Productive	1. Receptive2. Productive	1. (Mostly) Receptive2. (Mostly) Productive	1. Receptive2. Productive	1. Receptive2. Productive	1. Receptive	1. Receptive	1. Receptive2. Productive3. Receptive

If minimal context switching is preferable, then tasks should continue to be clustered into test sections based on skills grouping, though the implications for computer-adaptive administration must be considered. For example, tasks could follow the traditional SWRL grouping (e.g., IELTS and TOEFL); broader receptive or productive skill groups (e.g., DET and PTE Academic); or integrated skill groups mirroring the categories in the updated Common European Framework of Reference for Languages (CEFR; Council of Europe, 2020): Reception, Production, Interaction, and Mediation.

Gender/Sex

Gender/sex, the most commonly included demographic variable (16/88), has an unclear mediating impact with mixed findings. In half of the studies including this variable (8/16), across many contexts, gender/sex was not found to have a significant mediating effect (e.g., Anaya et al., 2022; Capron, 1933; Gruss & Clemons, 2023; Oh & Walker, 2006).

However, half of the studies did find gender/sex to significantly mediate ordering effects. For example, Plake et al. (1982, 1988) found a significant interaction between item order and gender, with males scoring higher in the E–H condition on statistics exams, and Plake and Ansorge (1984) found a significant sex by item order interaction on students’ anticipated performance (but not actual scores) on an educational psychology exam. Conversely, Kennedy and Butler (2013) observed that females outperformed males in the H–E condition. Likewise, in Harrington et al. (2019), H–E item ordering in computer science exams, despite little overall impact, significantly boosted female students’ confidence and performance, while males showed higher confidence and better performance with E–H ordering. Gender/sex has also been found to show complex relationships with item ordering and item content (e.g., Ryan & Chiu, 2001), as well as test section ordering and performance (e.g., Smeding et al., 2013, discussed previously). Ryan and Chiu (2001) found that, in mathematics placement exams, altering item order (random vs. E–H within content) generally did not change the overall patterns of differential item functioning (DIF), though E–H ordering of algebra items modestly improved performance for both genders.

Proficiency/Academic Ability

The interaction of language proficiency or domain-specific academic ability and ordering conditions is relevant to language test developers. Lower-proficiency test takers could be especially sensitive to ordering differences, which could provoke additional anxiety or require them to deploy more cognitive resources. However, because most ordering studies are not on language tests, such hypotheses cannot be confirmed.

One exception is Ahmadi et al. (2012), in which participants were split into three broad proficiency categories of beginner, intermediate, and advanced. An ordering effect was uncovered for beginners, who performed better when the listening section was first. Likewise, Kobayashi (2002) found that, in a reading comprehension test of Japanese university students (n = 754), the lower-proficiency group performed better with controlled task types such as cloze tests and reading passages that were more descriptive as opposed to densely informational. Although Kobayashi is not strictly related to ordering, the work of this nature points to the manner in which test specifications interact with language proficiency. In contrast to Ahmadi et al. (2012), Hodson (1984a), in a non-language assessment context (O-level chemistry), found that E–H item ordering significantly improved scores only for high-ability students, with no significant E–H effect on the overall group’s performance.

Şad (2020) also conducted an ordering study which included proficiency (referring to level of academic achievement) as a mediating variable. Turkish students in a teacher training program (n = 554) completed an MCQ test about measurement and assessment with E–H and H–E item orders. The author found no mediating effect for the proficiency group, regardless of difficulty order. As described previously, Baldwin and Howard (1983) found that GPA mediated the effects of course-sequenced versus scrambled item ordering for a financial accounting exam (n = 167). However, subsequently Baldwin et al. (1989) did not see this effect for a managerial accounting exam (n = 282).

These findings suggest that language proficiency cannot be easily grouped together with other proficiency types when considering ordering effects, that subject matter is also a mediating variable, and that the interaction of proficiency and difficulty ordering remains inconclusive.

Trait Anxiety

Previously, we described studies where state anxiety was an outcome variable, that is, where anxiety was affected by item-difficulty ordering. However, there are also a small number of studies (10/88, mostly from the 1960s and 1970s) that investigated test takers’ baseline anxiousness (i.e., trait anxiety) as a mediating variable. The studies from the 1950s and 1960s uncovered a relationship between trait anxiety and test performance under different conditions. For example, Munz and Smouse (1968) investigated the test performance of undergraduate psychology students (n = 120), finding only a significant interaction of item-difficulty ordering and test anxiety subtype. Specifically, participants classified as facilitators (meaning their anxiety leads to the optimal level of arousal) based on the Achievement Anxiety Test (AAT) scored higher with the E–H order. They therefore recommend using the H–E order, which was not shown to significantly interact with anxiety types.

Later studies using the AAT to operationalize trait test anxiety failed to find significant interactions between ordering and trait anxiety. Hambleton and Traub (1974) examined item-difficulty order and stress levels for high school students (n = 140) completing a math test. The inferential analysis, using baseline pulse as a covariate, did not find evidence to support the hypothesis that item-difficulty ordering differentially affected test takers of different anxiety levels. Similarly, other studies (e.g., Marso, 1970; Plake et al., 1981) examined the impact of item-difficulty ordering and test-taker anxiety levels on MCQ test performance of undergraduate students. Overall, there was no consistent significant main effect or interaction of trait anxiety in any such study.

More recently, Chen (2012) asked English-major Chinese university students (n = 250) to complete an MCQ cultural knowledge test. Participants also completed the Test Anxiety Inventory and were grouped into low-, mid-, and high-anxiety groups. The author found an interaction between item order and anxiety levels, with the performance of low-anxiety participants not significantly affected by fixed item orders. As in Munz and Smouse (1968), the E–H order leads to less anxiety for some but not all test takers, in this case, those with moderate to high test anxiety. Together, the findings from some of these studies support the position that ordering effects are moderated by test takers’ initial anxiety level in various ways, with stronger effects for more anxious students.

Several methodological and contextual factors likely contribute to the inconsistent evidence that anxiety mediates ordering effects. First, the operationalization of trait anxiety varied, including by instrument(s) used to measure anxiety (e.g., AAT alone vs. combined with the State-Trait Anxiety Inventory), AAT subscales used (e.g., debilitating scale alone vs. combined with facilitating scale), and the nature of the anxiety variable used in analyses (e.g., categorical vs. integer). Second, the studies took place in a variety of contexts, including both high school and university, as well as both authentic classroom exams and contrived laboratory experiments. Perhaps most importantly, the studies generally had modest sample sizes, ranging from 41–352 participants, meaning some analyses may have been underpowered. Collectively, the heterogeneity of study designs, contexts, and sample sizes makes it less likely that a uniform pattern of results would emerge.

In summary, findings are mixed concerning how ordering is mediated by commonly considered variables. However, there is some evidence that more anxious and lower‑proficiency test takers are impacted more by ordering and that E–H ordering may negate these unwanted effects. Mixed results for anxiety likely reflect differences in operationalization, study design, and contextual moderators.

Speededness

Speededness refers to “the extent to which test takers’ scores depend on the rate at which work is performed as well as on the correctness of the responses” (AERA et al., 2014, p. 223; see also Jurich, 2020). Prototypical speeded tests comprise items of similar difficulty levels and differentiate test-taker proficiency levels by the number of items completed within a strict time limit. Prototypical power tests, on the other hand, include items of varying difficulty and are designed to allow test takers enough time to attempt all items. Most high-stakes tests are not explicitly designed to be speeded tests, in that test takers are expected to attempt all tasks (Kane, 2020); however, high-stakes tests still have an element of speededness (Powers & Fowles, 1996). For speeded assessments, test ordering naturally affects performance as test takers may not reach later items on the test (Becker et al., 2022; Leary & Dorans, 1985; Sax & Cromack, 1966). Becker et al.’s (2022) simulation study demonstrated how different item orders can result in large score differences on a speeded test, corroborating the conclusion from earlier research on the impact of speededness (e.g., Kleinke, 1980). To mitigate this issue, Becker et al. (2022) suggest using a single fixed item order at the end of the test across forms and placing time-intensive items at the end of the test.

Many high-stakes language tests, such as IELTS, TOEFL, and TOPIK, aim to assess a wide range of language proficiency levels in a single fixed form and therefore include items of different difficulty levels, resembling power tests in their construction. However, low-proficiency test takers may not have enough time to thoroughly attempt all items, leading to differential speededness. In situations where test takers have no control over the speed of the stimulus presentation (e.g., listening tests), low-proficiency test takers may need to rush responses to items. Therefore, for language tests with a single form comprising items of widely varying difficulty and insufficient time limits for the lowest-proficiency test takers to attempt all items, E–H item ordering is arguably advisable to minimize unfairness based on proficiency. This recommendation does not apply to CATs and testing programs with multiple forms targeting specific ability levels, such as separate tests for each CEFR level. Further research is needed to determine the effects of different item orders in these cases.