Computer Versus Paper-Based Testing: Are They Equivalent When it Comes to Working Memory?

Abstract

School systems across the country are transitioning from paper-based testing (PBT) to computer-based testing (CBT). As this technological shift occurs, more research is necessary to understand the practical and performance implications of administering CBTs. Currently, there is a paucity of research using CBTs to examine working memory (WM) performance, even though CBTs may negatively influence performance. The present study compared a WM CBT and PBT and found enhanced WM performance on the PBT across several verbal and visuospatial WM tests. This pattern was evident even after age was controlled, indicating that test mode effects were persistent across ages (4-11 years). CBTs on WM performance may yield lower scores due to developmental WM differences, increased cognitive workload, test mode effects stemming from individual access to technology, and participant characteristics, such as developmental, biological, or gender differences. The presence of divergent WM in CBT and PBT indicates the need for additional options for children at risk of academic failure because of testing modality.

Keywords

working memory computer-based testing paper-based testing cognitive load test mode effect

School systems across the country are transitioning their standard testing modality from paper-based testing (PBT) to computer-based administration (Boeve, Meijer, Albers, Beetsma, & Bosker, 2015). Technology that incorporates computer-based testing (CBT) is becoming increasingly prevalent in universities and secondary schools alike (Sedlacik & Cechova, 2016). These changes are driven primarily by not only considerations of practicality, convenience, and efficiency but also large-scale school computer integration (Randall & West, 2014). Since the early 2000s, several U.S. states have required that their high school students engage in primarily online learning (Jacob, Berger, Hart, & Loeb, 2016).

CBT is promoted as a way of revolutionizing education and improving student performance and test scores through unique computerized test-taking tools. Proposed benefits of CBT include improved security, accelerated result reporting, more efficient year-round testing, flexible scheduling, environmental benefits, and increased student focus and enjoyment (Boeve et al., 2015). In addition, students are increasingly adept with computerized technology, and the widespread integration of desktop computers, laptops, iPad devices, and smartphones in schools makes students today part of a “digitally native” generation (Katz & Elliot, 2016).

On the contrary, administering strictly CBTs may cause some students to perform poorly simply due to a shift from PBTs, not because of lower ability—referred to as the “test mode effect” (Leeson, 2009). Early theorists argued that for CBTs and PBTs to be equivalent, meaning similar in both content and necessary cognitive abilities, the two tests should produce identical results. The presence of test mode effects suggests that CBTs introduce confounding variables that negatively affect test scores. Test mode familiarity (TMF) deals primarily with participant issues, and holds that some students’ performance may suffer if they have limited experience with CBTs (Hanho, 2014).

Criterion Test Equivalency

Despite the shift toward CBTs and students’ increasing familiarity with technology, findings in criterion-based tests (e.g., math and reading) have been mixed. Some studies have found no difference between the two testing modalities (Hosseini, Abidin, & Baghdarnia, 2014; Wang, Jiao, Young, Brooks, & Olson, 2007) while others have found test mode effects (Jeong, 2012; Khoshsima & Toroujeni, 2017; Noyes & Garland, 2008). For purposes of this article, recent research has found no significant differences between CBT and PBT performance in cognitive assessments (Collerton et al., 2007; Register-Mihalik et al., 2012).

Cognitive Assessment Equivalency

Collerton et al. (2007) compared the use of CBTs of cognitive function with PBTs. Eighty-one participants over the age of 85 were randomly assigned to the paper Wechsler Adult Intelligence and Memory Scale, or the computerized Cognitive Drug Research (CDR) verbal memory and attention assessment battery. These tasks were chosen on the basis that they assess the same cognitive areas as other 85+ cognitive studies, both take 30 min to complete, and have been previously validated to be accurate cognitive measures for people in this age group. Participants assigned to the CBT did not rate the test as difficult, unacceptable, or stressful. Participants who completed the PBT version finished the assessment at a higher speed, but did not score significantly better than those who took the CBT. The researchers concluded that TMF does not play a major role in these cognitive assessments and thus, CBTs and PBTs were equally feasible and valid measures of cognitive function.

In addition, maintaining format equivalence when examining paper-based and computer-based formats of cognitive intelligence scales (e.g., Wechsler Preschool and Primary Scale of Intelligence–Fourth Edition [WPPSI-IV]) has been examined looking at Q-interactive formats (i.e., iPads; Daniel, Wahlstrom, & Zhang, 2014; Raiford et al., 2016; Wechsler, 2008). The overall goal is to demonstrate raw score equivalence (e.g., equivalence across paper-based and digital administration) through reliability and validity analyses of the paper format and the subsequent application of the digital format. Raiford and colleagues (2016) tested this in 38 children aged 2 to 7 by comparing performance on an iPad-administered WPPSI-IV to a PBT WPPSI-IV. The effect sizes fell within the Q-interactive equivalence format, and the authors concluded that regardless of test format, the WPPSI-IV produces equivalent scores.

Working Memory (WM)

WM (i.e., the ability to process and recall information) is critical for a variety of activities at school, ranging from complex subjects such as reading comprehension, mental arithmetic, and word problems to simple tasks like copying from the board and navigating around school (Alloway & Copello, 2013). WM is important from kindergarten to the tertiary level and is an excellent longitudinal predictor of academic success (Alloway & Alloway, 2010).

There are multiple models of WM, some of which incorporate concepts of attention in memory (Engle, Kane, & Tuholski, 1999) and temporal duration in performing memory tasks (Barrouillet, Bernardin, & Camos, 2004), while others suggest that WM is an activated component of long-term memory (Cowan, 2005). The model used in the present study is Baddeley’s (2000) model, which consists of four components: the central executive, the phonological loop, the visuospatial sketchpad (VSSP), and the episodic buffer. The central executive is responsible for the high-level control and coordination of the flow of information through WM, including the temporary activation of long-term memory. It has also been linked with control processes such as switching, updating, and inhibition (Baddeley, 1996). The central executive is supplemented by two slave systems specialized for storage of information within specific domains. The phonological loop provides temporary storage for linguistic material, and the VSSP stores information that can be represented in terms of visual or spatial structure. The fourth component is the episodic buffer, responsible for integrating information from different components of WM and long-term memory into unitary episodic representations (Baddeley, 2000). This model of WM has been supported by evidence from studies of children, adult participants, and neuropsychological patients (Gathercole & Baddeley, 1993).

The key feature of WM is its capacity to both store and manipulate information. WM functions as a mental workspace that can be flexibly used to support everyday cognitive activities that require both processing and storage such as mental arithmetic. However, the capacity of WM is limited, and the imposition of either excess storage or processing demands in the course of an ongoing cognitive activity will lead to catastrophic loss of information from this temporary memory system. In contrast, short-term memory refers to the capacity of storing units of information, and is typically assessed by serial recall tasks involving arbitrary verbal elements such as digits or words. In a similar vein, recent models have been created to improve WM function and have been found to effective for cognitive improvement (Jaeggi, Buschkuehl, Jonides, & Shah, 2011).

WM and CBTs

Some aspects of WM, such as the VSSP, may be more affected by CBTs than others. The VSSP consists of two subsystems: one that maintains spatial information, and the other for visual information (Pickering, Gathercole, Hall, & Lloyd, 2001). Some researchers argue that developmental fractionation, or the differential maturation rates of visual and spatial memory, provides evidence that the two components of the VSSP may in fact be unrelated constructs (Hitch, 1990). Logie and Pearson (1997) investigated this in children (5-12 years) using a memory test for sequences of movements to targets, or visual patterns. Results illustrated that memory for patterns was better than memory for movement sequences in the oldest group. This was counterbalanced in another condition and the same pattern emerged. These findings support the idea that visuospatial WM includes distinct forms of storage for visual and spatial material due to developmental differences across ages.

In a similar vein, Choi and Tinker (2002) found that developmental differences in third and 10th graders affected performance on CBT assessments. Specifically, there was a higher mode effect for tasks involving a textual focus (i.e., phrases or words) compared with a sentence-scanning task. The researchers concluded that when students are younger, it is easier for them to scan for words on PBTs versus CBTs, possibly because of developmental fractionation in the VSSP. The older students stored and processed information differently than the younger students, which may result in test mode effects of the CBT. More recently, Colbert and Bo (2016) examined change-detection tasks (i.e., WM assessment) and evaluated the relationship to the Wechsler Intelligence Scale for Children (WISC-IV). Results indicated significant age-related improvements on verbal and visuospatial WM, meaning that as students age, multiple subsets of WM improve.

An alternative theory is that the activation of the subcomponents of the VSSP depends on a dynamic/static distinction in the stimulus rather than a spatial/visual one. Pickering et al. (2001) conducted an experiment with 5-year-old, 8-year-old, and 10-year-old children who completed a mazes visuospatial WM task. Static and dynamic presentation formats were used to observe the differences in the two subcomponents. In the mazes static condition, a child was presented with empty mazes in a booklet and asked to recall the different routes denoted by a clear red line. The maze booklet was presented to the child for 3 s and then the child was asked to redraw the routes after the booklet was removed. In the mazes dynamic condition, the child was presented with the same booklet, but the experimenter demonstrated the route the child should draw by tracing it with their finger. The child was then prompted to draw the route the experimenter just traced. Testing continued in both the static and dynamic maze tasks until the child incorrectly drew the patterns. Results indicated that the 8- and 10-year-old children performed significantly better than the 5-year-olds on the static version of the maze tasks, but not on the dynamic version. This suggests that the two components of visuospatial memory may be dependent on the static/dynamic features of the stimulus and that VSSP development occurs earlier with static stimuli than with dynamic stimuli. These findings are related to mode effects in that CBTs may affect the VSSP more in older children and lead to higher performance in the static condition. In the present study, we integrated age effects on testing modality using similar age bands confirmed by past WM modes (Alloway, Gathercole, & Pickering, 2006).

Current Study

As schools across the country change their testing modalities from PBT to CBT assessments, researchers and educators need to ensure that this shift is beneficial for all students. To date, little research has been conducted on the comparability of WM performance on CBTs to PBTs, and given the implications of WM in education (Alloway & Alloway, 2010), further investigation is needed to ensure that CBTs are not negatively influencing WM performance. Researchers should investigate how CBTs affect WM performance to improve learning support, and develop accurate cross-modal measures of WM.

Method

Participants

Overall, 1,339 students (656 males, 683 females, M_age = 7.54 years, age range = 4-11) were recruited from several schools in England. For the participants aged 4 to 11, a nationally representative demographic sample was recruited on the basis of the national average of performance on national assessments in English, mathematics, and science that pupils sit for in the final year of primary school at the age of 10 or 11. Schools in England are ranked on the basis of a combined or “aggregate” score achieved in the three tests—the maximum possible being 300 (Department for Education and Skills). Schools located in both urban and rural settings were selected for the normative sample represent a range of low, average, and high performance in the combined score of the national test results. Participant demographics can be found in Table 1.

Table 1.

Demographics Inclusive of Age Bands and Test Modality.

	N	M	SD
Age (4-11)	1,339	7.54	2.05
Females	683 (51%)
Males	656 (49%)
Age bands
4-6	472 (35.20%)
7-8	370 (27.70%)
9-11	497 (37.10%)
PBT	630 (47.10%)
CBT	709 (52.90%)

Note. PBT = paper-based testing; CBT = computer-based testing.

Materials and Procedure

Participants were randomly assigned to a paper-based version of the Automated Working Memory Assessment (AWMA) or the computer-based AWMA. The AWMA (Alloway, 2007) consists of tests measuring verbal short-term memory, verbal WM, and visuospatial memory. Standard scores (M = 100; SD = 15) for individual tests and composite scores for each memory component were generated automatically by the AWMA for each child based on their age.

The three verbal short-term memory measures were digit recall, word recall, and nonword recall. In each test, the child hears a sequence of verbal items (digits, one-syllable words, and one-syllable nonwords, respectively) and has to recall each sequence in the correct order. The stimuli were presented at a rate of one per second. The normative sample included individuals aged 4.5 and 22.5 years, and test–retest reliability is .88, .89, and .69 for digit recall, word recall, and nonword recall, respectively.

The three verbal WM measures were listening recall, backward digit recall, and counting recall. In the listening recall task, the child is presented with a series of spoken sentences, has to verify the sentence by stating “true” or “false,” and recalls the final word for each sentence in sequence. In the backward digit recall task, the child is required to recall a sequence of spoken digits in the reverse order. The numbers are presented at a rate of one per second. In the counting recall task, the child is presented with a visual array of red circles and blue triangles. He or she is required to count the number of circles in an array and then remember the tallies of circles in the arrays that were presented. Each visual array stayed visible until the child indicated that she or he had completed counting all the circles. For individuals aged 4.5 and 22.5 years, test–retest reliability is .88, .83, and .86 for listening recall, counting recall, and backward digit recall, respectively.

Two measures of visuospatial short-term memory were administered. In the mazes memory task, the child is shown a maze with a red path drawn through it for 3 s. She or he then has to trace in the same path on a blank maze presented on the computer screen. In the block recall task, the child views a video of a series of blocks being tapped, and reproduces the sequence in the correct order by tapping on a picture of the blocks. The blocks were tapped at a rate of one block per second. For individuals aged 4.5 and 22.5 years, test–retest reliability is .86, and .90 for mazes memory and block recall, respectively.

Tests were administered, in a fixed sequence design to vary task demands across successive tests. The CBT version of the memory tests was presented on a laptop computer with the screen resolution set to 600 by 480 pixels. For the spoken presentation of stimuli, audio files were recording using a minidisk player and then edited on the GoldWave program (2004). All picture files were created in Microsoft PowerPoint using the standard shape graphics. The test trials were presented as a series of blocks; each block consists of six trials. The CBT version used audio files, while the experimenter read out the stimuli in the PBT version to a group of participants. The timing for stimuli presentation was the same for both the CBT and PBT versions, as indicated above. Participants were required to respond immediately after stimulus presentation, though there was no set time limit for responding in either version. The instructions were presented as a sound file while the computer screen was blank. Practice trials followed the instructions. Participants in either version were not allowed to return to any questions (i.e., item review).

For scoring in the CBT version, the experimenter recorded the child’s response using the right arrow key on the keyboard (→) for a correct response and the left arrow key on the keyboard (←) for an incorrect response. The computer program automatically credits a correct trial with a score of 1. In the PBT version, the experimenter records the participant’s response in the score booklet. According to the “move on” rule, if a child responds correctly to the first four trials within a block of trials, they automatically receive credit for trials that were not administered. However, if three or more errors are made within a block of trials, the test is discontinued. In the CBT version, the scoring, move on, and discontinue rules are automated. The score for that test reflects the number of correct responses up to the point at which the test was ended. Additional information can be found regarding the psychometric properties (reliability and validity) can be found from the manual (Alloway, 2007). Further details of test reliability and validity are reported in Alloway, Gathercole, Kirkwood, and Elliott (2009), respectively.

Results

Age

Descriptive statistics are shown in Table 2, Table 3, and Table 4. To compare the effect of age and test type (CBT or PBT) on WM performance, three MANOVAs were conducted on the three WM assessments (verbal short-term, verbal, and visuospatial) with age bands (4-6, 7-8, 9-11) and test types as the fixed factors. The overall group term associated with Hotelling’s t test is reported in all instances, with Bonferroni correction post hoc comparisons set at an alpha level of p = .0125.

Table 2.

Descriptive Statistics as a Function of Test Types Separated by Age Band 4-6.

Verbal short-term	Digit recall		Word recall		Nonword recall
Verbal short-term	PBT	CBT	PBT	CBT	PBT	CBT
M	23.04	22.81	15.58	13.43	10.88	9.15
SD	4.31	4.34	4.32	3.21	3.07	3.02
Minimum	12.00	6.00	2.00	5.00	2.00	0.00
Maximum	38.00	43.00	55.00	24.00	19.00	18.00
Verbal working memory	Listening recall		Counting recall		Backward digit recall
Verbal working memory	PBT	CBT	PBT	CBT	PBT	CBT
M	8.17	6.00	15.37	11.54	9.66	6.63
SD	2.76	3.08	4.99	4.14	2.96	3.46
Minimum	2.00	0.00	0.00	2.00	0.00	0.00
Maximum	16.00	15.00	47.00	26.00	18.00	18.00
Visuospatial short-term memory	Block recall		Mazes
Visuospatial short-term memory	PBT	CBT	PBT	CBT
M	21.01	13.05	8.80	8.56
SD	3.71	3.75	5.11	5.46
Minimum	7.00	2.00	0.00	0.00
Maximum	30.00	25.00	23.00	23.00

Note. PBT = paper-based testing; CBT = computer-based testing.

Table 3.

Descriptive Statistics as a Function of Test Types Separated by Age Band 7-8.

Verbal short-term	Digit recall		Word recall		Nonword recall
Verbal short-term	PBT	CBT	PBT	CBT	PBT	CBT
M	26.23	25.57	18.62	16.56	12.57	11.02
SD	3.61	3.45	3.21	3.25	3.71	3.30
Minimum	10.00	17.00	11.00	5.00	6.00	0.00
Maximum	36.00	39.00	29.00	29.00	19.00	20.00
Verbal working memory	Listening recall		Counting recall		Backward digit recall
Verbal working memory	PBT	CBT	PBT	CBT	PBT	CBT
M	10.52	9.40	20.06	16.49	12.06	10.20
SD	3.20	2.75	5.04	4.42	3.35	3.49
Minimum	5.00	3.00	6.00	6.00	0.00	0.00
Maximum	22.00	18.00	32.00	31.00	23.00	25.00
Visuospatial short-term memory	Block recall		Mazes
Visuospatial short-term memory	PBT	CBT	PBT		CBT
M	24.37	17.88	15.23		14.28
SD	3.59	3.79	5.90		5.11
Minimum	14.00	7.00	2.00		1.00
Maximum	32.00	30.00	28.00		27.00

Note. PBT = paper-based testing; CBT = computer-based testing.

Table 4.

Descriptive Statistics as a Function of Test Types Separated by Age Band 9-11.

Verbal short-term	Digit recall		Word recall		Nonword recall
Verbal short-term	PBT	CBT	PBT	CBT	PBT	CBT
M	29.49	28.62	20.83	19.34	14.13	13.44
SD	5.52	4.74	3.64	3.58	2.72	3.22
Minimum	13.00	19.00	10.00	8.00	7.00	3.00
Maximum	60.00	45.00	37.00	30.00	23.00	21.00
Verbal working memory	Listening recall		Counting recall		Backward digit recall
Verbal working memory	PBT	CBT	PBT	CBT	PBT	CBT
M	12.51	12.15	23.63	21.01	14.39	13.16
SD	2.75	3.13	4.36	4.64	4.04	3.93
Minimum	0.0	5.00	12.00	6.00	6.00	6.00
Maximum	19.00	27.00	37.00	32.00	30.00	28.00
Visuospatial short-term memory	Block recall		Mazes
Visuospatial short-term memory	PBT	CBT	PBT	CBT
M	27.25	21.09	20.74	19.05
SD	4.17	3.88	6.20	4.97
Minimum	19.00	11.00	5.00	1.00
Maximum	60.00	33.00	50.00	36.00

Note. PBT = paper-based testing; CBT = computer-based testing.

The first MANOVA performed on the verbal short-term memory tests indicated significant main effects for age group, F(6, 2660) = 123.23, p < .001, $η_{p}^{2}$ = .22; test modality, F(3, 1331) = 38.89, p < .001, $η_{p}^{2}$ = .08; and a significant interaction, F(6, 2660) = 2.42, p = .025, $η_{p}^{2}$ = .01. There was a significant interaction for nonword recall, F(2, 1333) = 4.01, p = .018, $η_{p}^{2}$ = .01; but no significant interactions for digit recall, F(2, 1333) = .63, p = .532; or word recall, F(2, 1333) = 1.15, p = .316. Post hoc tests (Bonferroni adjusted, p = .0125) using a univariate ANOVA indicated that nonword recall performance was higher on the PBTs than CBTs for all age groups, and that the older the children, the better their performance on nonword recall tests.

The second MANOVA performed on the verbal WM tests indicated significant main effects for age group, F(6, 2480) = 175.47, p < .001, $η_{p}^{2}$ = .30; test modality, F(3, 1241) = 61.71, p < .001, $η_{p}^{2}$ = .129; and a significant interaction, F(6, 2480) = 3.85, p = .001, $η_{p}^{2}$ = .01. There was a significant interaction for both listening recall, F(2, 1243) = 9.57, p < .001, $η_{p}^{2}$ = .01, and for backward digit recall, F(2, 1243) = 6.52, p = .002, $η_{p}^{2}$ = .01. No significant interaction was found for counting recall, F(2, 1243) = 2.16, p = .116. Post hoc tests (Bonferroni adjusted, p = .0125) from a univariate ANOVA indicated that performance was higher on the PBTs than CBTs for all age groups on backward digit recall and listening recall, and that older children performed better on both listening recall and backward digit recall WM assessments.

The third MANOVA performed on the visuospatial memory tests indicated significant main effects for age group, F(4, 2482) = 273.70, p < .001, $η_{p}^{2}$ = .30; test modality, F(2, 1242) = 474.46, p < .001, $η_{p}^{2}$ = .43; and a significant interaction, F(4, 2482) = 2.62, p = .033, $η_{p}^{2}$ = .004. Post hoc tests (Bonferroni adjusted, p = .0125) from a univariate ANOVA indicated that performance improved as a function of age.

Gender

To compare the effect of gender on performances on the WM tests (CBT & PBT), three separate MANCOVAs were conducted on the memory scores, with age as a covariate and sex and testing types as the independent factors. The overall group term associated with Hotelling’s t test is reported in all instances, with Bonferroni correction post hoc comparisons set at an alpha level of .05.

The first MANCOVA performed on the verbal short-term memory tests indicated a significant group difference for testing type, F(3, 1332) = 41.16, p < .001, $η_{p}^{2}$ = .08, and for sex, F(3, 1332) = 4.11, p = .006, $η_{p}^{2}$ = .01. The interaction between testing type and sex was not significant, F(3, 1332) = 1.39, p = .242. Post hoc tests (Bonferroni adjusted, p < .05) indicated that performance was higher on the PBTs than CBTs, and that females outperformed males on all verbal short-term memory tests meaning that females had higher scores on PBTs compared with CBTs.

The second MANCOVA performed on the verbal WM tests indicated a significant group difference for testing type, F(3, 1242) = 63.77, p < .001, $η_{p}^{2}$ = .13, but not for sex, F(3, 1242) = 2.46, p = .06. The interaction between testing type and sex was not significant, F < 1. Post hoc tests (Bonferroni adjusted, p < .05) indicated that performance was higher on all PBT verbal WM tests.

The final MANCOVA performed on the visuospatial memory tests indicated a significant group difference for testing type, F(2, 1243) = 496.53, p < .001, $η_{p}^{2}$ = .44, and for sex, F(2, 1243) = 12.56, p < .001, $η_{p}^{2}$ = .02. The interaction between testing type and sex was also significant, F(2, 1243) = 5.14, p = 006, $η_{p}^{2}$ = .01. Post hoc tests (Bonferroni adjusted, p < .05) indicated that performance was higher on all PBT visuospatial WM tests, and that the males outperformed the females only on the paper-based mazes test.

Discussion

The primary finding of this study was that WM performance was better on a PBT WM assessment than a CBT assessment across several WM tests. This pattern was evident even after age was controlled, meaning that the CBT/PBT effects were persistent across ages (4-11 years). Additional analyses that divided children into theoretically derived developmental bands (e.g., 4-6, 7-8, 9-11) yielded similar results. To date, little research has been conducted comparing WM performance on CBTs to PBTs, but the present results suggests that CBTs may yield lower scores in assessing WM. Possible reasons include, increased cognitive load, technological mode effects due to socioeconomic status (SES), and developmental and biological gender differences. In addition, item review, or the ability to review and change answers, was not permitted on either the PBT or CBT AWMA, and may have influenced performance levels. WM assessments have an important temporal element, but item review may give underperforming students a chance to improve by taking more time to review their submitted answers.

Recall was significantly better on the PBT than the CBT across item type (digits, one-syllable words, and one-syllable nonwords) on verbal short-term memory. One explanation for the enhanced PBT performance may be the increased cognitive workload and subsequent decrease in WM capacity observed in past research on computer screens (Mayes, Sims, & Koonce, 2001; Noyes & Garland, 2003). If more resources are allocated to a WM task (e.g., rehearsal of a word list), the workload of the individual is increased, and their WM capacity is lowered. Mayes et al. (2001) investigated the effects of cognitive workload on WM capacity by comparing comprehension performance on a paper test to performance on a visual display terminal (VDT) test. Their findings indicated that VDTs increased reading time and cognitive load and negatively affected comprehension scores. Noyes and Garland (2003) speculated that the increased cognitive load may have decreased the amount of rehearsal processing, leading to decreased WM which affected comprehension.

The significant interaction effects for nonword recall may stem from the fact that verbal short-term memory does not operate in isolation, but in a complex cognitive system (Archibald, 2006). Nonwords high in “wordlikeness” (more similar to words that participants already know, known as the wordlikeness effect) may have played a role in the better performance of older children. The older the participant, the better their attention to nonwords that was developed through repetition accuracy for both low- and high-wordlike nonwords. Increased cognitive load from the WM task itself (i.e., high demand rehearsal activity) may have led to a decrease in ability to store information due to the technological mode effects of CBTs. Mayes et al. (2001) found that people required more time reading from computer screens than on paper to retain the same material, and argued that this effect was due to a higher cognitive workload CBTs placed on the information processing system. In this study, attending to the CBT may have been more difficult than attending to the PBT, which may explain the lower performance on this modality. On the contrary, Prisacari and Danielson (2017) did not find differences in cognitive load in those taking a CBT versus PBT assessment. Participants (n = 222) completed three chemistry CBT or PBT assessments and found no significant differences in either testing modality, but the researchers did find that students used a piece of scratch paper more for PBT assessments compared to CBT especially for algorithmic questions. The researchers believed that if cognitive load and certain behaviors of students are different between testing modes, teachers might have reason to select one mode over another. Using a piece of scratch paper may be more difficult in a CBT due to having to copy down the question and referencing both the computer and scratch paper, while on a PBT the question would already be near their scratch paper. To examine cognitive load, a paper N-back task should be compared with a computerized version. To date, this test has only been administered in a computerized format, but may provide exceptional training for improving WM (Soveri, Antfolk, Karlsson, Salo, & Laine, 2017).

Computer fatigue (CF) from technological mode effects such as computer resolution can influence performance and create higher cognitive load. Although the resolution used in this experiment was high (600 × 480 pixels), Ziefle (1998) found through the examination of eye movement parameters that reading from paper produces less fatigue than computers, particularly computers with poor screen resolution. Higher levels of fatigue in the CBT condition could be attributed to the flicker of the screen, phosphorescence effects, and the glare of the screen (Krummenacher, 1996). Although access to high-resolution computers has improved in schools, Grey, Thomas and Lewis (2010) found that the ratio of high-resolution computers to students was 1 to 5. Currently, CF may be less pronounced because of improvements in screen resolution and school computer availability, but needs to be examined at state and city levels to determine geographical disparities in technology access (Hohlfeld, Rizhaupt, Dawson, & Wilson, 2017). Indeed, SES may be a large factor in which schools receive these types of computers. For individuals in schools with access to better technology, mode effects may not be an issue, while less affluent students may suffer from computer-induced deficits due to less access and overall practice, which decreases their familiarity with test mode and hence performance (Dolan, 2016).

Other technological mode effects including browsing and tactile influences may have led to lower WM scores on CBTs. Participants may have encountered spatial knowledge issues on the CBT verbal WM counting recall assessment in which students were asked to count the number of circles and triangles presented. This task may have been easier on paper because the participant could browse the exam, locate and point to the shapes using their hand, and easily remember their locations, as opposed to viewing them on a computer screen. Touching (i.e., tactile stimulation) information has also been found to lead to better memory (Bigelow & Poremba, 2014). Participants were not as able to use these tactile skills on the CBT visuospatial mazes task which required participants to recall a drawn path in a maze using their finger. It is more difficult to use your finger on a computer screen due to having to stretch your arm upward and not wishing to diminish the screen’s value.

In addition to mode effects, biological and gender differences may have played a role in performance differences across testing modalities. A meta-analysis by Maeda and Yoon (2013) summarized five key factors that may explain the differences in male and female performance: strategic, experiential, affective, biological, and test administration. Experiential refers to findings that males play more computerized video games than females, which provides them with an advantage on visuospatial tasks (Cherney, 2008). In the current study, males’ elevated performance on the mazes visuospatial task may be attributable to more experience with computer games increasing their visuospatial performance. This would also relate to Maeda and Yoon’s (2013) biological concept regarding gender differences, including hormonal and neuronal brain differences involved in different spatial awareness and mental rotation ability (Koscik, Leary, Moser, Andreasen, & Nopoulos, 2009) developmental differences for each sex, and different gender approaches to problem solving (Geiser, Lehmann, & Eid, 2006). Female participants in the current study may have had lower self-efficacy than males due to commonly held beliefs regarding visuospatial tasks, possibly creating the performance results (Moè, 2009).

CBTs may also differentially affect participants with lower automaticity on cognitive tasks. Automaticity refers to the ability to perform a task efficiently due to repeated practice, and thus streamlines information processing. People with lower levels of automaticity and WM capacity may have more difficulty transferring information from sensory memory to WM, decreasing efficiency of memory search and retrieval processes (Endres, Houpt, Donkin, & Finn, 2015). Coupled with having to take the test on the computer, having generally lower automaticity and rehearsal processes may have exacerbated the difference in performance.

Strategies for automaticity in information processing may also be underdeveloped in those who do not have access to computers, further compounding the TMF effect. Mayer and Moreno (2003) noted that by enhancing automaticity in WM, space could be devoted to other cognitive demands, which in turn reduced cognitive load and led to better learning outcomes. Instructors should activate existing knowledge prior to instruction to facilitate learning new information. Improving attentional control and enhancing automaticity for WM demands may lead to higher metacognitive control and improved WM, which ultimately could improve classroom performance.

Wallace and Clariana (2000, 2005) argued that computer-based practice exams should be readily available if CBT’s are to be ubiquitously implemented to better prepare students for computerized assessments. Students who are more competitive, lack access to computers on a regular basis, are not confident with computers, and who perform poorly on WM assessments (possibly due to biological or gender effects), should have an option for a PBT. To understand the components of WM affected by test modality, future studies should include additional measures to examine specific WM processes such as phonological awareness, different testing modalities that include pointers to locate items on the screen and on paper, collect frequency data thereof, and observe other WM assessments besides the AWMA to see if other WM assessments create similar test mode effects.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Alloway

T. P.

(2007). Automated working memory assessment. London, England: Pearson Assessment.

Alloway

T. P.

Alloway

(2010). Investigating the predictive roles of working memory and IQ in academic attainment. Journal of Experimental Child Psychology, 106, 20-29.

Alloway

T. P.

Copello

(2013). Working memory: The what, the why, and the how. The Australian Educational and Developmental Psychologist, 30, 105-118.

Alloway

T. P.

Gathercole

S. E.

Kirkwood

Elliott

(2009). The cognitive and behavioral characteristics of children with low working memory. Child Development, 80(2) 606-621.

Alloway

T. P.

Gathercole

S. E.

Pickering

S. J.

(2006). Verbal and visuospatial short-term and working memory in children: Are they separable? Child Development, 77, 1698-1716.

Archibald

Gathercole

S. E.

(2006). Short-term working memory in children with specific language impairment. International Journal of Language and Communication Disorders, 41(6), 675-693.

Baddeley

A. D.

(1996). Exploring the central executive. Quarterly Journal of Experimental Psychology, Section A: Human Experimental Psychology, 49, 5-28.

Baddeley

A. D.

(2000). The episodic buffer: A new component of working memory? Trends in Cognitive Sciences, 4, 417-423.

Barrouillet

Bernardin

Camos

(2004). Time constraints and resource sharing in adults’ working memory spans. Journal of Experimental Psychology: General, 133, 83-100.

10.

Bigelow

Poremba

(2014). Acheilles’ ear? Inferior human short-term and recognition memory in the auditory modality. PLoS ONE, 9(2), e89914.

11.

Boeve

Meijer

Albers

Beetsma

Bosker

(2015). Introducing computer-based testing in high-stakes exams in higher education: Results of a field experiment. PLoS ONE, 10(12), e0143616.

12.

Cherney

I. D.

(2008). Mom, let me play more computer games: They improve my mental rotation ability. Sex Roles, 59, 776-786.

13.

Choi

Tinker

(2002). Evaluating comparability of paper-and-pencil computer-based assessment in a K-12 setting. Paper presented at the National Council on Measurement in Education, New Orleans, LA.

14.

Colbert

(2016). Evaluating working memory: Comparing change-detection tasks and Wechsler working memory subsets in school age-children. Journal of Clinical and Experimental Neuropsychology, 39, 636-645.

15.

Collerton

Arai

Barrass

Eccles

Jagger

Kirkwood

(2007). A comparison of computerized and pencil-and-paper tasks in assessing cognitive function in community-dwelling older people in the Newcastle 85+ pilot study. Journal of the American Geriatrics Study, 55, 1630-1635.

16.

Cowan

(2005). Working memory capacity. New York, NY: Psychology Press.

17.

Daniel

M. H.

Wahlstrom

Zhang

(2014). Equivalence of Q-interactive and paper administration of cognitive tasks: WISC-V (Q-Interactive Technical Report 7). Bloomington, MN: Pearson.

18.

Dolan

J. E.

(2016). Splicing the divide: A review of research on the evolving digital divide among K-12 students. Journal of Research on Technology in Education, 48, 16-37.

19.

Endres

M. J.

Houpt

J. W.

Donkin

Finn

P. R.

(2015). Working memory capacity and redundant information processing efficiency. Frontiers in Psychology, 6, 594.

20.

Engle

R. W.

Kane

M. J.

Tuholski

S. W.

(1999). Individual differences in working memory capacity and what they tell us about controlled attention, general fluid intelligence, and functions of the prefrontal cortex. In Miyake

Shah

(Eds.), Models of working memory: Mechanisms of active maintenance and executive control (pp. 102-134). New York, NY: Cambridge University Press.

21.

Gathercole

S. E.

Alloway

T. P.

(2008). Working memory and learning: A practical guide. Los Angeles, CA: Sage.

22.

Gathercole

S. E.

Baddeley

A. D.

(1993). Phonological working memory: A critical building block for reading development and vocabulary acquisition? European Journal of Psychology and Education, 8, 259.

23.

Geiser

Lehmann

Eid

(2006). Separating “rotators” from “non-rotators” in the mental rotations test: A multigroup latent class analysis. Multivariate Behavioral Research, 41, 261-293.

24.

Grey

Thomas

Lewis

(2010). Teachers’ use of educational technology in U.S. public schools: 2009. Washington, DC: National Center for Educational Statistics, Institute of Educational Sciences, U.S. Department of Education.

25.

Hanho

(2014). A comparative study of scores on computer-based tests and paper-based tests. Behaviour & Information Technology, 33, 410-422.

26.

Hitch

G. J.

(1990). Developmental fractionation of working memory. In Vallar

Shallice

(Eds.), Neuropsychological impairments of short-term memory (pp. 221-246). Cambridge, UK: Cambridge University Press.

27.

Hohlfeld

T. N.

Rizhaupt

A. D.

Dawson

Wilson

M. L.

(2017). An examination of seven years of technology integration in Florida schools: Through the lens of the levels of digital divide in schools. Computers and Education, 113, 135-161.

28.

Hosseini

Abidin

M. J. Z.

Baghdarnia

(2014). Comparability of test results of computer based tests (CBT) and paper and pencil tests (PPT) among English language learners in Iran. Procedia: Social and Behavioral Sciences, 98, 659-667.

29.

Jacob

Berger

Hart

K. C.

Loeb

(2016). Can technology help promote equality of educational opportunities? In Alexander

Morgan

(Eds.), The Coleman report and educational inequality fifty years later (pp. 242-271). New York, NY: Russell Sage Foundation.

30.

Jaeggi

S. M.

Buschkuehl

Jonides

Shah

(2011). Short- and long-term benefits of cognitive training. Proceedings of the National Academy of Sciences in the United States of America, 108, 10081-10086.

31.

Jeong

(2012). A comparative study of scores on computer-based tests and paper-based tests. Behaviour & Information Technology, 33, 410-422.

32.

Katz

I. R.

Elliot

(2016). Information literacy in digital environments: Construct mediation, construct modeling, and validation processes. In Information literacy: Research and collaboration across disciplines (pp. 97-116).

33.

Khoshsima

Toroujeni

S. M. H.

(2017). Transitioning to an alternative assessment: Computer-based testing and key factors related to testing mode. European Journal of English Language Teaching, 2(1), 54-73.

34.

Koscik

T. O.

Leary

Moser

D. J.

Andreasen

N. C.

Nopoulos

(2009). Sex differences in parietal lobe morphology: Relationship to mental rotation performance. Brain and Cognition, 69, 451-459.

35.

Krummenacher

(1996). Zur Auswirkung der Bildwiederholfrequenz von Computermonitoren auf sakkadische Augenbewegungen bei leseähnlichen Aufgaben [On the impact of computer monitor image repetition on saccadic eye movements in reading-like tasks]. Aachen, Germany: Verlag Mainz.

36.

Leeson

(2009). The mode effect: A literature review of human and technological issues in computerized testing. International Journal of Technology, 6, 1-24.

37.

Logie

Pearson

(1997). The inner eye and the inner scribe of visuo-spatial working memory: Evidence from developmental fractionation. European Journal of Cognitive Psychology, 9, 241-257.

38.

Maeda

Yoon

(2013). A meta-analysis on gender differences in mental rotation ability measured by the Purdue Spatial Visualization Tests: Visualization of rotations (PSVT:R). Educational Psychology Review, 25, 69-94.

39.

Mayer

R. E.

Moreno

(2003). Nine ways to reduce cognitive load in multimedia learning. Educational Psychologist, 38, 43-53.

40.

Mayes

D. K.

Sims

V. K.

Koonce

J. M.

(2001). Comprehension and workload differences for VDT and paper-based reading. International Journal of Industrial Ergonomics, 28, 367-378.

41.

Moè

(2009). Are males always better than females in mental rotation? Exploring a gender belief explanation. Learning and Individual Differences, 19, 21-27.

42.

Noyes

Garland

(2003). VDT versus paper-based test: Reply to Mayes, Sims and Koonce. International Journal of Industrial Ergonomics, 31, 411-423.

43.

Noyes

Garland

(2008). Computer- vs. paper-based tasks: Are they equivalent? Ergonomics, 51, 1352-1375.

44.

Pickering

S. J.

Gathercole

S. E.

Hall

Lloyd

(2001). Development of memory for pattern and path: Further evidence for the fractionation of visuo-spatial memory. The Quarterly Journal of Experimental Psychology, 54, 397-420.

45.

Prisacari

A. A.

Danielson

(2017). Computer-based versus paper-based testing: Investigating testing mode with cognitive load and scratch paper use. Computers in Human Behavior, 77, 1-10.

46.

Raiford

S. E.

Zhang

Drozdick

L. W.

Getz

Wahlstrom

Gabel

Daniel

(2016). WISC-V coding and symbol search in digital format: Reliability, validity, special group studies, and interpretation (Q-Interactive Technical Report 12). Bloomington, MN: Pearson.

47.

Randall

West

(2014). Technology integration in schools. In Handbook of research on educational communications and technology (pp. 841-853). New York, NY: Spinger

48.

J. K.

Kontos

Guskiewicz

Mihalik

Conder

Shields

(2012). Age-related differences and reliability on computerized and paper-and-pencil neurocognitive assessment batteries. Journal of Athletic Training, 47, 297-305.

49.

Sedlacik

Cechova

(2016). Computer assisted versus traditional testing: Statistical assessment of students’ performance in different types of tests. Proceedings of the European Conference on e-Learning, 650-656.

50.

Soveri

Antfolk

Karlsson

Salo

Laine

(2017). Working memory training revisited: A multi-level meta-analysis of n-back training studies. Psychonomic Bulletin & Review, 24, 1077-1096.

51.

Wallace

Clariana

(2000). Achievement predictors for a computer-applications module delivered via the world-wide web. Journal of Information Systems Education, 11, 13-18.

52.

Wallace

Clariana

(2005). Gender differences in computer-administered versus paper-based tests. International Journal of Instructional Media, 32, 171-179.

53.

Wang

Jiao

Young

M. J.

Brooks

Olson

(2007). A meta-analysis of testing mode effects in grade K-12 mathematics tests. Educational and Psychological Measurement, 67, 219-238.

54.

Wechsler

(2008). Wechsler Adult Intelligence Scale (4th ed.). Bloomington, MN: Pearson.

55.

Ziefle

(1998). Effects of display resolution on visual performance. Human Factors, 40, 555-568.