Learning Gains From the KinderTEK ® iPad Math Program: Does Timing of a Preventative Intervention Matter?

Abstract

A quasi-experimental design in six kindergarten classrooms (n = 123 students) was used to study the effects of the KinderTEK iPad–based math program on the math achievement of students in general education classrooms. Student math outcomes in three treatment (early start) classrooms were compared to math outcomes for students in three comparison (late start) classrooms. Results suggested that relatively brief exposure to KinderTEK produced gains on distal measures of early numeracy and that, on average, timing of intervention delivery did not impact end of year math outcomes. However, exploratory analyses suggested that earlier and longer use of KinderTEK may have provided a benefit for students most at risk in math. The utility of quasi-experimental studies within an overarching research program and implications for the adoption of technology-based math programs in kindergarten classrooms are discussed.

Keywords

differentiate instruction math elementary school response to intervention (RtI)instructional technology

Educators and policy makers have been troubled by the poor math performance of U.S. students for more than 50 years (National Defense Education Act Senate Bill 3187, 1958; National Mathematics Advisory Panel, 2008; National Research Council, 2001; U.S. National Commission on Excellence in Education, 1983; The White House, Office of the Press Secretary, 2019). Rigorous academic standards (e.g., Common Core State Standards Initiative, 2010) were introduced in an attempt to dramatically improve student math performance. Outcomes on recent national and international tests reveal that the desired effects have not yet materialized, particularly for students with disabilities, those with economic disadvantages, English learners, and many minorities (Bachman et al., 2015; Kainz, 2019; McFarland et al., 2019; National Center for Education Statistics, 2019a, 2019b; Provasnik et al., 2016; Schleicher, 2019). This suggests that many students are not benefiting from current instruction to the extent needed to succeed. The consequences of such underachievement affect not only individual students throughout school and their lives (Gaertner et al., 2014) but also the larger society. As one example, students who perform poorly in math or exhibit negative beliefs about their math abilities are less likely to be interested in or pursue careers in science, technology, engineering and mathematics (STEM, e.g., Holmes et al., 2018; Huang et al., 2019; Seo et al., 2019). This reduces the nation’s capacity to engage in STEM and innovate.

Poor math performance as late as high school can be traced to poor performance in kindergarten and preschool (e.g., Duncan et al., 2007; Engel et al., 2016; Morgan et al., 2011, 2016; Watts et al., 2016). Burchinal et al. (2019), Pace et al. (2019), and others have found early math achievement to be the best predictor of later math performance. Consequently, filling in math knowledge gaps when students first enter school and giving students a strong start in math is key to students’ later success.

This can be accomplished by fully utilizing core math instruction time in kindergarten classrooms in at least three ways. First, educators can make the core lessons beneficial to all students by embedding instructional design principles found to be effective with students struggling with math (Clarke et al., 2011b; Doabler et al., 2012). These principles include focusing on core concepts and visual models; carefully sequencing instruction to ensure prerequisite skills are learned first; demonstrating problem-solving strategies and procedures; providing numerous, scaffolded opportunities to respond; and providing immediate academic feedback. Second, educators should instruct students in critical conceptual aspects of math and build procedural knowledge (Bachman et al., 2015). For example, when teaching two-digit numbers, teachers can go beyond teaching students to “count on” from an anchor number like 10 or 20. Teachers can demonstrate that in a base-ten system, ones are grouped into groups of 10 and multidigit numbers are composed of some number of tens and some ones. These concepts are critical to support later learning about larger numbers, place value, and number operations (e.g., they help students grasp that tens can be grouped into groups of hundreds when students regroup in multidigit addition; they increase the saliency of procedural aspects of number operations). Third, educators can adapt their instruction to meet individual student needs and preferences by altering variables like content and pacing, presentation, when and how students respond, how much support is provided, and the specificity and timing of feedback. Parsons et al. (2018) and van Geel et al. (2019) provide reviews of many of the ways in which teachers can adapt (i.e., individualize or differentiate) instruction within the general education classroom. However, consistently providing differentiated learning experiences to large groups of students is no easy task; it is one that few educators fully embody (Parsons et al., 2018).

Technology Offers a Potential, Though Challenging, Solution

Technology offers one way for teachers to more fully utilize math instructional time. Individual learning technologies can be designed to implement all three practices just described and more (Outhwaite et al., 2019). Educators, researchers, and leaders of educational associations recognize technology’s potential to motivate and engage students in math, assess students’ math knowledge, and simultaneously provide instruction at different levels and paces to different students (Association for Supervision and Curriculum Development, 2011; Foster et al., 2016; Haßler et al., 2016; Higgens et al., 2019; Ninaus et al., 2017; Scherer, 2011). However, many technology-based early elementary math programs do not incorporate evidence-based instructional design principles, tend to favor basic facts and procedures over conceptual knowledge, and vary in whether—and the degree to which—they offer differentiated learning experiences.

Beyond these issues, challenges abound when considering using technology with elementary-aged children. These challenges include lack of technical infrastructure and support (e.g., poor Wi-Fi, limited technical support, poorly planned technology rollouts, insufficient training, or outdated software or equipment), reluctance of teachers to incorporate technology (whether due to interest, skill, available training, or time), mismatches between tools’ content, depth, breadth, and approach (e.g., poor alignment with mandated curricula and student needs), feasibility and fit of technology with a setting and age-group, and the rapidly changing technology landscape (Foster et al., 2016; Hawkins et al., 2017; Mac Callum et al., 2014).

Thus, to develop technology that fully supports educators and students, developers must (a) think like expert teachers as they choose and design content and approaches to differentiation and (b) prioritize product feasibility and effectiveness. There are good examples of this in the nontechnology sphere (e.g., the Moving Up! Mathematics series, Early Numeracy Intervention L1), and technology-based examples are starting to emerge (see Kiru et al., 2017, for review). Educators can also find guidance in identifying and choosing technology-delivered evidence-based programs (EBPs) to fit their particular contexts (e.g., Doabler et al., 2018; Hawkins et al., 2017; National Center on Intensive Intervention, n.d.; Nelson et al., 2016; U.S. Department of Education What Works Clearinghouse, n.d.). As will be described next, our research and development team has produced such an EBP to provide differentiated early math instruction to young students.

KinderTEK®

Overview and intended uses

KinderTEK (University of Oregon Digital Press) is an iPad app that helps students develop, maintain, and become fluent in critical early math concepts and skills via an engaging learning environment (Strand et al., 2015; Center on Teaching and Learning, n.d.). It includes 51 instructional lessons aligned with standards from three of the five Common Core State Standards in Mathematics (CCSS-M) kindergarten domains and was developed through a public/private partnership and three federally funded grants. KinderTEK developers adhered to an iterative design process that was guided by expertise from educators specializing in math instruction, special educators, and game developers.

The team conceptualized KinderTEK primarily as an individualized, prevention-oriented intervention for students with or at risk for math difficulties. Explicit instructional design features found to be particularly effective for struggling students lie at the heart of KinderTEK lessons. As the app delivers instruction and practice opportunities, KinderTEK goes beyond group-level best practices by differentiating instruction for each student. Moment-by-moment, the app determines what to do next for a student based on that student’s prior interactions with the program. Such differentiation is recommended by the Office of Educational Technology (2010) and mirrors what highly effective teachers do in the classroom, particularly during one-on-one instruction (Parsons et al., 2018).

KinderTEK was also intentionally designed to be used as an intervention by older students still struggling with kindergarten-level mathematics, as a practice and fluency-building activity for on-track kindergarten students, and as a school readiness tool for preschoolers. Its architecture allows it to function as a sequenced (i.e., with a set scope and sequence) or flexible (i.e., students or teachers choose content) curricular supplement. It is not designed to replace the introduction and practice of new content during core mathematics instruction, but it is appropriate as a replacement for math worksheets or as an option during math centers and small group instruction.

Teachers can choose from three instructional modes. Sequenced mode presents all KinderTEK lessons in a prespecified sequence that fosters students’ learning of prerequisite content before more advanced content. KinderTEK use can be 10, 15, or 20 min in length. Each time a student uses KinderTEK, they work on (though perhaps do not complete) a minimum of two lessons, complete with embedded assessment and rewards. During their next KinderTEK use, the student will resume unfinished lessons and continue working at their own pace through the KinderTEK curriculum. Directed mode functions like sequenced mode except that teachers constrain lesson content to one or two categories (e.g., only lessons involving math models like 10 frames and number lines, story problems, and counting). Exploratory mode offers untimed KinderTEK use during which students choose between all lessons and reward activities and work as much or as little as they desire in each. In most cases, teachers will choose one mode for a student to use throughout the year (e.g., at-risk students are likely to use sequenced mode; advanced students are likely to use exploration mode). In some cases, however, a teacher may have student use multiple modes (e.g., use sequenced mode during intervention time and directed mode in place of a worksheet to accompany specific core instruction).

Smooth integration into classroom routines

KinderTEK is delivered through devices many schools already utilize for a multitude of educational purposes. It requires minimal teacher facilitation. Once teachers have installed KinderTEK on the iPad and entered their teacher credential (e.g., at the beginning of the school year), students can use KinderTEK independently anywhere in the school. Teachers can be confident students will have productive, engaging math experiences and be done with KinderTEK when the teacher has planned because the app adjusts instruction to match whatever time is selected (e.g., if students typically engage in math centers for 15 min, teachers can set KinderTEK to last 15 min). Automatic saving of student progress allows students to resume gameplay at a subsequent time and ensures teachers see up-to-date data. Reports of student-level progress and formative assessment data are aligned with learning goals and CCSS-M standards to facilitate instructional planning and conversations with other educators and parents.

Teachers implementing KinderTEK receive support through professional development (PD) sessions (either in-person or online), on-demand technical assistance (through online chat, email, phone or in-person visits from the KinderTEK team), emails and newsletters, and the KinderTEK website. Before starting KinderTEK, teachers are encouraged to discuss with students their classroom rules and expectations around iPad use generally (e.g., carry iPads carefully, use headphones, use the assigned app) and KinderTEK use specifically (e.g., KinderTEK is math instruction and students need to try their best to learn). Teachers are also encouraged to have students watch an introductory video about KinderTEK and to walk students through the login process the first time. If desired, individual students can complete an in-app tutorial to practice listening to—and taking turns during—instruction and selecting or dragging objects to answer math questions, manipulate their digital scrapbook, and complete puzzles and memory games.

Most students have a successful first KinderTEK experience with no pretraining. We expect this is because students are provided with printed reminders of their three-animal passcode; the KinderTEK interface is simple, uncluttered, and easy to navigate; and instruction in KinderTEK includes modeling of what students are expected to do (e.g., drag an object, touch an object) before they are asked to do it. The most common hiccup to student success early on is that students treat KinderTEK like a typical app rather than as math instruction. Some students need to be reminded to “take turns” with the virtual guide who provides instruction during their KinderTEK journey, so that they don’t get frustrated by trying to answer early. Others must be reminded to persevere when the math gets difficult. As students work, KinderTEK alerts teachers if students are struggling with a given activity, so that the teacher can confirm students understand the interface and what the system is asking them to do.

Student experience

As students use KinderTEK, the virtual guide provides explicit demonstration and modeling of math models, strategies, and skills; scaffolded practice opportunities; and informative feedback. Capitalizing on the 1:1 digital environment, students’ progress through, and experience with, KinderTEK is based on the instructional mode, which customization features are active, and students’ unique response patterns within the app.

A major premise of KinderTEK is that students should, for most of their instructional time, work on material that is challenging to them. For some students, that is KinderTEK Lesson 1, but for others, challenge comes during Lesson 3, Lesson 10, or Lesson 22. Thus, KinderTEK administers a brief pretest before each lesson to determine whether the student needs to complete that lesson or has the prerequisite skills to move on to more advanced content (i.e., the next lesson’s pretest). If a student passes the pretest, that activity is marked for later review and the student moves on to the next activity. If they do not pass, the student enters the activity’s instructional phases. During instruction, the number and content of math items, scaffolding, and feedback are dynamically adjusted in response to student performance. A student struggling with a skill (e.g., counting objects) may be asked to solve the problem in a scaffolded way (e.g., by touching to count each butterfly before selecting the answer), whereas a student who has already grasped it will be allowed to select the answer immediately. Whether students complete an activity in minutes or across multiple KinderTEK sessions, students will eventually take an activity posttest. If they pass it, a new activity will be unlocked. As noted above, in timed instruction modes, the app ensures that students encounter a mix of instruction, review, and rewards each time they use KinderTEK.

Instruction, rewards, and interface navigation are supported by clear, appropriately paced audio using age-appropriate vocabulary with which children are already familiar or have been explicitly taught through KinderTEK. Further support comes from a handful of symbols and icons recognizable to students (e.g., common math symbols; arrows to move forward and back; and mini-replicas of KinderTEK features like the activity map, scrapbook, and puzzle) and visual models and prompts (e.g., on-screen action illustrates concepts and directs students’ attention).

To enhance motivation and engagement, teachers can customize many instructional settings to fit each student’s needs and preferences. To name some examples, sessions can be 10, 15, or 20 min long or open-ended; rewards can be tied more heavily to perseverance or mastery or be balanced; time in the scrapbook and activity center can come at the end of the session or also at the midpoint. As well, attention supports like on-screen and audible indicators, progress bars, and countdown timers can be adjusted, and students can be given more or less control over their experience (e.g., activity choice, pause option, and replay option).

Current Study’s Aims and Research Questions

Throughout the development of KinderTEK, we evaluated the feasibility and effectiveness of specific features and components via stakeholder user-tests, focus groups, and small learning trials. As complete prototypes and the full product emerged, our emphasis shifted to examining the feasibility of the program as a whole and its effectiveness at improving student outcomes using larger experimental and quasi-experimental studies. Results of such studies will be useful to practitioners deciding which interventions to implement, with whom, and in which ways (Nelson & McMaster, 2019). Given the myriad ways KinderTEK can be used with different student populations and in different contexts, multiple studies of feasibility are critical. The quasi-experimental study we present here focused on one such feasibility question: To what extent does the timing of KinderTEK use in general education classrooms affect student outcomes?

Our first aim was to examine how limited, early exposure to KinderTEK impacted student performance on a distal math measure assessing early numeracy skills (i.e., number identification, magnitude comparison, and missing number). We compared use of the program in the fall to business-as-usual (BAU) math instruction to answer our first research question: To what extent did fall-to-winter gains for students who used KinderTEK during the first half of the school year differ from those of their comparison peers who had not yet used KinderTEK? We hypothesized that treatment students using KinderTEK would make greater fall-to-winter gains on a standardized test of early numeracy than their comparison group peers.

Our second aim was to contrast two potential uses of the program: (a) throughout kindergarten (the early start treatment condition) and (b) in the second half of the year (the late start comparison condition). Our second question was as follows: Did starting KinderTEK earlier in the year result in different fall-to-spring gains than starting later in the year? We hypothesized that early start students would gain at least as much on the standardized test of early numeracy as their late start peers from fall to spring. Given KinderTEK’s differentiated approach and comparison group students’ greater development and exposure to classroom instruction before they encountered KinderTEK, it was not clear whether we should expect additional months of use to result in greater outcomes for early start students.

Our third aim was to explore group differences in effects for students by initial math skill. Hypotheses for these analyses were informed by three factors. First, KinderTEK was designed primarily to address the needs of struggling students. Consequently, initial KinderTEK lessons focus on foundational skills related to number modeling, identifying, and sequencing numbers. Second, we expected lower performing students to pretest into the earlier KinderTEK lessons. They would begin receiving instruction right away. Third, we expected higher performing students to test out of KinderTEK’s initial lessons (i.e., spend days and weeks completing pretests before encountering challenging material). Given these factors, we hypothesized lower performing early start students would outperform their late start comparison peers in the winter, and that exposure to those early foundational skills would continue to provide an advantage such that early start students’ spring scores would also be higher. We hypothesized higher performing early start students would show limited gains initially (i.e., perform similarly to their late start comparison peers in the winter). As they began engaging in KinderTEK instruction, we expected higher performing early start students to gain skills and outperform their late start comparison peers in the spring.

Method

Design

This study was part of a larger product development study included in our Office of Special Education Programs–funded work. That larger study required all participants to begin using KinderTEK starting by February and represented the first sustained implementation of the full KinderTEK product. As that study ramped up, we had the opportunity to implement the quasi-experimental study reported here, namely, we compared student math performance of three classes that started KinderTEK relatively early in the school year (treatment; early start) to the performance of three classes that did not start using KinderTEK until later in the year (comparison; late start). As shown in Figure 1, assessments were administered in the fall, before any classrooms used KinderTEK (T1); midway through the year, after the treatment students had used KinderTEK for several weeks, but the comparison group had not (T2); and at the end of the school year when both the early start and late start groups had used KinderTEK (T3).

Figure 1.
Study design.

Setting and Participants

The study was conducted in six kindergarten classrooms in two elementary schools in a single Pacific Northwest school district. One school had four kindergarten classes and the other had two. Teachers at both schools were ready for the larger implementation study earlier than expected, thus conditions were ripe for this quasi-experimental study. Classrooms whose teachers were ready to begin KinderTEK early (two in the larger school and one in the smaller school) were considered treatment (early start) classrooms. The remaining classrooms (again, two in the larger school and one in the smaller school) served as comparison (late start) classrooms. All classrooms used the same core math curriculum, and, within school, teachers had grade-level planning meetings and introduced lessons at the same pace using the same scripting, workbooks, worksheets, and resources. Following approved institutional review board procedures, all kindergarten students in the six classrooms (n = 135) were invited to participate, and 129 consented and completed the pretest assessment. Six students in the late start condition were excluded from this study because they also participated in a separate single-subjects study that gave them access to KinderTEK content earlier than their late start peers, resulting in a study sample of 123 students, 64 in the early start condition and 59 in the late start condition. Demographic data were available for all but one student: 50.4% were female, 15.4% were English learners, 10.6% were eligible for special education services, 57.7% were White, 26.8% were Hispanic, and 12.2% were of two or more race/ethnicity categories.

Intervention

KinderTEK (University of Oregon Digital Press) was the intervention during this study. Teachers were asked to implement KinderTEK in sequenced mode with all their students. We stated a usage goal of 15 min per day, 3 days per week, but did not enforce adherence to that schedule. Students were expected to use KinderTEK individually at their desks or at a station in the classroom. Teachers had the ability to change settings (e.g., mode, minutes, on-screen student supports, and reward timing), but were encouraged to use the default sequenced mode throughout the study.

PD and ongoing support

During this study, KinderTEK PD was provided primarily in person. Thus, prior to implementing KinderTEK in their classrooms, teachers were provided with a full day of in-person PD. The PD was led by the research team and program authors and was intentionally structured, so that as teachers shared their instructional goals and challenges related to kindergarten math, the presenters introduced and explained how the instructional principles and architecture underlying KinderTEK would help them meet students’ needs. Educators also played KinderTEK to better understand how students could experience the game, practiced performing common teacher tasks related to the program (e.g., how to add students and review progress reports), and were given time to plan how they would integrate KinderTEK into their classroom instruction. Research team members provided in-class support to teachers during their first days of KinderTEK use. As well, teachers had access to online KinderTEK supports (e.g., how-to guides and brief videos about using reports) and on-demand support (in person and virtually) from the research team throughout the study.

Treatment condition

Teachers in early start classrooms were asked to supplement their regular math instruction by having all students use KinderTEK starting in the fall. Two of the three teachers in early start classrooms received the PD in early November and began using KinderTEK in mid-November. The other teacher had completed the same PD and implemented KinderTEK for a short time at the end of the previous school year, so began implementing KinderTEK in mid-October, before the PD for the other early start teachers took place. Fidelity of implementation and dosage details for the analytic sample are provided in the Results section.

Comparison condition

Teachers in late start classrooms proceeded with their regularly planned math instruction in the fall. In mid-February, comparison teachers (and additional teachers taking part in the larger KinderTEK study) received the same PD that treatment teachers had received in the fall and their students began using KinderTEK. All app content and supports provided during comparison students’ use of the app were identical to those available to students in the treatment condition. Fidelity of implementation and dosage details for the analytic sample are provided in the Results section.

Measures

Students completed the Assessing Student Proficiency in Early Number Sense (ASPENS; Clarke et al., 2011a). The ASPENS assessment is a standardized, individually administered test of early number sense consisting of three timed subtests, each taking 1–2 min. In kindergarten, ASPENS assesses students’ ability to say the names of numerals (number identification), compare two numerals and determine which is greater (magnitude comparison), and identify the missing numeral in a string of three numerals (missing number). These skills are the focus of earlier KinderTEK lessons and certainly kindergarten core instruction; thus, the assessment is well-aligned with the content to which all students were exposed during this study. Individual subtest scores are weighted to form an overall ASPENS composite score. The ASPENS authors report test–retest reliability ranging from .71 to .90 and concurrent and predictive validity with the TerraNova Third Edition as between .57 and .63 (Clarke et al., 2012). The ASPENS Administrator’s Manual also provides normative benchmark information based on a sample of 353 kindergarten students in six schools from four districts in Ohio and California (Clarke et al., 2012). The ASPENS measures were administered at T1 (as a pretest measure and to categorize students by level of math risk), T2 (as an interim measure), and T3 (as a posttest measure).

Statistical Analysis

As a first step in the analytic process, each of the three ASPENS subtests and the ASPENS composite score were summarized descriptively (see Table 1) and evaluated for baseline equivalence. No statistically significant differences were found between the two conditions on any of the measures at T1. Gain scores based on differences from T1 to T2 and from T1 to T3 were calculated and evaluated for assumptions of normality using the R statistical programming language (R Core Team, 2018) package compareGroups (Subirana et al., 2014). For normally distributed measures, differences in gains were evaluated using an analysis of variance approach. For nonnormally distributed measures, differences were evaluated using the Kruskal–Wallis rank sum test. To account for the inclusion of multiple tests on the same sample, all p values were also adjusted using the Benjamini and Hochberg (1995) correction for false discovery. Estimates of effect size were computed using Hedges’s g (1981), which pools variances on the assumption of equal population variances, using n − 1 for each sample. Statistical significance was specified a priori at p < .05.

Table 1.
Early Start Versus Late Start Student Math Performance at T1, T2, and T3—Overall and by ASPENS Risk Category at T1.

Measure ASPENS Risk Group Start Type N T1 T2 T3 T1–T2 Gains T1–T3 Gains

M SD M SD M SD M SD M SD

ASPENS composite All students Early 58 41.09 35.50 76.86 44.86 101.76 38.51 35.77 26.31 61.25 29.70

Late 56 37.19 37.67 61.25 43.71 99.39 52.81 24.06 26.96 59.43 34.88

Intensive Early 18 6.28 5.47 39.61 23.27 71.88 21.18 33.33 24.26 65.82 21.60

Late 20 2.74 4.28 21.30 24.94 51.56 49.60 18.56 22.70 49.35 47.72

Strategic Early 26 36.77 11.68 73.38 32.95 103.04 30.56 36.62 27.27 65.84 25.81

Late 24 36.73 13.17 75.54 31.86 107.30 29.56 38.82 24.10 70.07 23.56

Benchmark Early 14 93.89 22.52 131.21 30.01 138.38 38.56 37.33 28.69 46.45 41.28

Late 12 95.52 28.84 99.25 37.85 148.00 39.25 3.73 23.44 52.48 29.59

ASPENS MC All students Early 58 6.21 7.46 13.55 9.77 18.49 9.57 8.69 7.08 12.33 8.87

Late 56 5.46 7.74 10.38 10.05 18.31 11.63 6.16 8.18 12.43 9.63

Intensive Early 18 1.17 1.47 5.89 4.03 11.47 6.47 5.22 4.12 10.24 6.59

Late 20 0.15 0.49 2.80 5.03 8.75 10.34 2.75 5.06 8.69 10.36

Strategic Early 26 3.62 3.13 12.15 7.41 19.12 7.46 7.42 7.11 15.36 7.38

Late 24 4.62 3.19 13.79 8.15 20.43 8.56 9.58 7.87 15.78 8.68

Benchmark Early 14 17.5 6.02 26.00 6.30 26.46 10.26 15.50 5.52 9.23 12.32

Late 12 16.00 10.21 16.17 12.50 27.00 9.86 5.00 10.61 11.00 8.83

ASPENS MN All students Early 58 4.86 4.49 9.29 5.89 11.91 5.18 4.43 3.83 7.11 4.00

Late 56 4.21 4.91 7.14 5.56 11.24 6.55 2.93 4.12 6.73 4.22

Intensive Early 18 0.67 1.03 4.50 3.63 8.00 4.76 3.83 3.96 7.29 4.74

Late 20 0.05 0.22 2.20 2.19 5.75 5.65 2.15 2.21 5.69 5.59

Strategic Early 26 4.73 2.62 9.69 4.98 12.52 3.92 4.96 3.87 7.60 3.00

Late 24 4.21 2.86 8.67 4.06 11.61 4.25 4.46 4.60 7.48 3.42

Benchmark Early 14 10.50 3.92 14.71 4.83 15.85 4.56 4.21 3.75 5.92 3.33

Late 12 11.17 4.34 12.33 5.73 17.83 4.90 1.17 4.75 6.67 3.47

ASPENS NI All students Early 58 17.41 14.49 28.72 16.39 38.20 14.41 11.31 12.10 21.13 13.44

Late 56 16.52 16.05 24.32 16.70 37.86 19.05 7.80 9.62 20.08 13.63

Intensive Early 18 2.50 3.73 17.44 11.67 30.82 7.19 14.94 11.82 28.76 8.52

Late 20 2.35 4.28 10.65 12.89 21.19 20.05 8.30 10.71 19.25 18.32

Strategic Early 26 17.85 6.94 26.58 12.84 36.76 14.37 8.73 11.16 19.24 12.70

Late 24 17.50 12.37 28.62 13.55 41.22 10.56 11.12 8.06 23.04 10.68

Benchmark Early 14 35.79 11.72 47.21 11.38 50.62 14.28 11.43 13.73 14.77 16.13

Late 12 38.17 7.60 38.50 11.09 53.67 13.36 0.33 6.61 15.5 10.79

Note. The full sample (“all students,” bolded) is comprised of students in the intensive, strategic, and benchmark categories. MC = magnitude comparison; MN = missing number; NI = number identification; ASPENS: Assessing Student Proficiency in Early Number Sense.

Follow-up exploratory analyses were then conducted to investigate whether there were group differences in gains by initial skill level, as measured by ASPENS composite scores at T1. Gains were summarized descriptively by ASPENS performance level, and effect sizes of group differences were calculated using Hedges’s g.

Results

Attrition and Missing Data

Across conditions, attrition and missing data impacted 7% of students at T2 and 14% of students at T3. Specifically, among students who were assessed at T1, ASPENS data were unavailable at T2 for 6 students and at T3 for 14 students, and 3 additional students had no KinderTEK use at any point in the year. Thus, our analyzed sample sizes were 114 students at T2 (93% of participants) and 106 students at T3 (86%). By condition, six treatment students (9.4%) and three comparison students (5.1%) were excluded from analyses of T2 data, and nine treatment students (14.1%) and eight comparison students (13.6%) were excluded from analyses of T3 data. These rates of attrition fall within the acceptable range for both overall and differential attrition as defined by the What Works Clearinghouse guidelines for assessing attrition bias (Institute of Education Sciences, 2014).

Levels of Students’ Mathematical Risk

Most students are not identified as having learning disabilities before or during kindergarten. Thus, performance on the ASPENS assessment at T1 was useful not only as a continuous variable utilized in primary outcomes analyses but also as a categorical variable used in our exploratory analyses. ASPENS provides three performance-level categories, based on students’ composite score (Clarke et al., 2012). Students at benchmark are performing at the level expected and are considered on track; those in the strategic zone are at risk for being below benchmark at end of year; and those in the intensive zone are clearly not on track to reach end-of-year benchmarks (i.e., they are performing below the level expected). The two conditions had similar proportions of students in each ASPENS performance level. The early start condition included 18 students (31%) identified as needing intensive support, 26 students (45%) identified as needing strategic support, and 14 students (24%) identified as needing benchmark support. The late start condition included 20 students (37%) identified as needing intensive support, 22 students (41%) identified as needing strategic support, and 12 students (22%) identified as needing benchmark support.

Fidelity of Implementation and Usage

As this was the first sustained, full-scale implementation of the complete KinderTEK product by teachers, fidelity of implementation was not a major focus of this or the parent study. Nonetheless, research staff regularly visited implementing classrooms to identify major implementation roadblocks and contextual characteristics that could influence further program revisions. KinderTEK gameplay data (i.e., log data) were collected to document dosage.

Treatment condition

During classroom visits (five to six per treatment classroom), 100% of students experienced KinderTEK individually, with headphones, at their desks, or at a station in the classroom. Log data confirmed that students in the treatment condition began using KinderTEK when requested (median start date = November 17, minimum = October 17, maximum = November 21) and indicated that 75% of students engaged in instructional activities between 10 and 15 min each session, confirming that teachers typically set students’ gameplay to last for 15 min, as requested. Most students experienced sequenced mode, as intended. Students in one class (33% of treatment students) used a combination of directed and exploration mode for several days in the latter part of the year.

Usage was far less than suggested and far more variable across classes and students than desired. Students in treatment classrooms used KinderTEK on an average of 28 days (standard deviation [SD] = 7.58, minimum = 12, maximum = 39) across an average of 24 weeks (SD = 2.99, minimum = 15, maximum = 39). Not surprisingly, given this dosage, students tended not to progress through the entire KinderTEK curriculum. They encountered an average of 23 different KinderTEK lessons (SD = 8.85, minimum = 9, maximum = 47), which represents just under half of the KinderTEK lessons (i.e., students encountered only the earliest content). As these standard deviations and ranges illustrate, students did show substantial variability in usage. Notably, even those students who had the highest usage encountered KinderTEK less than twice per week, and only three students encountered at least 80% of KinderTEK lessons.

During the PD and throughout implementation, we encouraged teachers to monitor students’ use of KinderTEK and review student reports. Monitoring could help teachers determine whether students (a) were working productively or whether they could benefit from technical assistance (e.g. adjusting headphones, replacing iPads with dead batteries) or KinderTEK setting adjustments (e.g., shorten KinderTEK sessions, increase the reward frequency, change modes), (b) needed assistance staying on task (in which case teachers could turn on the iPad’s guided access feature), or (c) could use teacher help with a skill with which they were continually struggling. Log and observation data indicated that treatment teachers provided technical assistance but seldom changed the default settings.

Comparison condition

A review of KinderTEK logs confirmed there was no treatment diffusion: as planned, KinderTEK was not accessed by students in the comparison group until after the winter break. The median KinderTEK start date for students in the comparison condition was February 15 (minimum = February 13, maximum = March 2). As with treatment students, researchers noted that comparison students appropriately engaged with KinderTEK, and 100% of students were using KinderTEK individually, with headphones, at their desks, or at a station in the classroom. Fewer visits (one to three per classroom) occurred in comparison classrooms because KinderTEK was implemented across fewer months and because visits were more difficult to schedule for some of these teachers. Similar to treatment students, log data revealed that comparison students had low and variable exposure to KinderTEK (see Table 2 for class-level results). Students in comparison classrooms used KinderTEK an average of 22 days (SD = 11.44, minimum = 1, maximum = 44) across an average of 11 weeks (SD = 2.49, minimum = 1, maximum = 15) and encountered an average of 22 different KinderTEK lessons (SD = 10.96, minimum = 2, maximum = 47). Notably, their total exposure to KinderTEK content was comparable to that of the treatment group, despite starting later in the year. Log and observation data indicated that like treatment students, 75% of comparison students engaged in instructional activities between 10 and 15 min per session and that like treatment teachers, comparison teachers provided technical assistance but seldom changed the default settings.

Table 2.
Average KinderTEK Usage by Classroom for Each Instructional Mode and Overall.

Start Type Class (n) KinderTEK Mode ^a M Total Days of Use M Cal. Weeks of Use M Days of Use per Cal. Week M Total Minutes of Instruction ^b M Lessons Seen in Mode M Different Lessons Seen

Early Class A (n = 21) Sequenced 20.71 — — 247.47 20.57 —

Overall 20.71 25.98 0.79 247.47 — 20.57

Early Class B (n = 18) Sequenced 31.50 — — 374.92 18.94 —

Overall 31.50 21.94 1.45 374.92 — 18.94

Early Class C (n = 19) Sequenced 25.11 — — 270.13 23.79 —

Directed 6.21 — — 51.72 5.79 —

Exploration 2.26 — — 16.79 8.16 —

Overall 33.42 25.11 1.34 338.64 — 29.63

Late Class D (n = 15) Sequenced 37.93 — — 460.43 31.93 —

Overall 37.93 12.78 2.97 460.43 — 31.93

Late Class E (n = 17) Sequenced 22.12 — — 230.25 24.00 —

Overall 22.12 12.52 1.77 230.25 — 24.00

Late Class F (n = 22) Sequenced 12.09 — — 136.90 13.64 —

Screening ^c 0.14 — — 0.94 0.36 —

Overall 12.18 9.53 1.53 137.84 — 13.64

Note. Total usage (“overall,” bolded) is the sum of use in the other modes.

^a Sequenced mode presents the KinderTEK curriculum such that students learn prerequisites before encountering more advanced content. Sessions are timed (e.g., 15 min). Directed mode allows teachers to select one to two content areas that are presented in a sequenced fashion in timed sessions. If students master the activities in both the assigned content areas, they review activities until the teacher makes a change. During this study, one teacher activated directed mode and selected “story problems” (which appear early in sequenced mode) and “decomposition of teens” (which are the last lessons in sequenced mode). Exploration mode is untimed, and students have control over what to work on (e.g., any math-focused lesson or reward activity) and for how long; they can exit lessons at any time. Screening mode is a noninstructional, timed mode that consists of just pretests for each activity. No feedback is provided.

^b Minutes of instruction refers to time spent in instructional activities (i.e., excluding transitions and reward activities that are part of the 15-min structured experience).

^c Only one student in this class used screening mode, for a total of 3 days.

Research Question 1

Our first research question examined whether students using KinderTEK in the fall gained more between fall (T1) and winter (T2) compared to their BAU comparison peers (i.e., students who had not yet used KinderTEK). As shown in Table 1, the treatment group had higher T1–T2 gains for all measures, suggesting that student use of KinderTEK resulted in learning across multiple components of early numeracy skills that transfers to a distal measure of math proficiency (ASPENS). As shown in Table 3, prior to applying the Benjamini and Hochberg (1995) correction for false discovery, the difference between conditions from T1 to T2 was statistically significant for the ASPENS composite score, F(1, 112) = 5.50, p = .021, Hedges’s g = 0.44, and for the magnitude comparison (Kruskal–Wallis χ² = 4.79, df = 1, p = .029, Hedges’s g = 0.33) and missing number subtests, F(1, 112) = 4.07, p = .046, Hedges’s g = 0.38. After applying the correction for false discovery, the p values for all four measures were smaller than the corresponding critical values. For all measures, reported effect sizes represent relatively small effects. Results for all measures are shown visually in Figure 2.

Table 3.
Results of T1–T2 Gain Score Analyses.

Measure Effect Test p BH Adj. p BH Critical Value Hedges’s g [95% CI]

ASPENS Composite 5.503 .021 .057 .025 .44 [.06, .81]

ASPENS Magnitude Comparison 4.791 .029 .057 .050 .33 [−.04, .70]

ASPENS Missing Number 4.070 .046 .061 .075 .38 [.00, .75]

ASPENS Number Identification 2.919 .090 .090 .100 .32 [−.05, .69]

Note. Control n = 58 and treatment n = 56. BH adj. p = adjusted p value using Benjamini and Hochberg’s (1995) correction for false discovery; BH critical value = value against which to compare the BH adj. p allowing for a false discovery rate of .10; CI = confidence interval; ASPENS: Assessing Student Proficiency in Early Number Sense.

Figure 2.
Early start versus late start student math performance at T1, T2, and T3 for all students by condition.

Research Question 2

The second research question asked whether starting KinderTEK earlier in the year resulted in larger beginning (T1) to end (T3) of year gains than starting later in the year. Like the trends observed from T1 to T2, the early start (treatment) group had larger T1–T3 gains for three of the four measures (see Table 1) although none were statistically significant. Indeed, students who started using KinderTEK later in the year made greater gains from T2 to T3, such that they ended the year, on average, with similar skills as their peers who started using KinderTEK earlier (depicted in Figure 2).

Differences by Skill Level

Exploratory results by risk category are based on the descriptive statistics reported in Table 1 and are depicted visually in Figure 3. Here, we describe the results for the ASPENS composite score, but each subtest showed similar patterns. As predicted, differences in gains from T1 to T2 were most pronounced for students in the intensive category (mean gain of 33.33 for the intensive treatment group compared to a mean gain of 18.56 for the intensive comparison group, Hedges’s g = 0.62, a medium effect). In contrast, students in the strategic category showed nearly identical gains from T1 to T2 in both conditions (mean gain of 36.62 for the strategic treatment group compared to a mean gain of 38.82 for the strategic comparison group, Hedges’s g = −0.08, a negligible effect). Unexpectedly, however, students in the benchmark category showed a trajectory from T1 to T2 that was similar to—but more pronounced than—that of students in the intensive category (mean gain of 37.33 for the benchmark treatment group compared to a mean gain of 3.73 for the benchmark comparison group, Hedges’s g = 1.23, a large effect).

Figure 3.
Early start versus late start student math performance at T1, T2, and T3 for ASPENS by ASPENS risk category at T1. ASPENS: Assessing Student Proficiency in Early Number Sense.

We also examined gains from T1 to T3, a time span that reflects similar overall levels of KinderTEK exposure, but concentrated in the later part of the school year for late start students. Over the course of the year, students in the early start intensive category made greater gains—and ended the year higher—than their late start intensive peers (mean gain of 65.82 for the intensive treatment group compared to a mean gain of 49.35 for the intensive comparison group, Hedges’s g = 0.44, a small effect), with nearly all of that difference occurring between T1 and T2. In contrast, students in the early start strategic and early start benchmark categories exhibited slightly lower gains from T1 to T3 than their comparison peers (mean gain of 65.84 for the strategic treatment group compared to a mean gain of 70.07 for the strategic comparison group, Hedges’s g = −0.17, a negligible effect; mean gain of 46.45 for the benchmark treatment group compared to a mean gain of 52.48 for the benchmark comparison group, Hedges’s g = −0.16, a negligible effect).

Discussion

This quasi-experimental study was designed to evaluate the effects of the KinderTEK iPad–based math program on the math achievement of students in general education classrooms. Student math performance in three treatment (early start) classrooms was compared to student math performance in three comparison (late start) classrooms to determine (a) whether brief exposure to KinderTEK meaningfully impacted student math achievement and (b) whether starting KinderTEK in the fall and continuing through the school year conveyed an advantage over using KinderTEK only in the second half of the year.

Findings related to the first research question suggest that kindergarten students do benefit from using KinderTEK relatively early in the school year. Given that the concept of numerical magnitude is a major focus of early KinderTEK lessons, the finding that treatment students outperformed their peers on the ASPENS magnitude comparison measure by a third of a SD is promising because it suggests that KinderTEK impacts a major skill it was designed to affect and ostensibly provides a strong foundation for later math concepts. It is noteworthy that even relatively brief KinderTEK use is also linked to gains of nearly half a SD on a multicomponent distal math measure (i.e., the ASPENS composite). These findings suggest KinderTEK is helping students learn relevant material in a meaningful way, beyond what is being taught in the core curriculum.

Analyses related to the second research question revealed that both early and late start students made comparable gains on the multicomponent distal math measure by the end of the school year. This suggests that KinderTEK can be successfully implemented in more than one way (i.e., early vs. later in the school year). Similar dosage has similar benefits regardless of timing. KinderTEK appears to be flexible enough to accommodate educators wishing to take additional time to identify kindergarten students who may benefit from supplementary math instruction. That students starting KinderTEK later in the year ended year on par with their early start peers may be accounted for by patterns of student use during the study. Dosage was much lower than expected and it is unclear whether both groups’ gains would have been different if usage had been higher. Indeed, as Pianta et al. (2019) note, “…rigorous instructional content is not sufficient, in and of itself, to produce improvements in child outcomes. Children also have to be exposed to a sufficient dosage of such content through instruction” (p. 2). Alternatively (and described next), KinderTEK’s design may be at play.

Recall that each KinderTEK lesson begins with a pretest that determines whether a student will complete a particular lesson’s instructional phase. Students with a good grasp of number sense typically spend many days of KinderTEK briefly experiencing and testing out of material they already know. Given that students in both conditions were (a) getting older and (b) receiving core math instruction throughout the study, we expected late start students to know more math when they began using KinderTEK in the winter than their early start peers did when they began KinderTEK in the fall. This was certainly true; late start students had an average winter (T2) ASPENS composite score of 21.3, compared to the average fall (T1) ASPENS composite score of 6.28 for early start students. Thus, we expected late start students to test out of more KinderTEK material and to move more rapidly through the KinderTEK curricular sequence than did their peers whose performance indicated that they needed the instruction offered in those lessons. This pattern confirms that KinderTEK functioned as intended (i.e., it systematically adjusted instruction based on student performance, allowing for a dynamic instruction approach), and the results of this study supplement a growing body of research on delivery models that modify instruction based on student response and observe positive learning gains (Al Otaiba et al., 2014; Coyne et al., 2013).

Exploratory analyses of gains by initial risk suggest students who are most at risk for math difficulty benefit from using KinderTEK earlier in the year (i.e., using KinderTEK as a preventative intervention). Students in the intensive category at T1 who used KinderTEK earlier in the year made greater gains by the end of the year than their intensive peers who started using KinderTEK later in the year (see Figure 3). This may be because these students had time to experience and master more of the KinderTEK curriculum and thus learned more math from it than did their peers. It may be because early use of KinderTEK helped these students gain more from their core classroom instruction than their peers. Regardless, this finding maps well onto the reading response-to-intervention literature that indicates providing immediate intervention (i.e., Tier 2 or 3 instruction) to students who perform poorly on a screening measure results in both immediate and cumulative effects compared to intervening only after evaluating students’ response to Tier 1 instruction (Al Otaiba et al., 2014; Connor et al., 2007). In a more recent study, researchers found a similar effect in kindergarten mathematics (e.g., Shanley et al., 2018), suggesting that if the probability of a student’s response to instruction can be accurately predicted from screening scores, it may be more productive and efficient to provide increasingly intensive interventions earlier in the school year rather than waiting to confirm that Tier 1 instruction is not sufficient.

Risk-subgroup analyses also showed that strategic students who used KinderTEK early in the year and those who started later in the year had nearly identical patterns of performance on the ASPENS assessment at all three time points. It may be that teachers are providing core class instruction particularly well-aligned with this (the largest) group’s needs, such that any additional effects of KinderTEK are negligible. Given the usage results, it is also quite possible that—regardless of start date—to make meaningful progress through KinderTEK and reach more advanced content and fluency, strategic students must use KinderTEK more often than they did during this study.

Benchmark students who used KinderTEK early in the year made greater gains than their BAU peers between T1 and T2. We hypothesize this is because the KinderTEK curriculum provided opportunities to review and build fluency on material benchmark early start students already knew as well as opportunities to learn material in KinderTEK that was not yet being covered in the core curriculum. Between T2 and T3, their performance leveled out while their late start peers' performance rose, such that by the end of the school year, both groups were performing similarly. This may be because core math instruction in the second half of the year leveled the playing field (such that all benchmark students now had substantial experience with skills tested through ASPENS), because of diminished interest or motivation of benchmark early start students to play KinderTEK, or because of high interest and motivation to learn by benchmark late start students.

Limitations and Future Research

Individualized, differentiated learning environments such as KinderTEK present challenges to traditional or small-scale research designs. Such programs can be used in many more ways than can be feasibly tested in any single study, and dosage can be defined in many ways (e.g., days of use, minutes of use, content coverage, time spent per skill, and more). In fact, this is a constant tension related to evaluating flexible, individualized instructional systems. The power of such systems is likely to come from their ability to adapt to each student’s learning needs and, in the case of KinderTEK, to teachers’ classroom goals and practices. Yet, obtaining large enough sample sizes and documentation of such variability represents a massive undertaking out of reach of many research teams.

More rigorous, large-scale efficacy studies of KinderTEK are underway, but the study presented here was quasi-experimental in nature and involved a relatively small number of participants in only six classrooms. As depicted in Figure 1, we examined KinderTEK use starting earlier in the year compared to use of KinderTEK only later in the year. Although there were measurable differences on a standardized assessment, this design paints only a partial picture into how KinderTEK would look (and have effects) under authentic conditions. This was most teachers’ first experience with KinderTEK; students used KinderTEK less often than recommended and inconsistently; and there was large variation across classes. We plan to conduct analyses of effects controlling for dosage (operationalized as days of use as well as progress and mastery in KinderTEK) to better understand this rich data set.

It would be interesting for future research to compare outcomes of students under these early start and late start conditions with those who did not use KinderTEK at any point in the year and to those who only used KinderTEK at the beginning of the year. Designing studies so that students use KinderTEK for a set number of days (e.g., 40 days of use) rather than over particular spans of time (e.g., “fall”), randomizing at the student level, and involving a much larger sample would also be informative, though more logistically challenging in school settings. Regardless of study designs, future work on this data set and those to come will explore in more detail the tremendous variability in data related to usage, perceptions, and math performance (including measures targeting a greater breadth of math concepts and skills than included on ASPENS) for full samples and for subgroups defined by classroom or risk status.

Systems like KinderTEK are attractive for teachers because of the options they provide. As teachers implement KinderTEK, they get more comfortable and become more interested in exploring and using those options. When done with care, such experimentation is an advantage and speaks to how KinderTEK can be used in different ways in the same classroom. As occurred in this study, it also complicates interpretations. Specifically, one of the three early start teachers switched all her students (i.e., a substantial portion of our early start sample) to directed mode for approximately the last third of their KinderTEK use. Modules activated by the teacher were “story problems” (six lessons; likely not new content for many of the students as these lessons are typically early in the KinderTEK sequence) and “decomposition of teen numbers” (two lessons; likely too advanced for many of the students as they are the last lessons in the KinderTEK sequence). Although the class may have benefited from story problem practice and exposure to the decomposition concepts, the teacher did not, as recommended, activate new modules when students mastered those they had been assigned. Because of this, students repeated the same activities many times. We do not know how students in that class would have performed on the T3 assessment if they had continued to progress in KinderTEK’s recommended order, constantly building on their existing knowledge while being pushed to gain new skills. To alleviate this in the future, we have refined teacher alert systems and are developing just-in-time, mini-PD modules to help teachers productively embrace KinderTEK’s flexibility. We continue to reinforce to teachers the importance of monitoring KinderTEK student data to inform implementation and their own instruction.

Given the limited research and findings for educational technology thus far (Kiru et al., 2017; Young et al., 2012), it is imperative to investigate more deeply the conditions under which technology is deployed and utilized and the extent to which it is effective. Thus, during such larger studies, it would also be useful to study whether and to what extent preparedness, implementation, and engagement variables mediate or moderate student outcomes and document the effects of specific program improvements to inform future development projects. We have attempted to do this during our iterative KinderTEK development. For example, our earliest KinderTEK prototype was subject to usability and feasibility testing before we conducted a small-scale randomized controlled trial (Strand Cary & Crowley, 2018). Since expanding the program, we have examined the effectiveness of specific supports and customization options (e.g., Shanley et al., 2019), the feasibility of the program for English learners in a summer school program (Strand Cary & Watkins, 2018), improvements in engagement and instructional time as a function of program improvements (Strand Cary & Kennedy, 2018), and supportiveness and effectiveness of varying PD approaches. We are currently conducting a federally funded KinderTEK efficacy study in two states. Beyond understanding whether and why KinderTEK is effective, by further exploring implementation data from all these studies, we hope to have a better understanding of what KinderTEK customization options and dosage are optimal and reasonable for different groups of students and contexts. This will inform our recommendations to educators. We believe comprehensive investigations like these are crucial for our team and all developers to help ensure the promise of educational technologies for the classroom by more effectively and efficiently developing, studying, refining, implementing, and scaling up effective educational approaches for a range of learning objectives and unique contexts.

Conclusion

Practical solutions for providing all students with individualized instruction within core instruction are critical as schools consider how best to provide support given limited resources. The implementation in general education kindergarten classrooms described here approximated authentic use of iPad apps and supplementary math instruction in kindergarten settings. KinderTEK is likely to be used most consistently and for the longest period of time as a supplement to typical instruction for students struggling with math, but it will also serve as a practice tool for students on track in their math learning or to identify and fill gaps in knowledge for students intermittently struggling. As students progress through KinderTEK, the program differentiates instruction so that students only work on content for which they need instruction or practice. If students have mastered the concepts covered in KinderTEK, they can use the program to build fluency. As the results of this study show, KinderTEK can be used effectively at different times in general education classrooms. Even the benchmark students showed initial gains after using KinderTEK, suggesting they benefitted from the program’s differentiation. That said, some student subgroups may benefit more from starting KinderTEK earlier in the year. In this study, early start intensive students’ gains were more linear than those of benchmark students’, but they ended higher than their late start comparison peers, suggesting an ongoing benefit of engaging with KinderTEK’s differentiated instruction and content.

Given the longitudinal development of math proficiency, it is imperative that all students exit kindergarten with a solid math foundation, so that they can engage successfully with later elementary school math content (Geary, 1993; Jordan et al., 2002). To simultaneously prevent long-term difficulties in math while potentially accelerating the learning of all kindergarten students, it is critical to use quality programs that meaningfully improve student outcomes and fully utilize available math instruction time by differentiating instruction at every opportunity. Technologies like KinderTEK are a promising means of achieving this goal.

Measure	ASPENS Risk Group	Start Type	N	T1	T2	T3	T1–T2 Gains	T1–T3 Gains
ASPENS composite	All students	Early	58	41.09	35.50	76.86	44.86	101.76	38.51	35.77	26.31	61.25	29.70
Late	56	37.19	37.67	61.25	43.71	99.39	52.81	24.06	26.96	59.43	34.88
Intensive	Early	18	6.28	5.47	39.61	23.27	71.88	21.18	33.33	24.26	65.82	21.60
Late	20	2.74	4.28	21.30	24.94	51.56	49.60	18.56	22.70	49.35	47.72
Strategic	Early	26	36.77	11.68	73.38	32.95	103.04	30.56	36.62	27.27	65.84	25.81
Late	24	36.73	13.17	75.54	31.86	107.30	29.56	38.82	24.10	70.07	23.56
Benchmark	Early	14	93.89	22.52	131.21	30.01	138.38	38.56	37.33	28.69	46.45	41.28
Late	12	95.52	28.84	99.25	37.85	148.00	39.25	3.73	23.44	52.48	29.59
ASPENS MC	All students	Early	58	6.21	7.46	13.55	9.77	18.49	9.57	8.69	7.08	12.33	8.87
Late	56	5.46	7.74	10.38	10.05	18.31	11.63	6.16	8.18	12.43	9.63
Intensive	Early	18	1.17	1.47	5.89	4.03	11.47	6.47	5.22	4.12	10.24	6.59
Late	20	0.15	0.49	2.80	5.03	8.75	10.34	2.75	5.06	8.69	10.36
Strategic	Early	26	3.62	3.13	12.15	7.41	19.12	7.46	7.42	7.11	15.36	7.38
Late	24	4.62	3.19	13.79	8.15	20.43	8.56	9.58	7.87	15.78	8.68
Benchmark	Early	14	17.5	6.02	26.00	6.30	26.46	10.26	15.50	5.52	9.23	12.32
Late	12	16.00	10.21	16.17	12.50	27.00	9.86	5.00	10.61	11.00	8.83
ASPENS MN	All students	Early	58	4.86	4.49	9.29	5.89	11.91	5.18	4.43	3.83	7.11	4.00
Late	56	4.21	4.91	7.14	5.56	11.24	6.55	2.93	4.12	6.73	4.22
Intensive	Early	18	0.67	1.03	4.50	3.63	8.00	4.76	3.83	3.96	7.29	4.74
Late	20	0.05	0.22	2.20	2.19	5.75	5.65	2.15	2.21	5.69	5.59
Strategic	Early	26	4.73	2.62	9.69	4.98	12.52	3.92	4.96	3.87	7.60	3.00
Late	24	4.21	2.86	8.67	4.06	11.61	4.25	4.46	4.60	7.48	3.42
Benchmark	Early	14	10.50	3.92	14.71	4.83	15.85	4.56	4.21	3.75	5.92	3.33
Late	12	11.17	4.34	12.33	5.73	17.83	4.90	1.17	4.75	6.67	3.47
ASPENS NI	All students	Early	58	17.41	14.49	28.72	16.39	38.20	14.41	11.31	12.10	21.13	13.44
Late	56	16.52	16.05	24.32	16.70	37.86	19.05	7.80	9.62	20.08	13.63
Intensive	Early	18	2.50	3.73	17.44	11.67	30.82	7.19	14.94	11.82	28.76	8.52
Late	20	2.35	4.28	10.65	12.89	21.19	20.05	8.30	10.71	19.25	18.32
Strategic	Early	26	17.85	6.94	26.58	12.84	36.76	14.37	8.73	11.16	19.24	12.70
Late	24	17.50	12.37	28.62	13.55	41.22	10.56	11.12	8.06	23.04	10.68
Benchmark	Early	14	35.79	11.72	47.21	11.38	50.62	14.28	11.43	13.73	14.77	16.13
Late	12	38.17	7.60	38.50	11.09	53.67	13.36	0.33	6.61	15.5	10.79

Start Type	Class (n)	KinderTEK Mode ^a	M Total Days of Use	M Cal. Weeks of Use	M Days of Use per Cal. Week	M Total Minutes of Instruction ^b	M Lessons Seen in Mode	M Different Lessons Seen
Early	Class A (n = 21)	Sequenced	20.71	—	—	247.47	20.57	—
Overall	20.71	25.98	0.79	247.47	—	20.57
Early	Class B (n = 18)	Sequenced	31.50	—	—	374.92	18.94	—
Overall	31.50	21.94	1.45	374.92	—	18.94
Early	Class C (n = 19)	Sequenced	25.11	—	—	270.13	23.79	—
Directed	6.21	—	—	51.72	5.79	—
Exploration	2.26	—	—	16.79	8.16	—
Overall	33.42	25.11	1.34	338.64	—	29.63
Late	Class D (n = 15)	Sequenced	37.93	—	—	460.43	31.93	—
Overall	37.93	12.78	2.97	460.43	—	31.93
Late	Class E (n = 17)	Sequenced	22.12	—	—	230.25	24.00	—
Overall	22.12	12.52	1.77	230.25	—	24.00
Late	Class F (n = 22)	Sequenced	12.09	—	—	136.90	13.64	—
Screening ^c	0.14	—	—	0.94	0.36	—
Overall	12.18	9.53	1.53	137.84	—	13.64

Measure	Effect Test	p	BH Adj. p	BH Critical Value	Hedges’s g [95% CI]
ASPENS Composite	5.503	.021	.057	.025	.44 [.06, .81]
ASPENS Magnitude Comparison	4.791	.029	.057	.050	.33 [−.04, .70]
ASPENS Missing Number	4.070	.046	.061	.075	.38 [.00, .75]
ASPENS Number Identification	2.919	.090	.090	.100	.32 [−.05, .69]

Footnotes

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Lina Shanley, Mari Strand Cary, and Ben Clarke are eligible to receive a portion of royalties from the University of Oregon’s distribution and licensing of certain KinderTEK-based works. Potential conflicts of interest are managed through the University of Oregon’s Research Compliance Services.

Funding

The author(s) disclosed receipt of the following financial support for the research and/or authorship of this article: This study was supported by the Institute of Education Sciences (Grant #R324A110286) and the Office of Special Education Programs, Office of Special Education and Rehabilitative Services (Grant #H327S140019).

ORCID iD

Mari Strand Cary

References

Al Otaiba

Connor

C. M.

Folsom

J. S.

Wanzek

Greulich

Schatschneider

Wagner

R. K.

(2014). To wait in tier 1 or intervene immediately: A randomized experiment examining first-grade response to intervention in reading. Exceptional Children, 81, 11–27. https://doi.org/10.1177/0014402914532234

Association for Supervision and Curriculum Developmentacd. (2011). Module 4: Reading: Using technology to differentiate instruction. In Technology in schools: A balanced perspective (2nd ed.). [Online course materials]. https://pdo.ascd.org/lmscourses/PD11OC137S/story_content/media/DI_Mgt-Syllabus.pdf

Bachman

Votruba-Drzal

El Nokali

Castle Heatly

(2015). Opportunities for learning math in elementary school: Implications for SES disparities in procedural and conceptual math skills. American Educational Research Journal, 52(5), 894–923. https://doi.org/10.3102/0002831215594877

Benjamini

Hochberg

(1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological), 57(1), 289–300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x

Burchinal

Foster

T. J.

Bezdek

K. G.

Bratsche-Hines

Blair

Verson-Feagons

, & the Family Life Project Investors. (2019). School-entry skills predicting school-age academic and social-emotional trajectories. Early Childhood Research Quarterly, 51, 67–80.

Center on Teaching and Learning. (n.d.). KinderTEK www.kindertek.org

Clarke

Gersten

R. M.

Dimino

Rolfhus

(2012). Assessing student proficiency in early number sense (ASPENS): Administrator’s handbook. Cambium Learning Group, Sopris Learning.

Clarke

Rolfhus

Dimino

Gersten

R. M.

(2011a). Assessing student proficiency in number sense (ASPENS). Cambium Learning Group, Sopris Learning.

Clarke

Smolkowski

Baker

S. K.

Fien

Doabler

Chard

(2011b). The impact of a comprehensive tier I core kindergarten program on the achievement of students at risk in mathematics. The Elementary School Journal, 111(4), 561–584. https://doi.org/10.1086/659033

10.

Common Core State Standards Initiative. (2010). Common core standards for mathematics. http://www.corestandards.org/the-standards/mathematics

11.

Connor

C. M.

Morrison

F. J.

Fishman

B. J.

Schatschneider

Underwood

(2007). Algorithm-guided individualized reading instruction. Science, 315(5811), 464.

12.

Coyne

M. D.

Simmons

D. C.

Hagan-Burke

Simmons

L. E.

Kwok

O.-M.

Kim

Fogarty

Oslund

E. L.

Taylor

A. B.

Capozzoli-Oldham

Ware

Little

M. E.

Rawlinson

D. A. M.

(2013). Adjusting beginning reading intervention based on student performance: An experimental evaluation. Exceptional Children, 80, 25–44. http://cec.metapress.com/content/F4321275232V11WX

13.

Doabler

C. T.

Smith

J. L.

Nelson

N. J.

Clarke

Berg

Fien

(2018). A guide for evaluating the mathematics programs used by special education teachers. Intervention in School and Clinic, 54(2), 97–105.

14.

Doabler

C. T.

Strand Cary

Jungjohann

Clarke

Fien

Baker

Smolkowski

Chard

(2012). Enhancing core mathematics instruction for students at risk for mathematics disabilities. Teaching Exceptional Children, 44(4), 48–57. https://doi.org/10.1177/004005991204400405

15.

Duncan

G. J.

Dowsett

C. J.

Claessens

Magnuson

Huston

A. C.

Klebanov

Pagani

L. S.

Feinstein

Engel

Brooks-Gunn

Sexton

Duckworth

Japel

(2007). School readiness and later achievement. Developmental Psychology, 43, 1428–1446. https://doi.org/10.1037/0012-1649.43.6.1428

16.

Engel

Claessens

Watts

Farkas

(2016). Mathematics content coverage and student learning in kindergarten. Education Researcher, 45(5), 293–300. https://doi.org/10.3102/0013189X16656841

17.

Foster

M. E.

Anthony

J. L.

Clements

D. H.

Sarama

Williams

J. M.

(2016). Improving mathematics learning of kindergarten students through computer-assisted instruction. Journal for Research in Mathematics Education, 47(3), 206–232.

18.

Gaertner

Kim

Desjardins

S. L.

McLarty

K. L.

(2014). Preparing students for college and careers: The causal role of algebra II. Research in Higher Education, 55(2), 143–165. https://doi.org/10.1007/s11162-0139322-7

19.

Geary

D. C.

(1993). Mathematical disabilities: Cognitive, neuropsychological, and genetic components. Psychological Bulletin, 114, 345–362. https://doi.org/10.1037/0033-2909.114.2.345

20.

Haßler

Major

Hennessy

(2016). Tablet use in schools: A critical review of the evidence for learning outcomes. Journal of Computer Assisted Learning, 32(2), 139–156. https://doi.org/10.1111/jcal.12123

21.

Hawkins

R. O

Collins

Hernan

Flowers

(2017). Using computer-assisted instruction to build math fact fluency: An implementation guide. Intervention in School and Clinic, 52(3), 141–147.

22.

Hedges

L. V.

(1981). Distribution theory for Glass’s estimator of effect size and related estimators. Journal of Educational and Behavioral Statistics, 6(2), 107–128. https://doi.org/10.3102/10769986006002107

23.

Higgens

Hiscroft-D’Angelo

Crawford

(2019). Effects of technology in mathematics on achievement, motivation, and attitude: A meta-analysis. Journal of Educational Computing Research, 57(2), 283–319.

24.

Holmes

Gore

Smith

Lloyd

(2018). An integrated analysis of school students’ aspirations for STEM careers: Which student and school factors are most predictive? International Journal of Science and Math Education, 16(4), 655–675. https://doi.org/10.1007/s10763-016-9793-z

25.

Huang

Zhang

Hudson

(2019). Impact of math self-efficacy, math anxiety, and growth mindset on math and science career interest for middle school students: The gender moderating effect. European Journal of Psychology of Education, 34(3), 621–640. https://doi.org/10.1007/s10212-018-0403-z

26.

Institute of Education Sciences. (2014). Assessing attrition bias (Version 3.0). https://ies.ed.gov/ncee/wwc/Document/243

27.

Jordan

N. C.

Kaplan

Hanich

L. B.

(2002). Achievement growth in children with learning difficulties in mathematics: Findings of a two-year longitudinal study. Journal of Educational Psychology, 94, 586–597. https://doi.org/10.1037/0022-0663.94.3.586

28.

Kainz

(2019). Early academic gaps and title I programming in high poverty, high minority schools. Early Childhood Research Quarterly, 47, 159–168. https://doi.org/10.1016/j.ecresq.2018.08.012

29.

Kiru

E. W.

Doabler

C. T.

Sorrells

A. M.

Cooc

N. A.

(2017). A synthesis of technology-mediated mathematics interventions for students with or at risk for mathematics learning disabilities. Journal of Special Education Technology. Advance online publication. https://doi.org/10.1177/0162643417745835

30.

Mac Callum

Jeffrey

Kinshuk. (2014). Factors impacting teachers’ adoption of mobile learning. Journal of Information Technology Education: Research, 13. http://www.jite.org/documents/Vol13/JITEv13ResearchP141-162MacCallum0455.pdf

31.

McFarland

Hussar

Zhang

Wang

Hein

Diliberti

Forrest Cataldi

Bullock Mann

Barmer

(2019). The condition of education 2019 (NCES 2019-144). U.S. Department of Education. National Center for Education Statistics. https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2019144

32.

Morgan

P. L.

Farkas

Hillemeier

M. M.

Maczuga

(2016). Who is at risk for persistent mathematics difficulties in the United States? Journal of Learning Disabilities, 49, 305–319. https://dx-doi-org.web.bisu.edu.cn/10.1177/0022219414553849

33.

Morgan

P. L.

Farkas

(2011). Kindergarten children’s growth trajectories in reading and mathematics: Who falls increasingly behind? Journal of Learning Disabilities, 44, 472–488. https://doi.org/10.1177/0022219411414010

34.

National Center for Education Statistics. (2019a). Highlights of U.S. PISA 2018 results web report (NCES 2020-166). Institute of Education Sciences. https://nces.ed.gov/surveys/pisa/pisa2018/index.asp

35.

National Center for Education Statistics. (2019b). The nation’s report card: NAEP report card: 2019 NAEP mathematics assessment—Highlighted results at grades 4 and 8 for the nation, states, and districts. https://www.nationsreportcard.gov/highlights/mathematics/2019/

36.

National Center on Intensive Intervention. (n.d.). Academic intervention tool chart [Searchable database]. https://charts.intensiveintervention.org/chart/instructional-intervention-tools?field_subject%5B%5D=math&field_grade%5B%5D=elementary

37.

National Defense Education Act, S. 3187, 85th Cong. (1958). https://www.govinfo.gov/content/pkg/STATUTE-72/pdf/STATUTE-72-Pg1580.pdf

38.

National Mathematics Advisory Panel. (2008). Foundations for success: The final report of the National Mathematics Advisory Panel. U.S. Department of Education.

39.

National Research Council. (2001). Adding it up: Helping children learn mathematics. The National Academies Press. https://doi.org/10.17226/9822

40.

Nelson

McMaster

K. L.

(2019). Factors that may influence treatment effects: Helping practitioners select early numeracy interventions. Learning Disabilities Research & Practice, 34(4), 194–206. https://doi.org/10.1111/ldrp.12208

41.

Nelson

Fien

Doabler

Clarke

(2016). Considerations for realizing the promise of educational gaming technology. Teaching Exceptional Children, 48(6), 293–300.

42.

Ninaus

Kiili

McMullen

Moeller

(2017). Assessing fraction knowledge by a digital game. Computers in Human Behavior, 70, 197–206.

43.

Office of Educational Technology. (2010). National educational technology plan: Transforming American education: Learning powered by technology. U.S. Department of Education, Office of Educational Technology. http://www.ed.gov/sites/default/files/netp2010.pdf

44.

Outhwaite

L. A.

Faulder

Gulliford

Pitchford

N. J.

(2019). Raising early achievement in math with interactive apps: A randomized control trial. Journal of Educational Psychology, 111(2), 284–298. https://doi.org/10.1037/edu0000286

45.

Pace

Alper

Burchinal

M. R.

Golinkoff

R. M.

Hirsh-Pasek

(2019). Measuring success: Within- and cross-domain predictors of academic and social trajectories in elementary school. Early Childhood Research Quarterly, 46, 112–125. https://dx-doi-org.web.bisu.edu.cn/10.1016/j.ecresq.2018.04.001

46.

Parsons

S. A.

Vaughn

Scales

R. Q.

Gallagher

M. A.

Parsons

A. W.

Davis

S. G.

Pierczynski

Allen

(2018). Teachers’ instructional adaptations: A research synthesis. Review of Educational Research, 88(2), 205–242. https://doi.org/10.3102/0034654317743198

47.

Pianta

R. C.

Whittaker

J. E.

Vitiello

Ruzek

Ansari

Hofkens

DeCoster

(2019). Children’s school readiness skills across the pre-K year: Associations with teacher-student interactions, teacher practices, and exposure to academic content. Journal of Applied Developmental Psychology, 66, 101084.

48.

Provasnik

Malley

Stephens

Landeros

Perkins

Tang

J. H.

(2016). Highlights from TIMSS and TIMSS Advanced 2015: Mathematics and science achievement of U.S. Students in grades 4 and 8 and in advanced courses at the end of high school in an international context (NCES 2017-002). U.S. Department of Education, National Center for Education Statistics. http://nces.ed.gov/pubsearch

49.

R Core Team. (2018). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/

50.

Scherer

(2011). Transforming education with technology. Educational Leadership, 68(5). http://www.ascd.org/publications/educational-leadership/feb11/vol68/num05/Transforming-Education-with-Technology.aspx

51.

Schleicher

(2019). PISA 2018: Insights and interpretations. OECD. https://www.oecd.org/pisa/PISA%202018%20Insights%20and%20Interpretations%20FINAL%20PDF.pdf

52.

Seo

Shen

Alfaro

E. C.

(2019). Adolescents’ beliefs about math ability and their relations to STEM career attainment: Joint consideration of race/ethnicity and gender. Journal of Youth and Adolescence, 48, 306. https://doi.org/10.1007/s10964-018-0911-9

53.

Shanley

Clarke

Anderson

Turtura

Doabler

Kurtz Nelson

(2018). Exploring the utility of assessing early mathematics intervention response via embedded assessment. [Manuscript under review].

54.

Shanley

Strand Cary

Turtura

Clarke

Pilger

Sutherland

(2019). Individualized instructional delivery options: Adapting technology-based interventions for students with attention difficulties. Journal of Special Education Technology. Advance online publication. https://doi.org/10.1177/0162643419852929

55.

Strand Cary

Crowley

(2018). Evaluation of a preventative iPad-based mathematics intervention for kindergarteners (KinderTEK Research Report 1). University of Oregon.

56.

Strand Cary

Kennedy

P.C.

(2018). KinderTEK observation data as evidence of improved KinderTEK iPad math structure and implementation (KinderTEK Research Report 3). University of Oregon.

57.

Strand Cary

Shanley

Clarke

(2015). Technology-based interventions: An approach to framing the development process (Technical Report No. 1601). University of Oregon.

58.

Strand Cary

Watkins . (2018). KinderTEK as a summer school math program for highly mobile students (KinderTEK Research Report 4). University of Oregon.

59.

Subirana

Sanz

Vila

(2014). Building bivariate tables: The compareGroups package for R. Journal of Statistical Software, 57(12), 1–16. http://www.jstatsoft.org/v57/i12/

60.

U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, What Works Clearinghouse. (n.d.). https://ies.ed.gov/ncee/wwc/FWW/Results?filters=,Math,K-12

61.

U.S. National Commission on Excellence in Education. (1983). A nation at risk: The imperative for educational reform. A report to the nation and the secretary of education, U.S. Department of Education and by the National Commission on Excellent in Education. https://www.edreform.com/wp-content/uploads/2013/02/A_Nation_At_Risk_1983.pdf

62.

van Geel

Keuning

Frèrejean

Dolmans

van Merriënboer

Visscher

A. J.

(2019). Capturing the complexity of differentiated instruction. School Effectiveness and School Improvement, 30(1), 51–67. https://doi.org/10.1080/09243453.2018.1539013

63.

Watts

T. W.

Duncan

G. J.

Siegler

R. S.

Davis-Kean

P. E.

(2016). What’s past is prologue: Relations between early mathematics knowledge and high school achievement. Educational Researcher, 43, 352–360. https://doi.org/10.3102/0013189X14553660

64.

The White House, Office of the Press Secretary. (2019, October 30). Statement from Secretary DeVos on 2019 NAEP results [Press release]. https://www.ed.gov/news/press-releases/statement-secretary-devos-2019-naep-results

65.

University of Oregon Digital Press. (2020). KinderTEK Math (Version Pro—2.4.5). https://apps.apple.com/us/app/kindertek-research/id1264317282

66.

Young

M. F.

Slota

Cutter

A. B.

Jalette

Mullin

Lai

Simeoni

Tran

Yukhymenko

(2012). Our princess is in another castle: A review of trends in serious gaming for education. Review of Educational Research, 82, 61–89. https://doi.org/10.3102/0034654312436980