Reliability and Predictive Validity of Preschool Web-Based Phonological Awareness Assessment for Identifying School-Aged Reading Difficulty

Abstract

This study examined whether web-based phonological awareness (PA) assessment in early childhood, specifically the year preceding formal schooling, could help educators reliably predict children’s risk for reading difficulties in the first year of school. Ninety Australian children participated in several web-based PA tasks with a high priority focus on phoneme-level knowledge at 4- and 4.5 years of age. Real and non-real-word reading ability were examined at age 5. Measures of internal consistency using Cronbach’s alpha were above .7 for all PA tasks, while test-retest reliability correlations were significant (p < .001). Regression analyses indicated that web-based PA measures at 4.5 years of age, alongside preschool-entry language ability, accounted for 79.52% and 74.08% of the variance in 5-year-old real-word and non-real-word decoding ability. Web-based PA assessment focused on phoneme-level skills may be a reliable and valid method of identifying risk for reading difficulties in the year preceding school entry.

Keywords

literacy assessment 3 to 5 years reading word discrimination/recognition/retrieval technology

The role and technical adequacy of web-based tools in supporting routine screening and monitoring of skills known to predict early reading success when administered in the year preceding school entry is limited. Reading competency is a high-stakes priority for countries worldwide with numerous international studies regularly documenting the reading proficiency, literacy levels, and educational progress of young learners (e.g., the Organisation for Economic Co-operation and Development’s (OECD) Programme of International Student Assessment [PISA] and the International Association for the Evaluation of Educational Achievement’s (IEA) Progress in International Reading Literacy Study [PIRLS]). Although a large number of children acquire the skills necessary to become skillful readers, a concerning number of children across developed countries continue to struggle with reading development. For example, in the most up-to-date data available from the PIRLS study, 24%, 17%, and 14% of Australian, British, and American 10-year-old students, respectively, performed at or below the low international PIRLS benchmark, with a further 7%, 5%, and 2% unable to read at the lowest (i.e., below the low) international benchmark (Mullis, Martin, Foy, & Drucker, 2012). Our digital age offers many opportunities to use technology to support the early screening and monitoring of skills known to predict and support reading proficiency prior to school entry, including phonological awareness (PA). The use of web-based technology to support PA assessment in the year preceding formal schooling (i.e., in Australia this period includes children aged 4 to 5 years, and is often termed “preschool,” the year before formal school entry at age 5) as a means of identifying risk for early reading difficulties is yet to receive detailed investigation in terms of both technical adequacy and appropriateness within early childhood environments. Identifying technical adequacy such as reliability and validity is important given the rapid increases in technological tools becoming widely available, and at times, non-researched, for educators today. To begin addressing this gap, this study investigated the reliability and predictive validity of web-based screening and monitoring of PA in preschool, the year preceding formal school entry where the majority of children are 4 years of age.

PA and the Early Prediction of Reading Difficulties

Among a constellation of skills that play an important role in reading development (e.g., vocabulary, reading fluency, comprehension strategies), the importance of strong PA and letter-knowledge ability in the early stages of learning to read has been spotlighted in a number of educational reviews and inquiries globally (e.g., Australian Government, 2005; Ehri et al., 2001; Rose, 2006; Tunmer, Chapman, Greaney, Prochnow, & Arrow, 2013). PA refers to the conscious awareness of and ability to manipulate the sound structure of spoken words at three distinct levels: syllable awareness, rhyme awareness, and phoneme-level awareness. In juxtaposition, letter-sound knowledge refers to an individual’s understanding of how phonemes are represented by graphemes in print. It has been reported that PA at the phoneme level (i.e., phonemic awareness) and knowledge of letter-sound correspondences are among some of the best school-entry indicators of how well a child will learn to read during the first 2 years of formal schooling (Ehri et al., 2001). This is because these two skills enable young readers to decipher an alphabetic script to which they can apply their vocabulary knowledge and subsequently comprehend the meaning of printed words (Catts, Herrera, Nielsen, & Bridges, 2015). Increased proficiency in these skills supports reading fluency and the application of reading comprehension strategies. Ensuring early childhood educators (ECE) are well equipped with reliable and valid methods of assessing an array of skills known to influence early reading success is critical for the early identification of reading difficulties; of which this study will focus on one of those skills, PA.

Traditional PA Assessment Methods in Early Childhood

Assessment is considered an important component of early childhood education worldwide with cross-national philosophies agreeing that high-quality early childhood education involves the coordination of multiple layers of service delivery in which effective assessment practices are central for informing the design and implementation of learning outcomes (e.g., Australian Government, Department of Education and Training, 2009; United Kingdom, Department for Education, 2014; United States National Association for the Education of Young Children [NAEYC], 2009). Recommendations from the early childhood field point to the benefits of assessment methods that employ multiple ways of collecting information as part of an ongoing process (Australian Capital Territory Government, 2014; Shepard, Kagan, & Wurtz, 1998). These include naturalistic observations, anecdotal notes, checklists, portfolios, and criterion reference measures.

Within this context it is important to understand the range of tools and methods currently available to ECE to support preschool screening and monitoring of PA. To identify the current assessment landscape, several international and national sources were reviewed prior to implementing the current study. This review focused on identifying tools commonly available to ECE that measure literacy ability and can be implemented with children younger than 5 years of age. As the focus was to review tools that can be implemented before formal school entry, which in Australia occurs at approximately 5 years of age, suitability for administration before 5 years of age was used as a criterion. Several databases including PubMed, ProQuest, and ERIC, and publisher websites such as Pearson and Pro-Ed were searched to locate assessments from 2005 to 2016. Apple iTunes and Google Play were searched for digital tools. Examples of search terms included PA assessment, phonemic awareness assessment, emergent literacy assessment, preschool literacy assessment, kindergarten literacy assessment, and reading disorder. Tools identified were grouped into six categories: (a) age, (b) developmental domains measured, (c) PA included or not, (d) PA at the phoneme level included or not, (e) modality of assessment delivery (i.e., paper- or technology-based), and (f) suitable or unsuitable for teacher administration. The tools identified using this search process are listed in Table 1. Other available tools likely exist outside the parameters of this search.

Table 1.

Early Childhood Assessment Tools That Include Measures of Early Literacy and can Be Implemented Before 5 Years of Age.

Tool	Age (in years)	PA included	PA at the phoneme level	Modality		Teacher administration
Tool	Age (in years)	PA included	PA at the phoneme level	Paper based	Technology based	Teacher administration
Brigance Comprehensive Inventory of Basic Skills–Revised (CIBS-R)	4 to 12	Yes	Yes	Yes		Yes
Brigance Diagnostic Inventory of Early Development II (IED-II)	Birth to 7	Yes	Yes	Yes		Yes
Clinical Evaluation of Language Fundamentals–Preschool, 2nd Edition (CELF-P, 2)	3 to 6	Yes	No	Yes^a		Restrictions
Comprehensive Test of Phonological Processing, 2nd Edition (CTOPP-2)	4 to 24	Yes	Yes	Yes		Restrictions
Differential Ability Scales–11 (DAS-11)	2½ to 17	No^b	No	Yes		Restrictions
Dynamic Screening for Phonological Awareness (DSPA)	4 to 6	Yes	Yes	Yes		Yes
Early Language and Literacy Classroom Observation Scale (ELLCO)	4 to 8–9	No	No	Yes		Yes
Emerging Literacy and Language Assessment (ELLA)	4½ to 9	Yes	Yes	Yes		Restrictions
High/Scope Preschool Child Observation Record (COR)	2½ to 6	Yes	Yes	Yes		Yes
Infant Toddler Environmental Rating Scale–Revised (ITERS-R)	Birth to 2½	No	No	Yes		Yes
Kaufman Survey of Early Academic and Language Skills (K-Seals)	3 to 6	No	No	Yes		Restrictions
Metropolitan Readiness Test, 6th Edition (MRT-6)	4 to 7	Yes	Yes	Yes		Yes
Phonological Awareness and Literacy Screenings–PreK (PALS–Pre-K)	4	Yes	Yes	Yes^a		Yes
Phonological Awareness Skills Mapping (PASM)	4 to 6	Yes	No^c	Yes		Yes
Preschool Early Literacy Indicators (PELI–PreK)	3 to 5	Yes	Yes	Yes		Yes
Preschool and Primary Inventory of Phonological Awareness	3 to 6	Yes	Yes	Yes		Yes
Profile of Phonological Awareness (PRO-PA)	3 to not specified	Yes	Yes		Yes^b	Yes
Qualls Early Learning Inventory (QELI)	4 to 6	No	No	Yes		Yes
Ready to Learn: A Dyslexia Screener	3½ to 6½	Yes	Yes	Yes		Yes
Screen of Phonological Awareness (SPA)	4 to 5	Yes	Yes	Yes		Yes
Teacher Rating of Oral Language and Literacy (Troll)	3 to 5	No	No	Yes		Yes
Test of Early Reading Ability, 3rd Edition (TERA-3)	3½ to 8½	No^b	No	Yes		Restrictions
Test of Preschool Early Literacy (TPEL)	3 to 5	Yes	Yes	Yes		Yes
Wechsler Individual Achievement Test, 2nd Edition (WIAT-II)–Australian Version	4 to 85	No	No	Yes^a		Restrictions
Woodcock-Johnson III Nu Complete (WJ III NU Complete)	2 to 90	Yes	Yes	Yes^a		Restrictions
Young Children’s Achievement Test (YCAT)	4 to 7	Yes	Yes	Yes		Restrictions

Note. PA = phonological awareness.

This tool is primarily paper-based but does have data entry software. ^bThis tool provides educators with instructions on an ipad application whereby the educator asks the questions, enters data directly, and receives results; the tool does not administer test items. ^cThis tool includes measures of PA or PA at the phoneme level but not for children in preschool; restrictions under “Teacher administration” indicate that the tool requires certain qualifications for a teacher to administer that assessment. Where an assessment provides upper age limits in years plus months, the age in years is documented in the table (i.e., 5 years 11 months is documented as 5 years instead of being rounded to 6 years) to represent the entire year within which the assessment could be used with a child.

Of the tools profiled in Table 1, 69% include a general PA measure whereas 62% include a PA measure at the phoneme level. Approximately 50% of the tools that include a PA measure and 46% of those that measure PA at the phoneme level can be administered by professionals with teaching degrees. Of the tools designed for teacher use, 76% include general PA measures and 71% include the evaluation of phoneme-level knowledge. Only 19% of the profiled tools used scoring software whereby an educator first administers a paper-based assessment and then manually enters the results into a software system. A single tool uses technology in the form of an ipad application prompting the teacher to ask a child specific questions, and then to enter the child’s response into the application. This application, Profile of Phonological Awareness (Pro-PA), does not require the child to engage with the technology by indicating their response to images, nor does the manual include information on the validity and reliability of the tool. None of the tools reviewed used technology to administer the assessment beyond recording and scoring responses; this feature emerges more as children enter formal schooling in tools such as the Performance Indicators in Primary School (Tymms, Merrell, & Buckley, 2016). Given our current technological age, there is scope for critically evaluating whether web-based methods of measuring skills such as PA are reliable and valid when implemented with young children in the year preceding formal entry to school.

Technology and PA Assessment in Early Childhood

Technology and its availability are swiftly changing methods of teaching and learning in homes, schools, and early childhood education environments. According to a joint position statement issued by the NAEYC and the Fred Rodgers Center for Early Learning and Children’s Media (NAEYC/Fred Rodgers Center, 2012), technology is an effective tool that can promote learning and development in early childhood when used intentionally and appropriately for the age and stage of each unique child, and confined within the limitations of its usage. The appropriate use of technology within such a setting should extend children’s access to and engagement with new learning, and form one of many possible options to support teaching and learning in the early years. The use of web-based technology as a method of assessment offers many advantages including the following: (a) it is time efficient; (b) it presents items accurately and consistently; (c) it is available on multiple mobile devices including tablets, Microsoft surfaces, mobile phones, and other internet-enabled devices; (d) it uses existing technology available to educators; (e) it is motivational for children who may struggle with other assessment methods; and (f) it is cost-effective as there is no or little need to purchase or print assessment forms (Bjornsson, 2008; Bridgeman, 2009; Carson, Gillon, & Boustead, 2011; Martin, 2008; Ripley, 2008).

In the current study, a web-based screening and monitoring tool focused on measurement of PA skills at the phoneme level in addition to letter-knowledge was developed. The creation of an essentially univariate tool emerged from the need for a time-efficient screening and monitoring process suitable for widespread use in the preschool environment, a feature which is reportedly popular when time restrictions are prevalent (Kamhi & Catts, 2012). The aim of developing a web-based PA screening and monitoring tool was to provide ECE with a user-friendly technological method of (a) identifying children at risk of reading difficulties; (b) profiling and monitoring strengths and weaknesses in early PA knowledge to inform both structured and unstructured teaching and learning opportunities; and (c) complementing existing assessment methodologies by providing an alternative option that may suit different educators, children, and early childhood settings.

The free web-based tool, titled the South Australian Phonological Awareness Database (SAPAD; Carson, 2015), comprises 10 play-based tasks each composed of several test items. Six of these tasks are specifically designed to probe PA knowledge in children aged 4 to 6 years old, particularly at the phoneme level. The tasks encourage children to help a character identify words that do not rhyme; identify the first and final sounds in words, segment and blend sounds in words, delete sounds to create new words; and to recognize letters and letter-sounds. Picture stimuli minimize demands on children’s working memory and tasks are short in duration, particularly for children aged 5 and under to accommodate shorter concentration spans. Each task includes questions that require a non-verbal response whereby children touch a picture to respond, a feature designed for younger children who may have speech difficulties, are reluctant to talk, or are shy. All tasks are administered on a tablet or mobile device via the internet as opposed to an “app,” which can limit usage based on device-type availability, and are automatically scored for educators. For older children (i.e., 5 to 6 years old), tasks can be self-administered by the child.

A guiding technology-based assessment tool recently researched by the author and colleagues (Carson, Boustead, & Gillon, 2014; Carson, Boustead, & Gillon, 2015) measured PA ability for children already in school, that is, older children aged 5 to 6 years. This version operated via desktop computers. A series of studies demonstrated that the administration of PA assessment tasks for school-aged children via a desktop computer was 30% faster than the paper-based alternative (Carson et al., 2011), predicted reading outcomes with up to 94% accuracy across the first year of school (Carson et al., 2014), and accurately differentiated between children who became good and poor readers by age 6 (Carson et al., 2015). In contrast, the current investigation aimed to determine whether web-based screening and monitoring of PA skills for younger children, specifically at 4 and 4.5 years of age before school entry, continues to be a reliable and valid method of predicting early reading outcomes. It is plausible that web-based measurement in the year preceding school entry could produce less predictive and reliable results due to a number of extraneous variables associated with the assessment of younger children. These include fatigue, hunger, short attention spans, mild sicknesses such as colds, and so forth. Therefore, the following research questions were addressed:

Can web-based delivery of PA tasks generate satisfactory levels of internal consistency and test-retest reliability when administered to children at 4 and 4.5 years of age, in the year preceding formal schooling?

Can web-based delivery of PA tasks at 4 years of age account for the majority of the variance in word decoding ability in the first year of formal schooling, at 5 years of age, when entered into a prediction model with multiple educational variables that share a relationship with school-aged reading outcomes?

Method

Participants

Ninety South Australian (47 females and 43 males) preschool children participated in this study. The children were aged between 3; 11 to 4; 02 (years; months; M = 4.01, SD = 0.04). In South Australia, children who turn 4 before the 1st of May begin preschool on the first day of Term 1 (typically early February), whereas children who turn 4 after the 1st of May begin preschool in the following year; preschool spans from early February to late December. Participants were recruited using a random sampling procedure whereby preschools in a metropolitan area were allocated an identification number, entered into the software program Research Randomiser, and contacted if the program selected them as a participating site. Site directors were then contacted and asked to disseminate consent forms to parents of children in their preschool who met the following criteria: (a) were aged between 3; 10 to 4; 02 at the start of the study, and (b) had no significant hearing, visual, cognitive, or physical impairment precluding the appropriate use of the web-based assessment. Eighteen children presented with spoken language difficulties with 15 having phonologically based speech deficits, one with receptive and expressive language delay, one with expressive language delay and speech deficits, and one child with receptive and expressive language delay accompanied by speech impairment. The remaining 72 children presented with typically developing spoken language skills.

Procedure

A 1-year longitudinal research design containing the following assessment components was employed: (a) a comprehensive baseline assessment at the start of the preschool year, (b) web-based PA screening and monitoring at the start and middle of the preschool year, and (c) measures of single real-word reading (RWD) and single non-word reading (NWD) during the first semester of formal schooling. Assessments took place in a quiet, but not separate, area of the children’s preschool by a qualified educator trained in the assessment procedures for this study. Baseline assessments took place over two sessions per child and lasted for approximately 30-min in duration, whereas subsequent assessment sessions were completed in one session of 20 to 30 min in duration. This research project represents the first year of a larger 2-year investigation. Specific details of each assessment measure, as well as the average performances for the sample, are provided below:

Comprehensive baseline assessments

A comprehensive baseline assessment was completed to profile each participant’s spoken language and PA ability on established, standardized, paper-based measures, before the administration of the web-based PA tasks. Baseline assessments included (a) the Clinical Evaluation of Language Fundamentals, Preschool, 2nd Edition (Australian and New Zealand Edition; CELF-P2; Wiig, Secord, & Semel, 2006) whereby subtests were used to obtain an index of expressive and receptive language functioning, in addition to a core language index; (b) the Preschool and Primary Inventory of Phonological Awareness (PIPA; Dodd, Crosbie, McIntosh, Teitzel, & Ozanne, 2000) whereby the subtests of rhyme awareness, phoneme isolation, and letter-knowledge were administered to profile PA ability using standardized information for Australian children aged 3 years 0 months to 6 years 11 months; and (c) the Diagnostic Evaluation of Articulation and Phonology (DEAP; Dodd, Hua, Crosbie, Holm, & Ozanne, 2006) whereby data from the phonology subtest were analyzed to obtain a percentage consonants correct (PCC) score for each participant. Table 2 illustrates the average performance of the sample of 90 children on baseline measures of speech, language, and PA at the start of the study.

Table 2.

Performance on Baseline Measures of Speech, Language, and PA for the Total Sample and for Children With Spoken Language Impairment.

Baseline Spoken Language Measures	CELF-P2		DEAP	PIPA
Baseline Spoken Language Measures	RLI	ELI	PCC-R	RA	IPI	LK^a
Total sample (n = 90)
M	101.21	98.03	95.72	7	8	8
SD	5.61	6.02	3.42	1.22	1.05	2.01
Range	75–119	71–120	92–100	5–9	8–13	7–12
Children with spoken language impairment (n = 18)
M	97.28	93.67	90	6.	6.	7
SD	8.6	9.6	2	1.08	1.09	0.70
Range	75–107	71–109	87–93	5–8	5–7	7–9

Note. PA = phonological awareness; CELF-P2 = Clinical Evaluation of Language Fundamentals, Preschool, 2nd Edition (Australian and New Zealand Edition)–Receptive and Expressive Language Indices (M = 100, SD = +/−15) (Wiig, Secord, & Semel, 2006); DEAP = Diagnostic Evaluation of Articulation and Phonology (Dodd, Hua, Crosbie, Holm, & Ozanne, 2006); PIPA = Preschool and Primary Inventory of Phonological Awareness Standard Scores (M = 10, SD = +/−3); RLI = Receptive Language Indices; ELI = Expressive Language Indices; PCC-R = Percentage Consonants Correct–Revised; RA = Rhyme Awareness Standard Score; IPI = Initial Phoneme Identity Standard Score; LK = Letter-Knowledge Standard Score (Dodd, Crosbie, McIntosh, Teitzel, & Ozanne, 2000).

Denotes where U.K. standard scores have been used due to the unavailability of Australian standard scores. Research suggests that the average PCC for children aged 4 years 0 months to 4 years 11 months is 92.7% (Shriberg, Austin, Lewis, McSweeny, & Wilson, 1997).

Web-based PA assessment

The web-based tool designed for the larger 2-year study included six measures of PA (five specifically at the phoneme level), two letter-knowledge tasks, and one single-word reading task. For the current study, age-appropriate tasks of rhyme oddity, initial phoneme identity, and letter-knowledge were measured at the start and middle of the preschool year. Rhyme oddity and initial phoneme identity tasks were modeled on earlier paper-based work by Gillon (2005), while the remaining tasks were modeled on more recent work by Carson and colleagues (2014, 2015). The task of final phoneme identity, while more challenging for this age range, was also included as an extension for children showing stronger PA development, and to avoid ceiling effects. Details of tasks included in this study are as follows: (a) Rhyme Oddity measures children’s ability to identify which word does not rhyme out of a choice of three words (each represented by a picture). For example, which word does not rhyme: fish, dish, or ball? Two practice items with feedback are provided, followed by 10 test items. (b) Initial Phoneme Identity measures children’s ability to identify which word, from a choice of three words (each represented by a picture), starts with a specified initial sound. For example, this is my friend dog; what word starts with the /d/ sound: moon, duck, or whale? Two practice items with feedback are provided, followed by 10 test items. (c) Final Phoneme Identity measures children’s ability to identify which word, from a choice of three words (each represented by a picture), ends with a specified final sound. For example, which word ends with the /t/ sound: hat, hole, or sun? Two practice items with feedback are provided, followed by 10 test items. And (d) Letter-Knowledge is measured by two separate tasks: letter-name recognition and letter-sound recognition. Both tasks require children to indicate which letter or sound from an array of six letters and/or sounds represents the letter or sound spoken by the web-based assessment program. A total of 18 letters and sounds are assessed. PA tasks and items used in this study are profiled in the appendix.

Early word decoding skills in the first year at school

RWD and NWD ability was assessed in the first semester of participant’s first year at school. Analysis focused on how children were applying their knowledge of PA and letter-sound relationships within the context of single written words. Probes commonly used in PA research were utilized and a scoring system sensitive in the detection of the ability to decode sound-symbol correspondences, as opposed to the accurate reading aloud of an entire printed word, was applied. In this study, non-word reading tasks from the Reading Freedom Diagnostic Reading Test (Calder, 1992) were used to measure application of PA knowledge and letter-knowledge to the decoding of single non-words. Non-words are divided into three sets: Set 1 contains simple consonant-vowel-consonant words with short vowels; Set 2 contains consonant-consonant-vowel-consonant words including initial consonant clusters, short vowels, and diagraphs; Set 3 contains consonant-vowel-consonant words involving long vowels and more complex vowel rules. This study used Sets 1 and 2, representing a total of 20 non-words. Children were asked to read aloud each non-word presented individually on a small card or to say aloud the sounds of the letters they recognized. Non-words were presented until the child produced either 10 consecutive errors including non-responses, or showed signs of struggling (i.e., asking to discontinue, avoidance strategies). One point was awarded for each correct phoneme–grapheme conversion.

Real words from the Australian Oxford Wordlists (Bayetto, Lo Bianco, & Scull, 2008) were selected to measure real-word reading ability. The Oxford Wordlists rank high frequency words found in children’s writing samples in the first 3 years of schooling. Twenty words were shown individually to children and after 10 consecutive errors or non-responses, the task was discontinued. Reading aloud of real words was scored based on both regular and irregular elements of the printed word whereby a score of 3 indicated the correct reading aloud of an element, 2 a plausible reading aloud of an element, and 0 an incorrect reading aloud of an element. A list of both real and non-real words used in this study are profiled in the appendix. Table 3 illustrates the average performance of children on the web-based PA tasks administered at the start and middle of the preschool year alongside their performance related to real and non-real-word reading ability at school entry.

Table 3.

Performance on Web-Based PA Tasks at the Start and Middle of the Preschool Year and on RWD and NWD at School Entry.

Age-selected web-based tasks
Total sample (n = 90)
Phonological Awareness and Early Reading Performance
	Start of the year				Middle of the year				School entry
	RO	IPI	FPI	LK	RO	IPI	FPI	LK	RW^a	NW^b
M	3.96	3.67	1.00	4.38	4.49	4.95	1.45	6.28	8.00	14.00
SD	0.21	0.10	0.08	0.05	0.32	0.25	0.92	0.12	0.49	0.38
Range	0–9	0–6	0–2	0–29	0–10	1–10	0–3	2–36	0–17	0–21
Children with Spoken Language Impairment (n = 18)
	Start of the year				Middle of the year				School entry
	RO	IPI	FPI	LK	RO	IPI	FPI	LK	RW^a	NW^b
M	3	2.6	0	2.01	3.67	3.56	1.00	4.58	5.94	11.89
SD	1.05	1.33	0.81	0.46	1.79	0.31	0.98	0.94	1.33	1.11
Range	1–5	1–5	0–2	0–11	0–10	1–10	0–3	2–20	1–15	4–18

Note. PA = phonological awareness; RWD = real-word reading; NWD = non-word reading; RO = rhyme oddity; IPI = initial phoneme identity; FPI = final phoneme identity; LK = letter-knowledge (a composite score of letter-name and letter-sound recognition).

Number of correct phoneme–grapheme or element conversions with a total of 60 possible points. ^bNumber of correct phoneme–grapheme conversions within a non-word with a total of 69 possible points (all scores are raw scores).

Scoring Reliability

At each of the three assessment points, 20% of data was selected for inter-rater reliability checking, by which an experienced researcher cross-checked data collected via paper-based recording methods with that collected by the automated scoring system in the web-based tool. This data were then cross-checked with data entered in the statistical software to ensure 100% reliability.

Results

Internal Consistency and Test-Retest Reliability

Internal consistency is a measure of reliability and evaluates the relationship between test items and response consistency to test items (Kimbrel et al., 2015). Cronbach’s alpha was used to evaluate internal consistency between test items in the PA tasks administered at 4 and 4.5 years of age. Test items within a task are considered internally consistent when Cronbach’s alpha scores are above .7 (Field, 2009). All tasks administered at 4 and 4.5 years of age generated Cronbach’s alpha scores above .7 and are illustrated in Table 4.

Table 4.

Cronbach’s Alpha Scores by Task at 4 and 4.5 Years of Age.

Reliability for Phonological Awareness Tasks	Preschool entry (4 years)	Mid-preschool year (4.5 years)
Rhyme oddity	.78	.82
Initial phoneme identity	.75	.80
Final phoneme identity	.70	.84
Letter-sound	.80	.81

Test-retest reliability indicates how consistent an assessment is over repeated administrations (Thorndike & Thorndike-Christ, 2010), and was calculated in this study by correlating each PA task with itself from the administrations at 4 and 4.5 years of age. Significant correlations, where p < .001, were identified for each task as follows: rhyme oddity (r = .74), initial phoneme identity (r = .76), final phoneme identity (r = .69), and letter-knowledge (r = .80). According to Hattie (2009), these correlations represent large effects.

Regression Analyses for Web-Based PA Assessment in Early Childhood

Regression analyses were used to identify which variables in preschool predicted single RWD and NWD ability in the first year of school. Interest was focused on whether PA tasks administered in a web-based format could support the prediction of early reading development, among a range of other educational variables with known relationships to reading outcomes. The predictor variables measured at preschool entry were (a) PA whereby a factor PA score was calculated using results from the rhyme oddity, initial phoneme identity, and final phoneme identity tasks; (b) a letter-knowledge composite score (combining letter-name and letter-sound recognition raw scores); (c) a core language standard score from the CELF-P2; (d) PCC as a measure of phonological accuracy; and (e) a socioeconomic status score of 1 or 0 (categorized as “high” or “low” using the Index of Community Socio-Educational Advantage; Australian Curriculum, Assessment and Reporting Authority [ACARA], 2015]). The independent variables were RWD and NWD raw scores.

Stepwise regression analyses were employed to identify which combination of predictor variables had the strongest relationship to school-aged RWD and NWD ability. Initial analysis generated two models. In the first model, PA ability alone accounted for 51.29% of the variance in RWD ability (F = 9.297, p = .003), and in the second model, PA and language ability combined accounted for 66.73% of the variance in RWD (F = 8.37, p < .001). In terms of NWD ability, the first model identified that PA alone accounted for 55.61% of the variance (F = 11.14, p = .001), whereas the second model showed that PA and language combined accounted for 70.22% of the variance (F = 9.48, p < .001).

A secondary analysis was undertaken to evaluate changes in the relationship between predictor variables and RWD and NWD, if PA ability during the middle of the preschool year (i.e., approximately 4.5 years of age) as opposed to preschool-entry PA ability (i.e., approximately 4-years of age) was entered into the regression analyses. All predictor variables were entered into a secondary stepwise regression analysis with preschool-entry PA ability being substituted with middle of the year preschool PA ability. Stepwise regression analysis showed that midyear PA ability alone accounted for 73.28% of the variance in RWD ability (F = 23.37, p > .000), and midyear PA ability in combination with language ability accounted for 79.52% of the variance in RWD ability (F = 12.30, p > .000). Stepwise regression analysis for NWD reading identified that midyear PA alone accounted for 68.89% of the variance in NWD (F = 18.26, p > .001) and combined with language accounted for 74.08% of the variance in NWD (F = 11.84, p > .001). By substituting preschool-entry PA ability with midyear PA ability (i.e., approximately 4.5 years of age instead of 4 years of age) into the regression analyses, an increase in both PA ability alone (Model 1) and PA and language ability (Model 2) were observed for both RWD ability (e.g., PA ability alone increased from 51.29% to 73.28% and PA plus language ability increased from 66.73% to 79.52%) and NWD ability (e.g., PA alone increased from 55.61% to 68.89% and PA plus language ability increased from 70.22% to 74.08%). These results are summarized in Table 5 below.

Table 5.

Relationships Between Preschool-Entry PA and Language Ability, and Midyear PA and Language Ability, on Real and Non-Real-Word Reading in the First Year at School.

Model 1: PA alone as a predictor of early reading
Preschool-entry PA	51.29% RWD, p = .00355.61% NWD, p = .001	Midyear PA	73.28% RWD, p > .00068.89% NWD, p > .001
Model 2: PA + language ability as predictors of early reading
Preschool-entry PA + language	66.73% RWD, p < .00170.22% NWD, p < .001	Midyear PA + preschool-entry language	79.52% RWD, p > .00074.08% NWD, p > .001

Note. PA = phonological awareness; RWD = real-word reading; NWD = non-word reading.

Discussion

This study employed a 1-year longitudinal design to evaluate the reliability and predictive validity of web-based PA screening and monitoring at 4 and 4.5 years of age on forecasting single-word reading ability in the first year of school. A number of language- and cognitive-based skills support skillful reading development, of which web-based PA ability was the focus in this study, and can be built upon through juxtaposition with web-based language measures in future investigations. Utilizing web-based technology to support screening and monitoring of important prereading skills, as demonstrated in this study, is one way in which identifying risk for school-aged reading difficulties can be achieved prior to formal reading instruction.

Predictive Validity and Reliability of Preschool Web-Based PA

The first research question investigated whether web-based administration of PA tasks at 4 to 4.5 years of age could generate satisfactory levels of internal consistency and test-retest reliability. Establishing satisfactory levels of reliability is highly important for ensuring measurement results are consistent and stable, and can therefore be confidently used to evaluate educational progress (Field, 2009). Data analysis using Cronbach’s alpha and correlation coefficients confirmed sufficient levels of internal consistency and stable repeatability of rhyme oddity, initial phoneme identity, final phoneme identity, and letter-knowledge items in the web-based tool. These results are important for ensuring ECE can confidently use the tool reported in this study with children who are 4-years of age knowing that the results they will obtain are reliable, and consistent when repeated for progress monitoring. It is important to note that different web-based tools may generate different levels of reliability as a function of design, study sample, and so forth, and thus ECE must be mindful of potential differences between tools when using these with young learners.

The second research question examined whether the web-based delivery of PA tasks at 4 years of age could account for a majority of the variance in the RWD and NWD at 5-years of age when entered into a regression model with multiple educational predictors. Stepwise regression analyses demonstrated that PA ability in the middle of the preschool year (i.e., age 4.5 years) alongside preschool-entry language ability (i.e., age 4-years), over and above other variables such as socioeconomic status, accounted for 79.52% of the variance in school-aged RWD reading ability and 74.08% of the variance in NWD reading ability. The results indicate that measurement of PA ability, particularly at the phoneme level, at age 4.5 years provides more predictive information than measurement of preschool-entry PA skills at 4 years of age. It is plausible that the types of tasks included in the web-based PA tool may have influenced this result in that awareness of the initial and final phonemes in words are likely to become more salient to children as they approach 4.5 years of age and older. It is not surprising that spoken language ability supported the prediction of school-aged single-word decoding ability alongside PA ability as research clearly demonstrates the importance of oral language skills to early reading development as well as its more enduring contribution to subsequent reading comprehension (Dickinson, Golinkoff, & Hirsh-Pasek, 2010). Investigating the integration of language-based measures into the web-based PA screening and monitoring process, that can continue to be time efficient and avoid overburdening attention span in young children, would be an ideal focus of future investigations.

Importantly, predictive validity results reported here (e.g., 79.52% for RWD and 74.08% for NWD) are stronger than those obtained using a similar version of the assessment with school-aged children in New Zealand (Carson et al., 2014), whereby school-entry (i.e., 5-years of age) web-based measures of PA (p < .001), alongside language performance (p = .004), explained 68.9% of the variance in reading ability at the end of the first year at school. While acknowledging this study took place within a different education system and cultural climate, these results suggest that using web-based measures to screen and monitor PA ability prior to school entry provides more powerful predictive information than using such a tool from school entry. This holds important implications for educational assessment practices in that initial evidence reported here suggests that measuring PA knowledge, particularly at the phoneme level, can help ensure children vulnerable for future reading difficulties are identified in a timely fashion, in the months before the commencement of formal reading instruction.

Further, these positive predictive values for different levels of education across research studies (e.g., preschool and school-aged children) and between countries reinforces the predictive validity of web-based methods for the early identification of reading difficulties both prior to and during the first formative year of schooling. Moreover, the results in this study provide initial evidence that web-based technology can act as a useful adjunct to current assessment methods employed by ECE and uncovers direction for future research to evaluate the appropriate and meaningful incorporation of technology, alongside traditional assessment methods, within early learning environments.

Implications for Early Childhood Assessment Practices

Assessment is an important component of early childhood educational practices worldwide. Regarding the assessment tools with a literacy focus available to ECE based on the parameters of our literature search, it appears that approximately two thirds include measures of PA, and of those tools designed for teacher use up to three quarters include PA skills. However, the use of technology to support assessment practices in the preschool year appears limited in terms of what is available, what has been researched, and how this could help ECE identify children who may or may not be at risk of early reading difficulties before they enter formal schooling. ECE may abstain from using technology as a medium of assessment for a number of reasons including (a) unease about the interplay between technology and the authentic play-based nature of the early childhood education setting, (b) limited access to technology, (c) limited professional understanding regarding how technology can be used appropriately to assist the assessment process and how this aligns with the context of early childhood pedagogy, and (d) preference for traditional assessment practices including observations and anecdotal note taking. Understand more about how technology can be capitalized within early childhood environments is essential to ensure educator confidence in utilizing tools, such as those described in this study, that can offer several benefits such as (a) standard and objective administration, (b) accurate recording and scoring, (c) utilization of skills that many children come to preschool possessing, and (d) supplementing existing paper-based methods. In light of such benefits, it is important to acknowledge that not all web-based assessments are of equal value. It is critical that ECE select tools that are at minimum evidence-based or are ideally researched accompanied by comprehensive information regarding technical adequacy such as validity and reliability. In addition, ECE can be encouraged to consider using measures that include a focus on phoneme-level skills particularly in the months leading up to school entry and, as seen with previous investigations, a univariate focus may offer a time efficient means of achieving widespread screening prior to engagement with formal reading instruction. Overall, research regarding the role technology-assisted assessment can play within early childhood environments is emergent and warrants ongoing investigation.

Limitations and Future Directions

A number of variables can be enhanced in future studies to address the limitations of the current study. These include expansion of the sample size, inclusion of populations who may be particularly interested in web-based assessment such as those located in rural and remote communities, the potential inclusion of language-based measures given the importance of these skills in the regression analyses, as well as understanding how familiarity with technology among younger children may influence performance. Further, the use of web-based methodologies for populations of children with diverse learning needs including those with hearing impairments, physical difficulties, and cognitive differences is of interest for future investigations.

Conclusion

The early identification of children at risk of reading difficulties provides a way to ensure that all young children have the opportunity to prosper in literacy development. International studies demonstrate that sizable proportions of children continue to struggle with reading development despite residing in developed countries with sophisticated education systems (Mullis et al., 2012). Understanding more about the role played by technology in helping educators measure precursory reading skills is critical to understanding how educators can truly harness the power of the digital age to support optimal teaching and learning for our youngest students.

Footnotes

Appendix

Acknowledgements

The author would like to express sincere thanks to the Faculty of Education, Humanities, and Law at Flinders University for the provision of an Establishment Grant to support software development and employment of research assistants. Gratitude is extended to all preschool sites, children, and families who participated in this study, in addition to Danielle Williams, Jennifer Francis, Clare Kelton, and Rebecca Smith for research assistant support.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Australian Capital Territory Government. (2014, July). Early intervention—An overview of best practice. Retrieved from http://www.communityservices.act.gov.au/__data/assets/pdf_file/0007/635695/Early-Intervention-Best-Practice.pdf

Australian Curriculum, Assessment and Reporting Authority. (2015). My school glossary: Index of community socio-educational advantage. Retrieved from http://www.acara.edu.au/myschool/myschool_glossary.html

Australian Government. (2005). Teaching reading: Report and recommendations (National inquiry into the teaching of literacy). Canberra: Author.

Australian Government, Department of Education and Training. (2009). Early Years Learning Framework (EYLF). Retrieved from https://www.education.gov.au/early-years-learning-framework

Bayetto

Lo Bianco

Scull

(2008). Australian Oxford wordlists. Retrieved from http://www.oxfordwordlist.com/pages/

Bjornsson

(2008). Changing Icelandic national testing from traditional paper and pencil based tests to computer based assessment: Some background, challenges and problems to overcome. Luxembourg: Office for Official Publications of the European Communities.

Bridgeman

(2009). Experiences from large-scale computer-based testing in the USA. Luxembourg: Office for Official Publications of the European Communities.

Calder

(1992). Reading freedom teacher’s manual. Glebe, Australia: Pascal Press.

Carson

(2015). South Australian Phonological Awareness Database (SAPAD) [online resource]. Retrieved from: sapad.org

10.

Carson

K. L.

Gillon

Boustead

(2011). Computer-administered versus paper-based assessment of school-entry phonological awareness ability. Asia Pacific Journal of Speech, Language, and Hearing, 14, 85–101.

11.

Carson

K. L.

Gillon

Boustead

(2013). Classroom phonological awareness instruction and literacy outcomes in the first year of school. Language, Speech and Hearing Services in Schools, 44, 147–160.

12.

Carson

K. L.

Boustead

Gillon

(2014). Predicting reading outcomes in the classroom using a computer-based phonological awareness screening and monitoring assessment (Com-PASMA). International Journal of Speech -Language Pathology, 16, 552–561.

13.

Carson

K. L.

Boustead

Gillon

(2015). Content validity to support the use of a computer-based phonological awareness screening and monitoring assessment (Com-PASMA) in the classroom. International Journal of Speech -Language Pathology, 17, 500–510.

14.

Catts

Herrera

Nielsen

Bridges

(2015). Early prediction of reading comprehension within the simple view framework. Reading and Writing, 28, 1407–1425.

15.

Dickinson

Golinkoff

Hirsh-Pasek

(2010). Speaking out for language—Why language is central to reading development. Educational Researcher, 39, 305–310.

16.

Dodd

Crosbie

McIntosh

Teitzel

Ozanne

(2000). Preschool and primary inventory of phonological awareness (PIPA). London, England: Psychological Corporation.

17.

Dodd

Hua

Crosbie

Holm

Ozanne

(2006). Diagnostic evaluation of articulation and phonology (DEAP). London, England: Psychological Corporation.

18.

Ehri

L. C.

Nunes

S. R.

Willows

D. M.

Schuster

B. V.

Yaghoub-Zadeh

Shanahan

(2001). Phonemic awareness instruction helps children learn to read: Evidence from the national reading panel’s meta-analysis. Reading Research Quarterly, 36, 250–287.

19.

Field

(2009). Discovering statistics using SPSS (3rd ed.). London, England: SAGE.

20.

Gillon

G. T.

(2005). Facilitating phoneme awareness development in 3- and 4-year old children with speech impairment. Language, Speech, and Hearing Services in Schools, 36, 308–324.

21.

Hattie

(2009). Visible learning. London, England: Routledge.

22.

Kamhi

A. G.

Catts

H. W.

(2012). Language and reading disabilities (3rd ed.). Upper Saddle River, NJ: Pearson Education.

23.

Kimbrel

N. A.

Flynn

E. J.

Carpenter

G. S.

Cammarata

C. M.

Leto

Ostiguy

W. J.

. . . Gulliver

S. B.

(2015). Internal consistency, test-retest reliability, and predictive validity for a Likert-based version of the Sources of Occupational Stress–14 (SOOS-14) Scale. Psychiatry Research, 228, 960–962.

24.

Martin

(2008). New possibilities and challenges for assessment through the use of technology. Luxembourg: Office for the Official Publications of the European Communities.

25.

Mullis

V. S.

Martin

M. O.

Foy

Drucker

K. T.

(Eds.). (2012). PIRLS 2011 international results in reading. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College.

26.

National Association for the Education of Young Children. (2009). Where we stand on curriculum, assessment, and program evaluation. Retrieved from http://www.naeyc.org/files/naeyc/file/positions/StandCurrAss.pdf

27.

National Association for the Education of Young Children/Fred Rodgers Center. (2012). Technology and interactive media as tools in early childhood programs serving children from birth through age 8. Retrieved from http://www.naeyc.org/files/naeyc/PS_technology_WEB.pdf

28.

Ripley

(2008). Technology in the service of 21st century learning and assessment: A UK perspective. Luxembourg: Office for Official Publications of the European Communities.

29.

Rose

(2006). Independent review of the teaching of early reading—Final report. Retrieved from http://www.standards.dfes.gov.uk/phonics/report.pdf

30.

Shepard

Kagan

Wurtz

(Eds.). (1998). Principles and recommendations for early childhood assessments. Washington, DC: National Educational Goals Panel.

31.

Shriberg

L. D.

Austin

Lewis

B. A.

McSweeny

J. L.

Wilson

D. L.

(1997). The speech disorders classification system (SDCS): Extensions and lifespan reference data. Journal of Speech, Language, and Hearing Research, 40, 723–740.

32.

Thorndike

R. M.

Thorndike-Christ

(2010). Measurement and evaluation in psychology and education (8th ed.). Boston, MA: Pearson Education.

33.

Tunmer

Chapman

Greaney

Prochnow

Arrow

(2013). Why the New Zealand national literacy strategy has failed and what can be done about it: Evidence from the Progress in International Reading Literacy Study (PIRLS) 2011 and Reading Recovery monitoring reports. Australian Journal of Learning Difficulties, 18, 139–180.

34.

Tymms

Merrell

Buckley

(2016). Children’s development at the start of school in Scotland and the progress made during their first school year: An analysis of PIPS baseline and follow-up assessment data. Retrieved from http://www.gov.scot/Publications/2015/12/5532/0

35.

United Kingdom, Department for Education. (2014). Statutory framework for the early years foundation stage (EYFS). Retrieved from https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/335504/EYFS_framework_from_1_September_2014__with_clarification_note.pdf

36.

Wiig

Secord

Semel

(2006). Clinical evaluations of language fundamentals preschool—Australian and New Zealand Edition (2nd ed.). San Antonio, TX: Psychological Corporation.