Literacy Assessment Via Telepractice Is Comparable to Face-to-Face Assessment in Children with Reading Difficulties Living in Rural Australia

Abstract

Background/Introduction:

Literacy difficulties have significant long-term impacts on individuals, and therefore early identification and intervention are critical. Access to experienced professionals who conduct standardized literacy assessments with children is limited in rural and remote areas. The emerging literature supports the feasibility of using telepractice to overcome barriers to accessing specialist literacy assessment. The current study sought to determine the feasibility and reliability of telepractice assessments, using consumer-grade technology, in children with reading difficulties.

Materials and Methods:

Thirty-seven children, aged 8 to 12 years, with reading difficulties, attended a multidisciplinary reading clinic. Children completed literacy assessments delivered via a web-based application by a remotely located research assistant. A teacher was stationed with the child and coscored the assessments. Scores and qualitative observations of the two assessors were compared.

Results:

Spearman's correlation analyses revealed strong agreement between telepractice- and face-to-face-rated scores (r = 0.79–0.99). Bland-Altman plots indicated excellent agreement between derived scores. Parents reported a high degree of comfort with the telepractice assessments. Clinicians reported the audio and video quality was sound in most cases.

Discussion/Conclusions:

Web-based technology can enable remote delivery of literacy assessments. The technology has the potential to increase the availability of assessments to meet the needs of children who live remotely, in a timely manner and at their family's convenience.

Introduction

Literacy difficulties are the most frequently diagnosed neurodevelopmental disorders in childhood with prevalence estimates of 10–16% in the Australian population.^1
–3 The impact of such difficulties on education, employment, and mental health outcomes is well understood.^4
–6 Identification of such difficulties through standardized literacy assessments is therefore vital in determining the need for assistance, informing intervention plans⁷, and reducing negative outcomes. Furthermore, such assessments are necessary to evaluate the efficacy of, and provide feedback regarding, literacy interventions. Psychologists and specialist teachers play an important role in the delivery of literacy assessments. Unfortunately, access to such professionals for children living in rural and remote areas is limited by the lack of availability of those with specific expertise in the assessment of literacy difficulties. Furthermore, the demand/need for these services is high for children in rural/remote areas given that they consistently have poorer academic performance than their urban counterparts.^8,9 Indeed, in the 2017 Australian National Assessment Program Literacy and Numeracy report,¹⁰ the highest percentage of children performing below the national standard for reading attended schools in rural and remote locations.

Telepractice, or the use of information and communication technologies to deliver services, may help improve access to literacy assessments for children in distal geolocations. To our knowledge, there have only been two studies that have examined the use of telepractice in individuals with literacy difficulties. In a 2016 feasibility study,¹¹ two adults with phonological post-stroke reading difficulties showed improvements in their ability to read target words after a period of intervention conducted remotely using videoconference technology. Waite et al.¹² investigated the reliability of internet-based assessments for 20 children aged 8–13 years, using several standardized tests including the Queensland University Inventory of Literacy, South Australian Spelling Test, and Neale Analysis of Reading Ability.¹² The results supported the reliability of telepractice assessments, however, issues with audio latency, break-up, and echo were noted. These issues made it difficult for assessors to provide timely prompts and were a distraction for participants. Furthermore, Waite's study did not examine participant and assessor acceptability of telepractice.

Current Paper

The aim of the current paper is to help address the clinical need for determining the feasibility and reliability of telepractice assessments. The limitations of the existing literature indicated a need for additional high-quality research using larger sample sizes, a wide range of the most current editions of tests, and technology with improved audio and visual quality. These limitations provided the impetus and rationale for the current study, which examined a group of children with learning difficulties. The study employed a comprehensive battery of standardized psychometrically sound literacy tests and involved clinical and teaching staff in real-world settings who took an active role in the process of setting up the remote assessment system as well as administering tests. The equipment utilized in the current study consisted of consumer-grade technology available for purchase at most office supply stores and a flexible, web-based application. The current paper aims to determine whether literacy assessments can be administered reliably via remote testing compared with face-to-face evaluation. The study also sought to obtain feedback from teachers and parents in relation to the child's behaviors and about their satisfaction with telepractice.

Materials and Methods

Ethics and Permissions

This study was approved by The Sydney Children's Hospital Network (LNR/16/SCHN/488) Ethics Committee and the New South Wales (NSW) Department of Education. All eligible children and parents or care providers, seen in the NSW Centre for Effective Reading (the Centre), were approached, and only those who signed informed consent were included.

Selection and Description of Participants

Thirty seven children (n = 37) presented to the Centre, a joint NSW Department of Education and NSW Health initiative. Assessments occurred between January and December 2017, and participants ranged in age from 8 to 12 years. Children with a diagnosis of Specific Learning Disorder with impairment in reading, who agreed to participate, were included. There were no exclusion criteria based on intellectual ability or language level.

Telepractice Technology

The assessments were delivered via a web-based application, “Coviu,” developed by the Commonwealth Scientific and Industrial Research Organisation in Australia, which uses peer-to-peer communication with full encryption, synchronized image viewing, remotely visible click-markers for pointing at images, and videoconference functionality. Inexpensive modern high-quality webcams and speakers were used, along with commercial touch screens and standard-issue computer equipment used in schools and in the hospital where the research was conducted.

Measures

Permission was granted for the use of copyrighted test material by the copyright holders of the tests. Pearson Australia Group Pty Ltd. (Woodcock Reading Mastery Test-III), MultiLit Pty Ltd. (MultiLit Sight Words and MultiLit Word Attack), Dalwood Assessment Centre (Dalwood Spelling Test [DST]) and Pro-Ed, Inc. (TOWRE-2). With this permission, all of the test materials were scanned and digitized for use in the telepractice sessions.

Woodcock Reading Mastery Tests-Third Edition

The Woodcock Reading Mastery Tests-Third Edition (WRMT-III)¹³ is a norm-referenced assessment that measures several aspects of reading achievement. It provides information about an examinee's reading performance that can be used to develop effective, individually tailored reading intervention programs. For the purposes of the assessment, the following subtests were administered: Word Identification, a measure of sight-word vocabulary; Word Attack, which measures the ability to decode unknown words; Passage Comprehension which measures an individual's ability to read and understand a short passage by asking them to supply a missing word.

Test of Word Reading Efficiency-Second Edition

The Test of Word Reading Efficiency-Second Edition (TOWRE-2)¹⁴ provides an efficient means of monitoring the development of two different word reading skills that are critical in the development of overall reading ability. The Sight Word Efficiency subtest measures a child's ability to recognize familiar words as whole units or sight words. The Phonemic Decoding Efficiency subtest measures a child's ability to sound out words quickly and accurately. The Pearson correlation coefficient based on independent scoring of a subset of protocols drawn from children in the normative sample was 0.99, providing evidence supporting the test's inter-scorer reliability.

MultiLit

The MultiLit Sight Words Test¹⁵ includes 200 words that have been identified as being high frequency words in the reading material of primary school aged children. The ability to read a number of words by sight facilitates reading fluency, which impacts comprehension.

The MultiLit Word Attack Test assesses a child's letter-sound knowledge and blending skills (single letters and letter combinations) as well as the ability to apply this knowledge to reading unfamiliar words.

Dalwood Spelling Test

The DST¹⁶ is a standardized test of spelling achievement for children from kindergarten to year 10. It has been designed to provide a fast and reliable means of identifying children with spelling weaknesses and for evaluating skill progress over time.

Surveys

Parents or carers completed a brief survey indicating the perceived comfort level of their child during the assessment as well as their own comfort level. There were questions requiring both a response on a Likert scale and an open-ended question to provide the parent or carer the opportunity to express additional thoughts or to provide feedback regarding the telepractice assessment.

Face-to-face teachers completed a project feedback form, which addressed the child's behavior and performance during the assessment. Teachers were asked to provide ratings on a Likert scale and make comments regarding compliance, anxiety, attention and fatigue. In addition, their thoughts around the use of technology, including the visual and audio quality throughout the assessment, and other comments were solicited.

Delivery Method

The study involved children undergoing assessment that was delivered via a web-based application by a research assistant (RA). The children were located remotely (in Dubbo, Wagga Wagga, Westmead, or Manly) and accompanied by local staff (face-to-face teacher). The RA (online) leading the test administration was located in Westmead, NSW. The RA interacted with the child remotely and presented test items. Scoring of the items was completed simultaneously by the RA and the face-to-face teacher.

The order of test administrations was randomized. The RA and the face-to-face teacher independently scored the assessments as they were delivered. This simultaneous scoring allowed for comparison of the ratings obtained in each environment while eliminating the test–retest or test learning effects that may have occurred if the child was assessed on two separate occasions.

Training of Staff

The study investigators included an RA (psychologist) and teachers with modest experience in technology. The investigators were trained and familiarized with the equipment and study procedures in one session. A manual was also developed for support staff to facilitate the telepractice assessments.

Statistical Analysis

The independent variable in this phase was the condition under which the assessment was conducted (telepractice vs. face-to-face). The dependent variables were: (i) number of sessions completed and local staff ratings of audio and visual quality, which provided a measure of feasibility; (ii) the literacy assessment scores given by the online and face-to-face staff, which were compared to provide a measure of reliability; and (iii) behavioral observation scores made by the local staff and RA which provided a measure of the tolerability of the Coviu application.

The agreement between the telepractice and face-to-face scores was examined using Bland–Altman measures of agreement.¹⁷ Spearman's correlation was used to provide measures of correlation (interrater reliability) for standard scores. Parent and teacher survey data were analyzed qualitatively.

Results

Thirty-seven children underwent literacy assessments with the RA administering eight assessments to each student. The assessment generally took 40–60 min to administer. The median age of participants was 10 years 1 month (age range 8–11 years), and 20 participants were male. All children were enrolled in regular mainstream classes. The median school year was 4 (range 2–6 years). Ten of the 37 children had a co-occurring diagnosis of attention-deficit/hyperactivity disorder (ADHD) confirmed by a pediatrician (23rd author). Scores determined by the face-to-face teachers were taken as the gold standard in the comparisons. On average, children's age equivalent scores on the WRMT-III Passage Comprehension (face-to-face) was 2.52 years below their chronological age.

Table 1 depicts the Spearman's correlation coefficients comparing the scores of the RA to the face-to-face teacher.

Table 1.

Spearman's Correlation Coefficients Comparing Scores from the Online to Face-To-Face Administration Modes

COMPOSITE	SPEARMAN'S CORRELATION COEFFICIENT r	SIGNIFICANCE (TWO-TAILED) p -VALUE
Multilit Sight Words raw	0.947	<0.01
Multilit Word Attack PL	0.793	<0.01
Dalwood Spelling raw	0.997	<0.01
WRMT Word ID SS	0.985	<0.01
WRMT Word Attack SS	0.962	<0.01
WRMT Passage Comp SS	0.988	<0.01
TOWRE SWE SS	0.987	<0.01
TOWRE PDE SS	0.859	<0.01

	SPEARMAN'S CORRELATION NO ADHD r	SPEARMAN'S CORRELATION ADHD r	p -VALUE
Multilit Sight Words raw	0.934	0.918	<0.01
Multilit Word Attack PL	0.767	0.811	<0.01
Dalwood Spelling raw	0.996	0.998	<0.01
WRMT Word ID SS	0.984	0.971	<0.01
WRMT Word Attack SS	0.945	0.988	<0.01
WRMT Passage Comp SS	0.988	0.957	<0.01
TOWRE SWE SS	0.986	0.990	<0.01
TOWRE PDE SS	0.875	0.862	<0.01

WRMT, Woodcock Reading Mastery Tests; TOWRE, Test of Word Reading Efficiency; ADHD, attention-deficit/hyperactivity disorder.

Spearman's correlation analyses revealed very strong agreement between telepractice and face-to-face rated scores for all but one test. The test that yielded the lowest agreement nonetheless fell at the upper end of the strong-agreement range based on Spearman's correlation coefficient interpretation guidelines.¹⁸

On some occasions, there were technical difficulties (thought likely to be due to insufficient bandwidth availability) leading to problems setting up the document camera, temporary screen freezing, the need to refresh the Coviu connection, or restart the browser. Although these difficulties caused slight delays in telepractice assessment, they did not prevent valid completion of the evaluation.

The analysis procedure proposed by Altman and Bland¹⁷ allows the visualization of the mean differences and the extreme limits of agreement (±1.96 standard deviation of the difference) presented by the remote and face-to-face scores. Bland–Altman plots (Fig. 1) indicated that, in general, the mean differences between face-to-face and videoconference modes of assessment were generally small with the largest standard score discrepancy being 14 points in the TOWRE Phonemic Decoding subtest.

Fig. 1.

Bland–Altman plots and scatterplots illustrating interrater agreement and reliability across subtests. TOWRE-2, The Test of Word Reading Efficiency-Second Edition; WRMT-III, The Woodcock Reading Mastery Tests-Third Edition.

Assessors reported high levels of comfort regarding the use of telepractice to deliver assessments. Many of the open-ended statements relating to the telepractice assessment experience reflected a high degree of acceptability. Parents of participants reported overall positive behaviors in the children (see Table 2 for a sample of responses).

Table 2.

Parent Comments from the Satisfaction Survey Regarding the Telehealth Assessment

PARENT	COMMENT
3	It's a great idea for families not from Sydney. (Child) loved the touch screen.
5	(Child) really enjoyed working with (RA) and using computer. I like (RA)'s positivity and encouragement.
10	Great Idea!! My son was very excited to do it and really enjoyed it. Commented on the way home it was “Awesome.” Thanks.
12	(Child) enjoyed doing it on the screen. She said she could clearly understand what to do. (RA) was nice.
45	(Child) really liked the remote assessment. She appeared comfortable and relaxed during the assessment and did not have any concerns. I believe the remote assessment would be beneficial for remote children.

RA, research assistant.

Discussion

The current study, the largest of its kind to date, extends prior research and demonstrates the feasibility, reliability, and acceptability of telepractice literacy assessments. A high level of agreement between literacy assessments conducted face-to-face and via telepractice was found, similar to the results of the study conducted by Waite et al.¹² The reliable, remote evaluation of literacy skills is important to clinicians and teachers who work with children with literacy difficulties for at least two reasons: (i) such difficulties are common, and many children living remotely do not have easy access to assessments, and (ii) children who have completed an intervention program can have postintervention assessments conducted in a timely, standardized, and consistent manner.

The correlations between face-to-face and telepractice evaluations were slightly lower for the MultiLit Word Attack Test (placement level) compared with other literacy scores in our study (Spearman's correlation coefficient of 0.79 compared with others at 0.86 and, most generally, above 0.95). The lower agreement may be due to the less than ideal audio quality necessary to hear the child's reading of nonwords partly due to the child's position relative to the microphone which was embedded in the webcam. This positional issue problem may have been compounded when decoding nonwords, a task which may have evoked higher anxiety in children with reading difficulties and which may have led to clarity difficulties.

This study has several strengths, including a larger sample size than the telepractice studies published to date, inclusion of participants with a relatively wide range of intellectual abilities (some with comorbid diagnoses of ADHD), and use of the most recent versions of literacy tests. This study demonstrated real-world use of consumer grade materials and thus may have immediate applications in other real-world contexts.

To our knowledge, this is one of two studies of remote literacy assessments in children with literacy difficulties using a comprehensive battery of tests and with typical assessors of literacy tests who have modest technological knowledge. This study collected rich qualitative data that extended beyond face-to-face versus telepractice assessment score comparisons. Demonstrating the reliability of literacy assessments conducted via web-based technology has the potential to extend the reach of such assessments to children in remote locations nationwide. It is hoped that this approach will be adopted more widely to increase access to these valuable assessments, thereby increasing equity to our rural and remote population.

The limitations of the study should be considered when interpreting these results. We obtained a convenience sample. The enrolment of participants was selective for children who have significant literacy difficulties who have not responded to universal and targeted interventions. The generalizability of the results to children with milder or no difficulties is unclear, given that the selection was not from the general population. The study should be replicated in more diverse groups to ensure that results are easily generalizable to the broader population. Although our study found high ratings overall for the quality of audio and visual aspects of the web-based technology, there were occasional technical issues based around connectivity (suboptimal bandwidth). These technical issues were characterized by slow connections, lags in responses, and dropped calls and were addressed by providing a dedicated broadband cable connection and increasing the bandwidth for multiple users. Dedicated connections need to be considered when including web-based technology as a service delivery option. This technology may not always be possible for all services and organizations, given the potential costs. The remote location of the assessor also presents challenges. This person cannot directly manage a child's behavior or activities, and so an individual is needed to be present on site to support the child, for example, to provide certain physical materials when needed or provide behavioral management when the child is having difficulties engaging in tasks or loses concentration. It is interesting to note that, in our study, there were minimal negative effects on concentration, level of engagement, and participation in the telepractice assessments even in children diagnosed with ADHD. Regardless, in the event of technical difficulties, it is essential to have troubleshooting information and contingency plans available to allow the assessments to be validly completed.

Conclusions

Our study indicates that web-based technology can allow assessments of literacy skills to be conducted remotely. This technology has the potential to increase the availability of assessments to meet the ever-growing need to support children who live remotely from services, in a timely manner, and at their family's convenience. Telepractice has the potential to improve time and caseload management efficiency therefore resulting in more cost-effective service delivery. In our model, telepractice assessment required a second person to be present next to the child in each remote location to help with equipment positioning. Whether another staff member, such as a teacher's aide or nonteaching staff, would be able to serve in this role may impact the ultimate efficiency and cost savings of implementing the telepractice. In addition, ethical and privacy issues regarding the exposure of test materials and confidential results to untrained individuals need to be taken into consideration.

Footnotes

Acknowledgments

This study was funded by the NSW Department of Education. The Department was involved in discussions on the design of the study, editing flow charts, planning the information and consent forms, and preparing the social stories.

Disclosure Statement

S.P. is the CEO of Coviu, the Web-based technology used in this study, which may pose a competing financial interest in the results of the submitted manuscript. The remaining authors do not have any commercial associations that might create a conflict of interest in connection with the submitted manuscript.

References

Australian Dyslexia Association, Inc., Queensland. Dyslexia in Australia. 2014. Available at http://dyslexiaassociation.org.au/what-is-dyslexia

Castles

, Wheldall

, Nayton

. Should we do away with dyslexia? The conversation website. 2014. Available at http://theconversation.com/should-we-do-away-with-dyslexia-24027

Moll

, Kunze

, Neuhoff

, Bruder

, Schulte-Körne

. Specific learning disorder: Prevalence and gender differences. PLoS One, 2014; 9:e103537.

Beitchman

, Young

. Learning disorders with a special emphasis on reading disorders: A review of the past 10 years. J Am Acad Child Adolesc Psychiatry, 1997; 36:1020–1032.

Kempe

, Gustafson

, Samuelsson

. A longitudinal study of early reading difficulties and subsequent problem behaviors. Scand J Psychol, 2011; 52:242–250.

Mammarella

, Ghisi

, Bomba

, Bottesi

, Caviola

, Broggi

, et al. Anxiety and depression in children with nonverbal learning disabilities, reading disabilities, or typical development. J Learn Disabil, 2016; 49:130–139.

Griffin

, Nix

Assessment and reporting: A new approach. Sydney, Australia: Harcourt Brace Jovanovich, 1991.

Squires

Responding to isolation and educational disadvantage. Educ Rural Australia, 2003; 13:24.

Deparment of Education. Rural and remote education blueprint,. 2014. Available at https://education.nsw.gov.au/teaching-and-learning/curriculum/rural-and-distance-education/rural-and-remote-education/blueprint-for-action (last accessed on July 11, 2018 ).

10.

Committee MSO. National assessment program literacy and numeracy. Achievement in reading, writing, language conventions and numeracy. Canberra, Australia: MCEECDYA Senior Officials Committee. 2009. Available at https://www.nap.edu.au/docs/default-source/default-document-library/naplan-national-report-2017_final_04dec2017.pdf?sfvrsn=0 (last accessed on July 11, 2018 ).

11.

Getz

, Snider

, Brennan

, Friedman

. Successful remote delivery of a treatment for phonological alexia via telerehab. Neuropsychol Rehabil, 2016; 26:584–609.

12.

Waite

, Theodoros

, Russell

, Cahill

. Assessment of children's literacy via an internet-based telehealth system. Telemed J E Health, 2010; 16:564–575.

13.

Woodcock

RW.

Woodcock reading mastery tests, 3rd ed. Bloomington, MN: Pearson, 2010.

14.

Torgesen

, Rashotte

, Wagner

. TOWRE: Test of word reading efficiency:. Austin, TX: Pro-Ed, 1999.

15.

MultiLit. In: Wheldall

, Macquarie University, Macquarie University Special Education Centre, MultiLit Research Unit, eds. MULTILIT sight words: Manual (revised)/MULTILIT. Macquarie Park, New South Wales: MultiLit Pty Ltd., 2007.

16.

Dalwood Assessment Center, ed. Dalwood spelling test/Dalwood Assessment Centre. Seaforth, NSW: Dalwood Assessment Centre, 2008.

17.

Altman

, Bland

. Measurement in medicine: The analysis of method comparison studies. Statistician, 1983; 32:307–317.

18.

Cohen

Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Earlbaum Associates, 1988:20–26.