Interrater Reliability of Early Childhood Education Professionals Involved in Developmental Surveillance for Autism Spectrum Disorder and Related Conditions

Abstract

VALID AND RELIABLE TOOLS have recently been developed to accurately detect early signs of Autism Spectrum Disorder (ASD) and other developmental challenges in children as young as 12 months of age. Translation of research findings to practice and policy through routine implementation of evidence-based tools in the community, particularly early childhood education and childcare settings, is limited. This study establishes that the interrater reliability (IRR) of early childhood educators in administering the Social Attention and Communication Surveillance System (SACS-R) is very high (k = 0.909). This paper reports the results of the first step in the Right Kids, Right Time, Right Services project—a prospective cohort study that aims to implement and evaluate routine developmental surveillance for early signs of social and communication challenges in young children in childcare settings.

Get full access to this article

View all access options for this article.

References

American Psychiatric Association. (1994). Diagnostic and Statistical Manual of Mental Disorders (4th ed.). Washington, DC: American Psychiatric Association.

American Psychiatric Association. (2013). Diagnostic Statistical Manual of Mental Disorders (5th ed.). Washington, DC: American Psychiatric Association.

Armstrong

M. F.

, & Goldfeld

(2008). Systems of early detection in Australian communities: The use of a developmental concern questionnaire to link services. Australian Journal of Advanced Nursing, 25(3), 36–42.

Australian Bureau of Statistics (ABS). (2012). Autism in Australia. Retrieved from www.abs.gov.au/AUSSTATS/abs@.nsf/Lookup/4428.0Main+Features12012?OpenDocument.

Australian Government Department of Education Employment and Workplace Relations (DEEWR). (2009). Belonging, Being & Becoming: The Early Years Learning Framework for Australia. Canberra, ACT: Commonwealth of Australia. Retrieved from https://docs.education.gov.au/system/files/doc/other/belonging_being_and_becoming_the_early_years_learning_framework_for_australia.pdf.

Bailey

D. B.

, Hebbeler

, Scarborough

, Spiker

, & Mallik

(2004). First experiences with early intervention: A national perspective. Pediatrics, 113(4), 887–896. Retrieved from www.scopus.com/inward/record.url?eid=2-s2.0-1842481946&partnerID=40&md5=94522adae89df112a6fb0d93014cde8b.

Baio

(2012). Prevalence of autism spectrum disorders—Autism and Developmental Disabilities Monitoring Network, 14 sites, United States, 2008. Morbidity and Mortality Weekly Report, 61(SS-3), 1–19. Retrieved from www.scopus.com/inward/record.url?eid=2-s2.0-84859394070&partnerID=40&md5=91d81545fcd52d631021b568a07802ac.

Barbaro

, & Dissanayake

(2009). Autism spectrum disorders in infancy and toddlerhood: A review of the evidence on early signs, early identification tools, and early diagnosis. Journal of Developmental and Behavioral Pediatrics, 30(5), 447–459. Retrieved from www.scopus.com/inward/record.url?eid=2-s2.0-74049154641&partnerID=40&md5=f4c048441560679e9792e27a5019c0f1.

Barbaro

, & Dissanayake

(2010). Prospective identification of autism spectrum disorders in infancy and toddlerhood using developmental surveillance: The social attention and communication study. Journal of Developmental and Behavioral Pediatrics, 31(5), 376–385. https://dx-doi-org.web.bisu.edu.cn/10.1097/DBP.0b013e3181df7f3c

10.

Bent

C. A.

, Dissanayake

, & Barbaro

(2015). Mapping the diagnosis of autism spectrum disorders in children aged under 7 years in Australia, 2010–2012. Medical Journal of Australia, 202(6), 317–321. https://dx-doi-org.web.bisu.edu.cn/10.5694/mja14.00328

11.

Boyle

C. A.

, Boulet

, Schieve

L. A.

, Cohen

R. A.

, Blumberg

S. J.

, Yeargin-Allsopp

& Kogan

M. D.

(2011). Trends in the prevalence of developmental disabilities in US children, 1997–2008. Pediatrics, 127(6). https://dx-doi-org.web.bisu.edu.cn/10.1542/peds.2010-2989

12.

Branson

, Vigil

D. C.

, & Bingham

(2008). Community childcare providers' role in the early detection of autism spectrum disorders. Early Childhood Education Journal, 35(6), 523–530. https://dx-doi-org.web.bisu.edu.cn/10.1007/s10643-008-0243-6

13.

Centers for Disease Control and Prevention. (2014). Prevalence of autism spectrum disorder among children aged 8 years—Autism and Developmental Diabilities Monitoring Network, 11 sites, Unites States, 2010. Morbidity and Mortality Weekly Report, 63(2), 1–22. Retrieved from www.cdc.gov/mmwr/pdf/ss/ss6302.pdf.

14.

Chakrabarti

, & Fombonne

(2005). Pervasive developmental disorders in preschool children: Confirmation of high prevalence. American Journal of Psychiatry, 162(6), 1133–1141. https://dx-doi-org.web.bisu.edu.cn/10.1176/appi.ajp.162.6.1133

15.

Charman

(2003). Screening and surveillance for autism spectrum disorder in research and practice. Early Child Development and Care, 173(4), 363–374. Retrieved from www.scopus.com/inward/record.url?eid=2-s2.0-33750332670&partnerID=40&md5=07116d849073f73cd8ae97418e05273a.

16.

Charman

, & Baron-Cohen

(2006). Screening for autism spectrum disorders in populations: Progress, challenges, and questions for future research and practice. In Charman

& Stone

(Eds.), Social and communication development in autism spectrum disorders (pp. 63–82). New York, NY: Guilford Press.

17.

Cohen

(1960). A coefficient of agreement for nominal scales. Educational and Pscyhological Measurement, 20(1), 37–46.

18.

Cooperative Research Centre for Living with Autism Spectrum Disorders (Autism CRC). (2014). Annual Report 2013–2014. Indooroopilly, Qld: Autism CRC, The University of Queensland.

19.

Council of Australian Governments (COAG). (2009). Regulation Impact Statement for Early Childhood Education and Care Quality Reforms. Canberra, ACT: Australian Government. Retrieved from http://ncac.acecqa.gov.au/reports/report-documents/NCAC%20response%20to%20RIS.pdf.

20.

Council on Children with Disabilities, Section on Developmental Behavioural Paediatrics, Bright Futures Steering Committee, Medical Home Initiatives for Children with Special Needs Project Advisory Committee. (2006). Identifying infants and young children with developmental disorders in the medical home: an algorithm for developmental surveillance and screening. Pediatrics, 118(1), 405–420. https://dx-doi-org.web.bisu.edu.cn/10.1542/peds.2006-1231.

21.

Dawson

(2008). Early behavioral intervention, brain plasticity, and the prevention of autism spectrum disorder. Development and Psychopathology, 20(3), 775–803. Retrieved from www.scopus.com/inward/record.url?eid=2-s2.0-48949116395&partnerID=40&md5=84bfd62762638b344feb2c146faa0642.

22.

Dawson

, Rogers

, Munson

, Smith

, Winter

, Greenson

, … Varley

(2010). Randomized, controlled trial of an intervention for toddlers with autism: The early start Denver model. Pediatrics, 125(1), e17–e23. Retrieved from www.scopus.com/inward/record.url?eid=2-s2.0-74049153754&partnerID=40&md5=219677c9035e071f860a458c55c7e711.

23.

Department of Education and Training (DET). (2016). Australian early development census national report 2015. Retrieved from www.aedc.gov.au/resources/detail/2015-aedc-national-report.

24.

Dereu

, Warreyn

, Raymaekers

, Meirsschaut

, Pattyn

, Schietecatte

, & Roeyers

(2010). Screening for autism spectrum disorders in flemish day-care centres with the checklist for early signs of developmental disorders. Journal of Autism and Developmental Disorders, 40(10), 1247–1258. https://dx-doi-org.web.bisu.edu.cn/10.1007/s10803-010-0984-0

25.

Garg

, Lillystone

, Dossetor

, Kefford

, & Chong

(2014). An exploratory survey for understanding perceptions, knowledge and educational needs of general practitioners regarding autistic disorders in NSW, Australia. Journal of Clinical and Diagnostic Research, 8(7), PC01–PC09. Retrieved from www.scopus.com/inward/record.url?eid=2-s2.0-84904809746&partnerID=40&md5=8656ae3ac429dba000ada4a62ccc92d9.

26.

Glascoe

F. P.

(2000). Early detection of developmental and behavioral problems. Pediatrics in Review, 21(8), 272–279; quiz 280. Retrieved from www.scopus.com/inward/record.url?eid=2-s2.0-0034241396&partnerID=40&md5=75acd5e19ff6d18860d0852610410bc8.

27.

Glascoe

F. P.

(2005). Screening for developmental and behavioral problems. Mental Retardation & Developmental Disabilities Research Reviews, 11(3), 173–179. https://dx-doi-org.web.bisu.edu.cn/10.1002/mrdd.20068

28.

Gold Coast Medicare Local. (2012). Gold Coast health and wellbeing plan. Gold Coast, Qld: Australian Government Department of Health and Ageing.

29.

Howlin

, Magiati

, & Charman

(2009). Systematic review of early intensive behavioral interventions for children with autism. American Journal on Intellectual and Developmental Disabilities, 114(1), 23–41. Retrieved from www.scopus.com/inward/record.url?eid=2-s2.0-64949112889&partnerID=40&md5=e9fb1ec1bf28074b81c714d348627252.

30.

Johnson

, & Myers

(2007). Identification and evaluation of children with autism spectrum disorders. Pediatrics, 120(5), 1183–1215. https://dx-doi-org.web.bisu.edu.cn/10.1542/peds.2007-2361

31.

Kellett

(2011). Engaging children and young people. Centre for Children and Young People Background Briefing Series, No. 3. Lismore, NSW: Centre for Children and Young People, Southern Cross University.

32.

King

T. M.

, & Glascoe

F. P.

(2003). Developmental surveillance of infants and young children in pediatric primary care. Current Opinion in Pediatrics, 15(6), 624–629. Retrieved from www.scopus.com/inward/record.url?eid=2-s2.0-0345687433&partnerID=40&md5=d0dd5fc4207a0e15248bc7bb976a63d8.

33.

Kottner

, Audige

, Brorson

, Donner

, Gajewski

B. J.

, Hróbjartsson

, Streiner

D. L.

(2011). Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. International Journal of Nursing Studies, 48(6), 661–671. https://dx-doi-org.web.bisu.edu.cn/10.1016/j.ijnurstu.2011.01.016

34.

Kraemer

H. C.

(1992). Evaluating medical tests. Objective and quantitative guidelines. London, UK: Sage Publications.

35.

Limbos

M. M.

, & Joyce

D. P.

(2011). Comparison of the ASQ and PEDS in screening for developmental delay in children presenting for primary care. Journal of Developmental and Behavioral Pediatrics, 32(7), 499–511. https://dx-doi-org.web.bisu.edu.cn/10.1097/DBP.0b013e31822552e9

36.

Lord

, Risi

, DiLavore

P. S.

, Shulman

, Thurm

, & Pickles

(2006). Autism from 2 to 9 years of age. Archives of General Psychiatry, 63(6), 694–701. Retrieved from www.scopus.com/inward/record.url?eid=2-s2.0-33744909414&partnerID=40&md5=692899978f257e39391ed059bcafb403.

37.

Mandell

D. S.

, Ittenbach

R. F.

, Levy

S. E.

, & Pinto-Martin

J. A.

(2007). Disparities in diagnoses received prior to a diagnosis of autism spectrum disorder. Journal of Autism & Developmental Disorders, 37(9), 1795–1802.

38.

McLean

, & Crais

E. R.

(2004). Procedural considerations in assessing infants and preschoolers with disabilities. In McLean

, Worley

& Bailey

J.D.B.

(Eds.), Assessing infants and preschoolers with special needs (3rd ed., pp. 100–122). Upper Saddle River, NJ: Pearson Education, Inc.

39.

Moore

, & Grove

(2008). Best practice guidelines for parental involvement in monitoring and assessing young children. Melbourne, Vic.: Early Childhood Programs Division Office for Children and Early Childhood Development.

40.

National Association for the Education of Young Children (NAEYC). (2009). Where we stand on curriculum, assessment, and program evaluation. Washington, DC: NAEYC.

41.

Olga Tennison Autism Research Centre. (2015). Advances in early detection of Autism. Retrieved 20 January, 2015, from www.latrobe.edu.au/otarc/researchers/early-detection-research.

42.

Oosterling

I. J.

, Wensing

, Swinkels

S. H.

, Van Der Gaag

R. J.

, Visser

J. C.

, Woudenberg

, Buitelaar

J. K.

(2010). Advancing early detection of autism spectrum disorder by applying an integrated two-stage screening approach. Journal of Child Psychology and Psychiatry and Allied Disciplines, 51(3), 250–258. https://dx-doi-org.web.bisu.edu.cn/10.1111/j.1469-7610.2009.02150.x

43.

Pinborough-Zimmerman

, Bilder

, Bakian

, Satterfield

, Carbone

P. S.

, Nangle

B. E.

, McMahon

W. M.

(2011). Sociodemographic risk factors associated with autism spectrum disorders and intellectual disability. Autism Research, 4(6), 438–448. https://dx-doi-org.web.bisu.edu.cn/10.1002/aur.224

44.

Pinto-Martin

, Dunkle

, Earls

, Fliedner

, & Landes

(2005). Understanding and addressing the challenges of disability. Developmental stages of developmental screening: Steps to implementation of a successful program. American Journal of Public Health, 95(11), 1928–1932.

45.

Portney

, & Watkins

(2009). Foundations of clinical research: applications to practice (3rd ed.). Upper Saddle River, NJ: Pearson Education, Inc.

46.

Prior

, Roberts

, Rodger

, & Williams

(2011). A review of the research to identify the most effective models of practice in early intervention for children with autism spectrum disorders. Canberra, ACT: Australian Government Department of Families, Housing, Community Services and Indigenous Affairs. Retrieved from www.dss.gov.au/sites/default/files/documents/10_2014/review_of_the_research_report_2011_0.pdf.

47.

Rogers

S. J.

, & Vismara

L. A.

(2008). Evidence-based comprehensive treatments for early autism. Journal of Clinical Child and Adolescent Psychology, 37(1), 8–38. Retrieved from www.scopus.com/inward/record.url?eid=2-s2.0-42549130526&partnerID=40&md5=090fd92d0c8b99411cf4412dbc5bea63.

48.

Sayal

(2006). Annotation: Pathways to care for children with mental health problems. Journal of Child Psychology and Psychiatry and Allied Disciplines, 47(7), 649–659. Retrieved from www.scopus.com/inward/record.url?eid=2-s2.0-33745033300&partnerID=40&md5=a53be690460931207a4f7a92ee1c1521.

49.

Trivette

C. M.

, Dunst

, & Hamby

(2004). Sources of variation in consequences of everyday activity settings on child and parent functioning. Perspectives in Education, 22(2), 17–35. Retrieved from www.scopus.com/inward/record.url?eid=2-s2.0-4043111376&partnerID=40&md5=dfaba88c0385301ef714a0ace2808164.

50.

United States Department of Health and Human Services. (2014). Maternal, infant and child health—Objectives: Morbidity and mortality. Healthy People 2020. Washington, DC: United States Department of Health and Human Services. Retrieved 18 January, 2016, from www.healthypeople.gov/node/3492/objectives#4873.

51.

Veness

, Prior

, Bavin

, Eadie

, Cini

, & Reilly

(2012). Early indicators of autism spectrum disorders at 12 and 24 months of age: A prospective, longitudinal comparative study. Autism, 16(2), 163–177. https://dx-doi-org.web.bisu.edu.cn/10.1177/1362361311399936

52.

Webb

S. J.

, & Jones

E. J. H.

(2009). Early identification of autism: Early characteristics, onset of symptoms, and diagnostic stability. Infants and Young Children, 22(2), 100–118. Retrieved from www.scopus.com/inward/record.url?eid=2-s2.0-68549127118&partnerID=40&md5=b71f7f09e0603b04fbaa4d88f5158eab.

53.

Webb

S. J.

, Jones

E. J. H.

, Kelly

, & Dawson

(2014). The motivation for very early intervention for infants at high risk for autism spectrum disorders. International Journal of Speech-Language Pathology, 16(1), 36–42. Retrieved from www.scopus.com/inward/record.url?eid=2-s2.0-84892575644&partnerID=40&md5=a02a08753a405f570a58b7de497c0ad6.

54.

Yeargin-Allsopp

, & Boyle

(2002). Overview: The epidemiology of neurodevelopmental disorders. Mental Retardation and Developmental Disabilities Research Reviews, 8(3), 113–116. https://dx-doi-org.web.bisu.edu.cn/10.1002/mrdd.10030

55.

Zwaigenbaum

, Bauman

M. L.

, Stone

W. L.

, Yirmiya

, Estes

, Hansen

R. L.

, … Wetherby

(2015). Early identification of autism spectrum disorder: Recommendations for practice and research. Pediatrics, 136(1), S10–S40. https://dx-doi-org.web.bisu.edu.cn/10.1542/peds.2014-3667C

56.

Zwaigenbaum

, Bryson

, & Garon

(2013). Early identification of autism spectrum disorders. Behavioural Brain Research, 251, 133–146. https://dx-doi-org.web.bisu.edu.cn/10.1016/j.bbr.2013.04.004

57.

Zwaigenbaum

, Bryson

, Lord

, Rogers

, Carter

, Carver

, … Yirmiya

(2009). Clinical assessment and management of toddlers with suspected autism spectrum disorder: Insights from studies of high-risk infants. Pediatrics, 123(5), 1383–1391. Retrieved from www.scopus.com/inward/record.url?eid=2-s2.0-66049107663&partnerID=40&md5=bae10b705e178c47142763d78faa6da8.