Examining the psychometric properties of the ELLECCT: A commentary on Weadman,Serry and Snow (2022)

Commentary

High-quality teacher–child interactions can support improved child language outcomes, and observational tools seeking to define and capture such interactions play a key role in supporting the occurrence and quality of language supporting exchanges. By providing a research-based ‘gold standard’, these instruments can inform practitioners and support the implementation and monitoring of efforts to improve practice. They can also serve as a tool for researchers investigating the impact of interventions and training on teaching practice, and the efficacy of strategies and approaches in terms of language outcomes. If standardised, they can provide a means of exploring the relationship between contextual factors, such as group size, and the quality of extratextual talk. To meet these demands, tools need to be valid, reliable and easy to use. Here, we consider the development of the ‘Emergent Literacy and Language Early Childhood Checklist for Teachers’ (ELLECCT) (Weadman et al., 2021) and the ways in which it can contribute to the current tools available to capture interactions around book reading. An important point to note at the outset is this is not a tool that could be easily used by teachers themselves but rather a tool that evaluates teachers. Ratings are based on transcriptions which are scored for extratextual utterances, non-linguistic features and paralinguistic aspects. Given the complex data collection and scoring, it is our view that the ELLECCT’s primary use will be for research: professionals who are not trained in text analysis are unlikely to find it easy to use.

Weadman et al. (2021) explain that the development of the ELLECCT went through four phases: content validity, face validity, intra-rater reliability and inter-rater reliability. Indeed, one of the strengths of the article is the detail that is provided for each phase of development. A second major strength of the study is the focus on a specific domain of practice rather than capturing the general features of classroom activities. From a psychometric viewpoint, the validity of observational tools is enhanced by close alignment between instruments and the purposes for which they are employed (American Educational Research Association et al., 2014). Furthermore, by adopting a narrower focus on a specific dimension of practice, it is possible to construct a more fine-grained set of criteria against which practice can be evaluated. This increases the likelihood that the course of professional growth (Markussen-Brown et al., 2017), and relationships between specific practices and student outcomes (Connor, 2013), will be accurately captured. Domain-specific measurement approaches should be welcomed, especially when they incorporate aspects which have previously been under-investigated. The current observational measure aims to incorporate both oral language and emergent literacy strategies in combination with paralinguistic and nonverbal features that are known to support preschool children’s engagement with text, thus making an important new contribution to the set of measures available.

What is the evidence base for the focus on interactive book reading and the inclusion of specific items in the ELLECCT? There is a growing literature supporting the role of interactive book reading in enhancing children’s oral language and literacy skills both in quality first teaching (Cabell et al., 2019; Dobinson & Dockrell, 2021) and in more targeted interventions for children needing extra support to develop their oral and written language skills (Towson et al., 2021). As such, the domain chosen to capture teacher talk provides a good basis for coding activities to enhance language and literacy. The items for inclusion in the ELLECCT were created in two ways. First, a review process and an evaluation by 10 practitioners were used to establish content validity. The identification of items for content validity was completed through an extensive but not systematic review of the relevant literature. This raises the question as to how the authors arrived at the conclusion that only items which were ‘well established’ in the literature were selected and why items which failed to reach the content validity threshold were included because they were ‘deemed important’. Second, a Delphi process with 12 practitioners was used to establish face validity. Application of the Delphi technique has been increasing in recent years as a means of deriving expert consensus. There are several advantages to this approach, but we need to remember that the Delphi technique is based on opinion, and a consensus does not necessarily mean the correct items have been identified. For a Delphi process to be illuminatory, the panel should represent a wide range of expertise and experience. Although Weadman et al. (2021) explain their efforts to maximise diversity while avoiding potential difficulties with the panel reaching consensus, it is not clear that 12 participants (6 speech and language therapists and 6 early childhood teachers) from a narrow geographical area (i.e. two Australian states) meet this criterion. In sum, there is indicative evidence to support claims of content and face validity, but the next steps in the ELLECCT’s development require further benchmarking of the items to the evidence base and, we would argue, benchmarking against other reliable and valid measures.

Analysis of intra- and inter-rater reliability was based on an extensive set of video recordings of a single observation (N = 32) of representative early childhood teachers reading a book of their choice to the preschoolers in their class. Most of the observations included were whole-class shared book reading; yet as book reading is more interactive in small groups (Morrow & Smith, 1990), it could be argued that small group shared book reading would be the preferred context in which to validate the ELLECCT. Excellent intra-rater reliability was obtained across the first four ELLECCT sections. For the inter-rater reliability, excellent agreement was obtained for ‘prompts’, ‘responsive statements’ and ‘print knowledge’. By contrast, much lower inter-rater reliability was evident for ‘paralinguistic and nonverbal features’. This is disappointing given these items are part of the novel contribution of the tool. Unlike other sections in the checklist, this section was scored using categorical data and as such may be less sensitive to capturing nuances that might occur in book reading.

In conclusion, the rationale for an approach that measures both verbal and nonverbal engagement is well articulated, and a tool like ELLECCT has the potential to inform practice. Weadman et al. (2021) have collated initial evidence regarding the validity and reliability of the checklist which should now be extended to further explore its use in different contexts and with different stories read with the children. Specifically, the factor structure of the tool, the predictive validity of its various subscales in relation to child language outcomes and the concurrent and divergent validity of the tool in relation to existing instruments all should be explored. Conducting these investigations with larger samples including a diverse range of practitioners, children and settings will lend weight to the conclusions which can be drawn when the measure is used. While time-consuming, these efforts would be worthwhile, potentially leading to the consistent application of a cohesive measure across interactive reading studies, something which would be of considerable benefit to both research and practice in this domain.