Abstract
Newcomer, P. L., & Hammill, D. D. (2009). Pragmatic language observation scale. Austin, TX: Hammill Institute.
In Word of Mouth 15.1, I reviewed the Focus on the Communication Outcomes of Children Under Six (FOCUS), an assessment designed to measure change in preschool children receiving speech and language therapy. Attention is given to how children are communicating in naturalistic contexts. As I mentioned in that review, the Scope of Practice for the American Speech Language Hearing Association (2007) advocates the use of the World Health Organization–International Classification of Functioning (WHO-ICF) for all clinical and research work in speech. The ICF distinguishes between outcomes that are specific skills and outcomes that reflect functioning in naturalistic contexts. The Pragmatic Language Observation Scale (PLOS) is intended as a screener for students who exhibit pragmatic deficits in the classroom, but I believe it could also be used with students with specific language impairment to evaluate the ways they are communicating outside the therapy room.
The PLOS uses teachers’ ratings to assess students’ everyday classroom oral language behaviors. It has 30 items that evaluate students on a variety of communication skills, the majority of which are considered pragmatic skills (e.g., shares information, sticks to the topic when speaking, adjusts language to different social situations), but items also assess phonological (e.g., has intelligible speech), semantic (e.g., retrieves words quickly), and syntactic skills (e.g., uses acceptable grammar). The PLOS is designed for students ages 8 years, 0 months through 17 years, 11 months. Teachers or professionals knowledgeable of the student rate the 30 communication behaviors on a 5-point Likert-type scale (i.e., 1 and 2 = below average, 3 = average, 4 and 5 = above average). The rating can be completed in 5 to 10 min.
In completing the PLOS, the teacher or examiner compares the target students to other students in the class or age-range. The single form for the PLOS has the rating scale on one side and the summary sheet on the other. The student’s total score is converted to a standard score (called the Pragmatic Language Observation Index [PLOI]) and a percentile rank. Based on the PLOI, the examiner selects the best descriptive term—very poor, poor, below average, average, above average. There is also a section for interpreting the difference between the PLOI and the index score on another language test. This could be useful for students who may demonstrate language skills in structured test situations but who do not use them in social situations.
The PLOS can be used to justify a referral, expand the scope of language assessment, and compare teacher ratings with students’ scores on formal tests.
Development
Commonly used language assessments do not include norm-referenced, reliable, valid measures of pragmatic use in social situations. Several standardized assessment tools evaluate students’ declarative pragmatic knowledge, but this knowledge does not necessarily translate to appropriate pragmatic behavior in actual social situations. The Clinical Evaluation of Language Fundamentals–Fifth edition (Semel, Wiig, & Secord, 2003) includes an informal Pragmatic Profile, but does not include normative data or standard scores.
In the initial development of the PLOS, the authors reviewed the literature related to pragmatic language to identify behaviors that pertain to use and understanding of oral language in academic and social settings in schools. The focus was on language use related to conversations and sharing of information, not specific language knowledge such as defining words or labeling parts of speech. An item analysis was conducted on the entire normative sample. Each item was correlated with the total test score to determine the items’ discriminating power. All items met or exceeded the criteria selected for items to be retained. The final roster of items had a mean discriminating power of .82. The authors also reviewed the literature that supports the accuracy of teacher judgments/ratings.
Technical
The PLOS was normed on 994 persons between ages 8 and 17 years in 13 states representing the Northeast, Midwest, South, and West and included a representative sample of students from White, Hispanic, African American, and Asian populations. Parents’ education attainment was also representative of the school-age population. Details of the population sample are provided in the manual. Because students are compared with their same-age peers, scores on the PLOS are not significantly related to age. Hence, one table is provided to convert total PLOS scores to the standard PLOI and a percentile rank.
The PLOS has very high reliability. Across age groups, the items of the test have a coefficient alpha correlation of .98 and a standard error measure of 2. Test retest reliability for 38 11- to 12-year-olds and 41 15- to 16-year-olds was .99.
Criterion-prediction validity was evaluated based on the degree to which scores on the PLOS were related to other measures of oral language, reading, and writing. Scores on the PLOS correlate highly with frequently administered assessments. The PLOS correlates most highly with scores on the Test of Language Development, Fourth Edition (TOLD-4), but also correlates highly on The Test of Reading Comprehension, Fourth Edition (TORC-4; Brown, Wiederholt, & Hammill, 2009) and The Reading Observation Scale (ROS; Wiederholt, Hammill, & Brown, 2009), and moderately on The Written Language Observation Scale (WLOS; Hammill & Larsen, 2009b) and the Test of Written Language, Fourth Edition (TOWL-4; Hammill & Larsen, 2009a). The percentage of agreement between the PLOS and TOLD-4 was .83. The sensitivity index (correctly identifying students with impairment on the PLOS who are language impaired [as identified by language impairment on the TOLD-4]) was .73, and the specificity index (correctly identifying students who are not language impaired) was .85.
Commentary
The Diagnostic and Statistical Manual of Mental Disorders–Fifth edition (DSM-V; American Psychiatric Association, 2013) has a category for language impairment, which comprises deficits in vocabulary, sentence structure, and discourse organization (connecting sentences to describe, explain, or have a conversation), and a separate category for social (pragmatic) communication disorder identified as deficits in using communication for social purposes, such as greeting and sharing information in a manner; impairment of the ability to change communication to match context or the needs of the listener; difficulties following rules for conversation and storytelling; and difficulties understanding what is not explicitly stated (e.g., making inferences) and nonliteral or ambiguous meanings of language. Although the intent behind the PLOS is to evaluate pragmatic impairment, the authors acknowledge that the descriptor items address all aspects of language. This could be justified by claiming that, unlike traditional standardized tests that assess students’ knowledge of language or use of language in structured, decontextualized settings, the PLOS evaluates how students actually use their language in social and academic situations. There is a potential problem, however, with including both structural and pragmatic language items in the scale. The fact that the PLOS includes both speech and structural language items, in addition to the pragmatic items, confounds its use as a pure assessment of pragmatics and would increase its correlations with traditional language tests. The PLOS may overestimate the pragmatic skills of students who have social pragmatic communication disorders but good speech and structural language skills.
The tests used to establish validity of the PLOS all focus on assessment of structural aspects of language. It would be interesting to know how the PLOS would correlate with standardized tests that assessed elements of pragmatics and that were available when the PLOS was being developed, for example, the Children’s Communication Checklist-2 (Bishop, 2006), the Test of Language Competence (Wiig & Secord, 1989), the Test of Pragmatic Language (Phelps-Terasaki & Phelps-Gunn, 1992), or the Comprehensive Assessment of Spoken Language (CASL; Carrow-Woolfolk, 1999).
Summary
The PLOS is a standardized, normed-referenced rating scale designed to assess the oral communication behaviors of students in a classroom setting. It could be a useful adjunct to traditional standardized language assessment, particularly in cases where there are concerns about students’ language functioning in the classroom, yet their performance on tests assessing vocabulary, morphology, and syntax are not significantly low. Although the PLOS assesses language use in classroom contexts, it may not adequately assess the range of pragmatic difficulties exhibited by students with social communication disorders.