Abstract
Test Description
The Comprehensive Executive Function Inventory (CEFI; Naglieri & Goldstein, 2013), published by Multi-Health Systems Inc. (MHS), is a new executive function (EF) rating scale for children and youth ages 5 to 18 years. The CEFI strives to accurately assess EF abilities based on self, parent, and teacher reports, and provides specific and individualized intervention recommendations.
Naglieri and Goldstein (2013) define executive functions as “a set of cognitive processes that control and manage other cognitive processes” (p. 5). The CEFI assesses behaviors that are associated with EF (e.g., inhibitory control, working memory), and determines an individual’s profile of EF strengths and weaknesses. Test items were constructed based on the premise that EFs are involved in higher order cognition, as well as the regulation and control of spontaneous actions towards goal-directed behavior (for review, see Chapter 2 of the manual). Given that evidence suggests that many symptoms of psychopathology stem from core deficits in EF (Royall et al., 2002), a strong understanding of EF strengths and weaknesses, such as those captured by the CEFI, may be useful in understanding if and how a person manifests EF skills in multiple settings.
The CEFI provides an overall full scale executive functioning standard score and individual subtest standard scores on nine components of EF, including Attention, Emotion Regulation, Flexibility, Inhibitory Control, Initiation, Organization, Planning, Self-Monitoring, and Working Memory. In addition, each rater’s responses are analyzed on a Consistency Index, Negative and Positive Impression Scales, and number of omitted items. Such analyses serve to identify biased (overly positive or negative) and/or inconsistent responses to strengthen reliability and validity of the ratings.
Administration and Scoring
The CEFI is a Level B measure and may be used or administered by individuals who have undergone formal training in standardized psychological or educational testing (e.g., psychology, education, medicine, or social work). It may be administered individually or in a group setting through paper-and-pencil or online format, both of which are completed in approximately 15 minutes. There are three different forms: Parent (for children ages 5-11 and 12-18, respectively), Teacher (ages 5-11 and 12-18, respectively), and Self-Report (age 12-18).
Unique to the CEFI, there is a 6-point Likert-type scale (Never, Rarely, Sometimes, Often, Very Often, Always) for each of the 100 questions for each form. The raters are instructed to respond to each item based on the behaviors observed during the past 4 weeks. The use of multiple forms allows the administrators to attain a comprehensive understanding of the child’s EF skills from a number of sources (e.g., parent and teacher). The authors further suggest providing the CEFI forms to several teachers to understand performance across multiple classroom settings (Naglieri & Goldstein, 2013). This information, along with the results from other assessments, could be utilized to create a thorough intervention plan and monitor treatment progress.
Standard scores, percentile ranks, and classification equivalency (e.g., Average or High Average) information are provided in tabular format in Chapter 4 of the manual. The CEFI can be scored by hand, using scoring software, or on the MHS Online Assessment Center, with the authors recommending the latter two options to increase reliability (Naglieri & Goldstein, 2013). The scoring software is easily accessible and portable, although similar to other MHS measures (e.g., Conners-3), the USB drive must be inserted into the computer while scoring. Data can be entered twice for each respondent to ensure data-entry accuracy and unfinished data entry can be saved and re-accessed when required. Chapter 4 of the manual provides a detailed step-by-step and illustrated guide to hand scoring. Overall, the scoring system seems to be well-documented, clearly described, and easily accessible.
Test Materials and Stimuli
The CEFI is user friendly for administrators, adolescent, parent, and teacher responders. Respondent instructions are labeled clearly and coherently and written at approximately a seventh grade reading level, with important information bolded. To ensure that items are rated appropriately and accurately, abbreviated instructions are repeated on the second page of the response form. Item wording is suitable for a wide range of responder comprehension levels (e.g., children to mature adults), with test items for all forms written at approximately a fifth grade level. Items are identical across the three response forms, facilitating consistency in analyses and interpretation.
Several resources assist in results interpretation. First, Chapter 4 of the manual provides guidelines for interpretation of raw and standard scores, confidence intervals, and comparison of scoring patterns. Two case studies are provided to further aid interpretation. Second, scoring software reports define the cognitive abilities and behavior each subscale measures. In addition, these reports provide intervention strategies for each individual.
Test Manual
Overall, the technical manual of the CEFI is well organized. Despite the extensive amount of tables included in the manual, it is simple to navigate to desired information. The table of contents specifically for tables and figures provides extra support in this process. The appendices that contain the tables used for scoring also clearly indicate which tables are used for which form and age group. In addition, the authors provide a thorough explanation of the theoretical framework, developmental processes, normative samples, and reliability and validity of the CEFI. Together, the technical manual of the CEFI is a comprehensive resource that arms the administrator with valuable information.
Technical Adequacy
Development and Standardization
According to the CEFI manual, collection of the standardization data for the CEFI was completed between January and December, 2011. Over 5000 CEFI protocols were completed, encompassing both online and paper-and-pencil versions of the parent, teacher, and self-report scales. Data were collected from all 50 U.S. States and included consideration of age, gender, geographical region, race/ethnicity, parental education level, and eligibility for special education services. Overall, the CEFI standardization sample is consistent with the 2009 U.S. census. No separate Canadian norms were created for this measure.
Reliability
Internal Consistency
The interrelatedness of the measure’s items, or internal consistency, was calculated using Cronbach’s α. Full scale scores for the normative sample (5-11 years and 12-18 years, respectively) as well as the clinical/educational sample for parent, teacher, and self-report scales were strong, with alpha values ranging from .97 to .99. At the subtest level, parent and teacher alpha values were strong, with the majority of scores greater than .85 and only two parent scales (Flexibility and Self-Monitoring for the clinical/educational sample) scoring below .80. The self-report scale subtests (age 12-18 only) were slightly lower, with alpha values ranging from .70 to .86.
Test-Retest Reliability
To examine test-retest reliability, the CEFI was administered to the same individuals rating the same child at two separate time points ranging from 7 to 30 days apart. Results of these correlations reveal excellent test-retest reliability. Across all forms, corrected r for the full scale scores ranged from .77 to .91 (all p < .001) and from .74 to .91 (all p < .001) on the subscales. Of particular note, parent and teacher test-retest reliability scores were consistently .80 or higher, with lower reliabilities found only on the self-report measure (corrected r = .74 to .86).
Interrater Reliability
The degree of agreement between two raters (two parents or teachers of the same child) was also provided. On the full scale score, agreement between parent raters was strong (corrected r = .88) but was slightly lower between teacher raters (corrected r = .68). At the subtest level, similar to the full scale reliability, strong parent rater correlations (range of corrected r = .73 to .86) and adequate teacher ratings (range of corrected r = .54 to .68) were found.
Validity
Content Validity
A thorough review of current research and theoretical literature, along with the test authors’ extensive clinical and research knowledge contributed to the creation of the CEFI items. Together, nine EF-related behaviors were identified, an operational definition was formed, and test items were created to map onto each of these areas, resulting in a measure with strong content validity.
Construct Validity
Analyses to determine the construct validity of the CEFI included Exploratory Factor Analysis (both item- and scale-level) and congruence analysis (to determine consistency between demographic groups, including race/ethnicity, across ages, and Canada–U.S. populations). Full details of each of these analyses are included in the technical manual. Overall, the CEFI produced a unidimensional factor structure consistent with the nine EF-related behaviors proposed by the authors and was consistent across gender, age group, race/ethnicity, and clinical/educational status.
Of particular interest for the authors of this test review, analyses comparing English-speaking Canadian and U.S. populations found no CEFI score differences between the samples; effect sizes between Canadian and U.S. samples were small (d range = −.04 to .16) and country of residence was not found to be significant in any analyses, indicating that this measure is also applicable for use with Canadian youth.
Criterion-Related Validity
To examine the criterion-related validity of the CEFI, clinical populations who have been previously found to have EF deficits were examined. Specifically, ratings of children and youth with Attention-Deficit/Hyperactivity Disorder (ADHD), Autism Spectrum Disorder (ASD), or Mood Disorders were compared to the general population. Statistically significant moderate to large effect sizes (Cohen’s d = −.62 to −1.59, all p <.001) were found across all three forms of the CEFI for these populations.
Relationship with other constructs
The relationship between the CEFI and a number of other measures was analyzed. Specifically, the CEFI was compared to another commonly used rating scale for EF, the Behavior Rating Inventory of Executive Functioning (BRIEF; Gioia, Isquith, Guy, & Kenworthy, 2000; Guy, Isquith, & Gioia, 2004), as well as other measures of cognitive and academic functioning, including the Cognitive Assessment System (CAS; Naglieri & Das, 1997), Wechsler Intelligence Scale for Children—Fourth Edition (WISC-IV; Wechsler, 2003), and Woodcock-Johnson III Tests of Achievement (WJ-III Ach; Woodcock, McGrew, & Mather, 2001). These comparisons were made with clinical samples only, with one sample consisting solely of children with ADHD and the other a mixed sample, including those with anxiety or mood disorder, ASD, or Learning Disorder.
When comparing the CEFI and the BRIEF, it was found that the full scale score on the CEFI was strongly correlated with the BRIEF Global Executive Composite for the parent (ADHD sample: corrected r = .85; Mixed sample: corrected r = .78), teacher (ADHD: corrected r = .64; Mixed: corrected r = .66), and self-report (ADHD: corrected r = .68; Mixed: corrected r = .63) scales (all p < .01), further supporting criterion-related validity. Additionally, the test authors highlight one important difference between the CEFI and the BRIEF: the CEFI was normed on a national sample while the BRIEF was not nationally normed, which may account for slight disparities between the two measures.
Moderate correlations between the CEFI and WISC-IV full scale scores (corrected r = .39, p < .05) and subscale scores (corrected r range from .27 to .44) indicate that the CEFI measures a related but distinct construct than the WISC-IV. The CAS provides a more targeted assessment of cognitive abilities related to EF and, accordingly, the CEFI was found to have a significant moderate correlation with this measure (corrected r = .45, p < .01) and stronger correlations with subtests tapping EF ability. A significant moderate correlation with the WJ-III Ach was also found (corrected r = .51, p < .01).
Commentary and Recommendations
Strengths
The CEFI’s features strongly suggest that it offers useful information on children’s EF for both researchers and clinicians. One noteworthy aspect of the CEFI lies in the authors’ intensive effort put forth in constructing the measure itself. The extensive, up-to-date theoretical background research on EF supports the rationale for the test item choices. The well-formed questions allow concise representation of the behavioral trend seen in each child and may accurately capture the child’s EF abilities. As well, construction of the CEFI included a thorough consideration of the representativeness of the normative sample. The CEFI utilized a sample representative of the 2009 U.S. national census whereas other similar EF measures were normed based on regional samples (e.g., BRIEF; Gioia et al., 2000) or on the 2000 U.S. census (e.g., Barkley Deficits in Executive Functions Scale—Children and Adolescents; Barkley, 2012). Also, the authors considered the extension of this measure to a Canadian population, and found similarity between the U.S. and Canadian samples, thus supporting the usability of the measure in Canada. The reliability and validity data for the CEFI also indicate that this measure is technically sound.
The usability of the CEFI is also a strength. Response form instructions are clearly and concisely labeled, which may contribute to greater levels of response accuracy. Additionally, the response form demonstrates consistency across all three versions, facilitating direct comparisons between self, parent, and teacher reports. The computer scoring software is easy to navigate and work through successfully.
Of particular importance and interest for practitioners, the scoring software generates an excellent individualized report, complete with clear results and specific, relevant intervention suggestions. The report provides recommendations for addressing an individual’s limitations within a particular EF domain. The interpretive report also provides clear and applied examples of how problematic behaviors relate to EF, adding to the face validity of the measure. This information may be especially useful when liaising with parents or teachers, and when explaining test results and recommendations.
Limitations
Upon reviewing this measure, few limitations were noted with the measure itself. However, there were some issues with the scoring software that caused minor frustration. Although MHS was very responsive and helpful, the USB scoring key required replacing twice due to malfunctioning software and the third key only worked with a separately downloaded patch. As well, the scoring software appears to work only on PC-based computers and not on the Mac platform, which may be problematic for some users. Although these issues are not directly related to the construction and validity of the CEFI, they do contribute to the overall usability of the measure.
In conclusion, the CEFI is a well-designed, robust measure for assessing child and youth EF abilities. It was constructed based on a sound theoretical framework, utilized representative samples, and provided strong reliability and validity. The clear instructions on the record forms, simple scoring methods, and well-structured score reports makes the CEFI very user-friendly. Together, the CEFI appears to provide a strong and useful measure of EF abilities for use in both clinic and research.
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.