Abstract
Test Description
General Description
The Delis Rating of Executive Function (D-REF), authored by Dean Delis and published by Pearson, is a set of rating scales designed to assess executive functions and their constituent sub-processes in children and adolescents between the ages of 5 and 18. More specifically, the D-REF is a supplemental assessment of children and adolescents demonstrating behavioral or cognitive difficulties often associated with attention-deficit/hyperactivity disorder (ADHD), autism spectrum disorder (ASD), traumatic brain injury (TBI), learning disorders/disability, and other developmental, psychiatric, neurological, or medical conditions known to affect executive functions.
The D-REF is grounded in the broad conceptualization that executive functions are the product of many higher order cognitive processes providing the basis for goal attainment. In the description underlying the structure of the D-REF, Delis emphasizes the current literature on executive function regarding the roles of self-regulation and management, as it affects behavioral functioning. As successful goal attainment is related to the integration of emotional, behavioral, and cognitive functioning, the D-REF assesses these constructs within four broad areas of global executive function: Attention/Working Memory (AWM), Activity Level/Impulse Control (AIC), Compliance/Anger Management (CAM), and Abstract Thinking/Problem Solving (APS). The D-REF is intended to be a first-line assessment from which to direct additional investigation with other standardized executive functions assessments.
The D-REF has three forms, Parent/Caregiver, Teacher, and Self-Report. The Self-Report is available for ages 11 to 18 years. The Parent/Caregiver and Teacher forms are available for children ranging from 5 to 18 years of age. Forms are parallel, with 36 items each. Administration is standardized on English-speaking raters only and is intended to be administered online, with in-person explanation of the purpose of the assessment to ensure valid completion. Without Internet access, however, forms may be provided for paper–pencil administration. All forms may be completed in approximately 10 min.
Items on the D-REF have been written at a fourth-grade reading level, and ask the rater to indicate the frequency of the listed behavior within the most recent 6-month period. Responses are based on a time-frequency scale. Respondents are directed to indicate whether a listed behavior happens: S/N, seldom/never; M, at least every 1 to 3 months; W, at least once per week; or D, daily.
Results and interpretive information may be obtained either from the manual or from the online/program options. The manual provides guidelines for interpretation of obtained T-scores for broad areas (Total Composite, Core Indices, and Clinical Indices) followed by scales and items based on the child’s age and gender. As the D-REF’s purposes are to assess the frequency of problem behavior(s) and to identify specific behaviors most distressing to the parent, teacher, and/or child, interpretation should lead to intervention planning.
Specific Description
Core index scores
All forms of the D-REF contain four Core Index scores that may be used to assess aspects of executive functioning. The Core Index T-scores have a mean of 50 and a SD of 10.
Behavioral Functioning (BF)
This index measures the child’s ability to regulate his or her behavior to meet the demands of the environment.
Emotional Functioning (EMF)
This index assesses the child’s ability to regulate emotions to meet the demands of the environment.
Executive Functioning (EXF)
This index addresses the higher level cognitive ability to effectively adapt and function within the environment.
Total Composite (TC)
Items on this index measure the ability to plan, execute, and regulate cognitive, emotional, and behavioral functions.
Clinical index scores
These indices are based on diagnostic criteria for disorders found in childhood and are more specific than items within the Core Index. Three Clinical Indices are common to all forms. In addition to Clinical Indices, the APS index is found on the parent and teacher forms.
AWM
This index measures inattention, deficient multi-tasking ability, forgetfulness, poor working memory, and disorganization.
AIC
Addresses the child’s symptoms of hyperactivity, impulsivity, and deficient self-monitoring.
CAM
Items on this index assess symptoms of mood lability, sensitivity to criticism, frustration tolerance, and rule-breaking behavior.
APS
This area examines concrete thinking, cognitive inflexibility, and poor decision making/problem solving.
Scoring System
Scoring software
To score the D-REF from a paper–pencil administration, the practitioner transfers answered items into the scoring software. The scoring and reporting of data gathered from the D-REF is completely automated. The scoring software includes procedures for handling missing item responses.
Interpretation
There are several ways to examine and interpret results from the D-REF. The D-REF utilizes T-scores for the Total Composite, Core Indices, and Clinical Indices. In all three score index types, T-scores below 55 are considered to be Within Normal Limits for executive functioning behaviors. Scores from 55 to 59 are considered Borderline Elevated. Children with mild levels of executive functioning problems fall within the 60 to 70 T-score range. Scores of 70 or greater are considered severely atypical and represent significant concerns in executive functioning behaviors. As stated previously, obtained scores can be compared with general population norms or sex-adjusted norms.
Steps for interpreting the profile of scores are provided in the manual. The Total Composite T-score should be examined first to provide a general overview of the child’s overall executive functioning. Next, the Core Index T-scores should be analyzed. The focus of interpretation lies in the Core Indices as these areas provide a more detailed, specific picture of the child’s problems and deficits. Finally, the Clinical Index scores can be examined to identify patterns of problem areas related to specific deficits in executive functioning or clinical subtypes. In addition, the top five stressors, which are specific items identified by the rater as causing the most distress for the child or the informant, can be examined. Although the top five stressors may not be the most frequently occurring behaviors, they can cause significant impairment in the child’s life and should be targeted when creating an intervention.
Technical Adequacy
Test Construction and Item Analysis
The D-REF was developed as a screening measure to identify problematic behaviors as seen by parents, teachers, and students. Items were constructed based on behaviors the author felt represented executive functioning deficits. This measure is not intended to replace a comprehensive assessment of executive functioning, as the D-REF should be used in conjunction with other measures to provide a comprehensive profile. Following exploratory factor analysis from the pilot study results, the BF, EMF, and EXF indices were defined and conceptualized. Each form contains an identical 36-item, 4-point Likert-type scale for behavioral ratings and a 36-item list of the behaviors upon which the respondent is asked to indicate five most problematic behaviors. The Clinical Indices have equal representation with 6 items; however, the Core Indices have unequal item representation, with EMF under-represented with only 8 items and EXF over-represented with 17 items.
Normative Sample
The normative sample (n = 1,062) for the D-REF was based on the 2010 U.S. Census Bureau statistics to ensure that the sample was nationally representative on demographic characteristics such as age, geographic region, race/ethnicity, and parent education level. Gender was evenly distributed across all demographic characteristics.
Reliability
Internal consistency
Internal consistency was calculated using Cronbach’s alpha. Alpha coefficients range from .64 to .99 across the forms. Cronbach’s alpha for Core Index scores were strong on the Parent forms, ranging from .86 to .97, strong on Teacher forms (.80-.99), and moderate to strong on the Self-Report forms (.77-.91). The Cronbach’s alpha for the Total Composite score was very strong across all three forms, ranging from .91 to .99. For the Clinical Index scores, alpha coefficients were in the moderate to strong range on the Parent and Teacher forms (.76-.95), while coefficients on the Self-Report form were low to moderate (.64-.87).
Test–retest reliability
Test–retest reliability was measured by having 50 parents, 54 teachers, and 47 individuals complete the appropriate rating form on two occasions, with the interval ranging from 7 to 56 days. Across the forms, corrected Pearson’s r correlations range from .62 to .90, indicating a moderate to strong correlation.
Standard error of measurement (SEM)
SEM was calculated for all three forms and all index scores to provide information regarding measurement error for each score. The SEM in the indices is consistent with other measures of this type across forms, with a range of ±1 to ±6. The widest SEM range is found on the Clinical Index scores due to an admitted low sample size in the normative group.
Validity
Content validity
Content validity was developed based on a comprehensive literature review on the aspects of executive functioning the D-REF seeks to measure, as well as the clinical experience of the author. When creating the D-REF, Delis focused on linking test items to specific executive functions and measuring the frequency of certain behaviors. Data collected during the pilot and standardization phases of the test were utilized to determine final item composition.
Construct validity
Intercorrelations of index scores were conducted to substantiate convergent and discriminate validity with both the normative sample and a clinical sample. The correlations between Core Index (.55-.93) and Clinical Index scores (.52-.96) indicated strong relationships for the normative sample across the three forms. Correlations for the clinical sample revealed moderate to strong relationships on both the Core Index (.39-.78) and Clinical Index (.49-.96) scores, with the lowest correlations on the Teacher-Rating Forms. Cross-form comparisons resulted in consistently lower correlations than within-form comparisons. The normative sample correlations for the Core and Clinical Indices range from weak to strong across all forms (.11-.55). The clinical sample correlations for the Core and Clinical Indices range from weak to strong across all forms (.10-.60).
Concurrent validity
D-REF scores were compared with those obtained on the Behavior Rating Inventory of Executive Functioning (BRIEF; Gioia, Isquith, Guy, & Kenworth, 2000) to provide evidence for concurrent validity. For the Parent form, correlations between the Core and Clinical Indices and the Indices on the BRIEF were moderate to strong (.33-.80). Correlations between Teacher ratings on the Core and Clinical Indices and BRIEF Indices were weak to strong (.29-.80). The correlation between Self-Report Core and Clinical Indices and BRIEF Indices were also weak to strong (.13-.78). Indices measuring similar constructs correlated in the moderate to high range across forms.
In addition, the author conducted analysis of performance across clinical groups, including those meeting diagnostic criteria in the Diagnostic and Statistical Manual of Mental Disorders (4th ed., text rev.; DSM-IV-TR; American Psychiatric Association, 2000) as having ADHD, Autism, or Asperger’s Disorder to assess concurrent validity and clinical usefulness of the D-REF. Across all clinical groups, large effect sizes were found for clinical as compared with non-clinical groups, suggesting the utility of the D-REF in differential diagnosis of children with deficits in executive functions relative to their non-clinical peers.
Commentary and Recommendations
The D-REF is an appropriate measure to utilize in conjunction with other measures of executive functioning. The length of the forms makes the separate forms convenient and user-friendly. The identified problem areas provide a basis for developing individualized interventions. The reliability across forms in the Core and Clinical Indices were adequate to good, with the Self-Report form having the lowest reliability coefficients. Online administration and scoring options allow for flexibility in completing forms, and the interpretation of obtained T-scores is consistent with other scales of this type.
The D-REF’s lack of a theoretical underpinning is a weakness of the scale overall. Even though the scale is based on current literature of executive functions and their implications, having a theoretical model may have improved the statistical properties of this measure. The detail given describing the norming process is inadequate in that the reader is unable to discern whether all forms were completed for each case, the total number of cases for all respondents, or any combination of this information. While the description of executive function (and its history), scale development, and the measure itself are thorough, the discussion of executive function is excessive at several points in the manual. Improving the norming and reliability of the Self-Report form would be ideal as some of the obtained scores lack sufficient reliability. Practitioners should be cautious in their use of this, or any, self-report scale alone as a measure of executive function.
To improve applicability across cultures, creation and standardization of, for example, Spanish forms would provide greater utility for the D-REF. A specific normative sample for Spanish-speaking populations would be optimal, as the D-REF in its current state does not address culturally/linguistically diverse populations. Practitioners need tools for assessing executive function which are appropriate for use with culturally/linguistically diverse populations.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.