Abstract
Executive functions (EFs) are higher level self-regulatory constructs involved in complex and novel activities (Barkley, 2012; Brown, 2013). EFs include metacognitive “cold” aspects of regulation such as working memory, planning, and attentional control as well as behavioral and emotional “hot” aspects of regulation such as emotional and inhibitory control (Gioia, Isquith, Guy, & Kenworthy, 2000). EFs serve to effectively guide one’s actions toward future self-serving goals (Barkley, 2012). Research has substantiated the negative impact of EF deficits (EFDs) on occupational performance among people with attention deficit hyperactivity disorder (ADHD; Biederman et al., 2006; Chiang & Gau, 2014).
ADHD is a prevalent lifelong disorder involving impaired EFs that has a negative impact on participation and quality of life (Antshel, Hier, & Barkley, 2013; de Schipper et al., 2015; Willcutt, 2012). For people with chronic health conditions such as ADHD, successful participation requires self-awareness of the disorder and its implications as well as self-awareness of strengths and resources. The client’s ability to identify strengths and challenges is a powerful agent in collaborative goalsetting and in transferring strategies to daily life (Toglia, 2011). Therefore, for children with ADHD, a collaborative child- and family-centered inquiry into the child’s EF profile would improve ADHD interventions.
Several child self-rating tools of cognitive functioning have been developed, such as the Subjective Awareness of Neuropsychological Deficits Questionnaire for Children, used with children ages 9–16 yr, which was validated with children with epilepsy (Hufford & Fastenau, 2005), and the Children’s Organizational Skills Scale, for children ages 8–13 yr, which addresses metacognitive EFs of planning and organization (Abikoff & Gallagher, 2009). Yet, the literature has been inconclusive regarding young children’s ability to identify challenges and set goals.
Studies of the self-perception of children with ADHD have documented strong evidence for overestimation of competence, termed positive illusory bias (PIB), in academic, social (Whitley, Heath, & Finn, 2008), and cognitive (Steward, Tan, Delgaty, Gonzales, & Bunner, 2017) domains. In people with ADHD, PIB has been attributed to psychological, protective (Owens, Goldfine, Evangelista, Hoza, & Kaiser, 2007), and neurocognitive (McQuade et al., 2011) mechanisms. Children with ADHD were, however, able to rate their occupational competence in everyday activities using a pictorial assessment (Ayuso & Kramer, 2009). In addition, children with various disabilities were able to use pictorial assessments to rate their own abilities and use this information for goal setting (Cermak & Bissell, 2014; Missiuna, Pollock, Law, Walter, & Cavey, 2006).
Therefore, despite the known PIB, it was possible that, given appropriate accommodations, children with ADHD would be able to identify at least some of their EF challenges and participate in goal setting. This assumption was further supported by findings demonstrating 8- to 10-yr-old children’s awareness of their self-regulation problems using the self-rated pictorial assessment of Self-Regulatory Function (Rizzo, Steinhausen, & Drechsler, 2010). However, these findings were based on a small heterogeneous sample of 21 children (only 6 of whom had ADHD) with a variety of behavioral, developmental, and academic difficulties.
The Pictorial Interview of Children’s Metacognition and Executive Functions (PIC–ME; Traub Bar-Ilan et al., 2014) was designed to evaluate the self-perception of EFDs among young children (ages 5–10 yr) with ADHD in occupational contexts. The overarching aim of this study was to develop an assessment designed to enable children to access self-knowledge, using nonthreatening pictorial portrayals of common EF challenges that children encounter in daily life. The goals were to establish the PIC–ME’s content validity and to examine its initial validity and reliability.
Method
Development of the PIC–ME
The PIC–ME was developed in tandem with the Cognitive–Functional (Cog–Fun) treatment of children with ADHD (Maeir et al., 2014), which aims to promote the acquisition of executive strategies in occupational contexts. The Cog–Fun prescribes a child- and family-centered approach that (1) includes nonthreatening communication about challenges, (2) uses the child’s perspective as the starting point, and (3) includes strengths in the communication process. Thus, the PIC–ME was developed to measure the child’s self-perception of EFDs and strengths and to engage the child in collaborative goal setting. Item development was based on Brown’s (2009) comprehensive clinical model, including six EF clusters that characterize the challenges of people with ADHD in real-world contexts:
Activation (organizing tasks and materials, estimating the time required for task completion, prioritizing tasks, and getting started on tasks)
Focus (focusing on the target stimuli, sustaining and maintaining focus on a given task, ignoring distractions intrinsic or extrinsic to the person, and shifting focus to tasks)
Effort (regulating alertness, sustaining effort, and processing speed)
Emotion (managing frustration and modulating emotions)
Memory (using working memory and accessing recall)
Action (monitoring and regulating action).
Eight occupational therapists created a pool of items reflecting EF challenges in daily life (38 items) and common strengths (9 items) among children with ADHD (Climie & Mastoras, 2015; Ek et al., 2007). Each item was supplemented with several illustrated scripts that depict a cartoon character (Daniel, in both the boy and girl versions) experiencing EF challenges or strengths in different contexts (home, school, and community; Figure 1). For example, the item “Daniel can’t find his things” was elaborated on with “he can’t find his belongings in his room” and “he can’t find his notebook in his schoolbag.”
Pictorial Interview of Children’s Metacognition and Executive Functions: Example item with supplemental scripts.
Expert Review and Content Validation
The 47 items were reviewed by 11 expert clinicians and educators (kindergarten–Grade 5). These experts were 10 occupational therapists and 1 neuropsychologist with ≥10 yr of clinical experience (mean [M] = 21.36, standard deviation [SD] = 8.27). Four of the experts were academic researchers who have published in international journals on child executive functioning, 4 were clinical instructors in child development, and 3 were directors of pediatric occupational therapy in national education services. They were asked to classify the 38 EF items into one of Brown’s six EF clusters (enumerated earlier) and to rate the degree to which each item represented the chosen EF cluster; to rate the degree to which each item represented typical child activity; to rate the clarity of each item’s language to children; and, finally, to add EF challenges or strengths that were not included in the item pool.
The reviewers rated the items on a 4-point ordinal scale for their relevance to the chosen EF scale and to typical child activity (1 = not relevant, 2 = somewhat relevant, 3 = quite relevant, 4 = highly relevant) and for appropriateness and clarity of the wording for children (1 = not appropriate, 2 = somewhat appropriate, 3 = quite appropriate, 4 = highly appropriate). This review led to classification of the items into EF scales on the basis of ≥70% agreement among raters, removal of 4 items with low agreement, improvement in the wording of 23 items, and adding 1 strengths item.
The content validity index of EF items (the degree to which the item represents EF scale) ranged from 0.72 to 1.00; for the entire scale, it was 90.51. Content validation was conducted with 12 children (6 with ADHD, 6 typically developing; ages 5.1–9.1 yr). Children were asked to describe the pictures, leading to further changes to the items (11 pictures were removed, and 14 were modified). The final version consisted of a booklet with 34 items representing six EF challenges and 10 items representing strengths (Figure 2).
Pictorial Interview of Children’s Metacognition and Executive Functions: Scales and items.
The goals of the next stage of this study were to examine the PIC–ME’s initial reliability and validity. Specifically, the research questions were as follows:
What is the internal reliability of the items on each scale and the total score?
What is the correlation between parent PIC–ME scores and an established ecological EF measure?
Is there a difference between groups with and without ADHD on parent and child PIC–ME scores?
What is the rate of parent–child dyad agreement within and between groups?
How many items do children with ADHD identify for goal setting?
Participants
The study group included 100 children with ADHD (64 boys, 36 girls), ages 5–10 yr, referred for Cog–Fun from January 2013 to December 2013 from two ADHD clinics in central Israel, and 44 typically developing children (22 boys, 22 girls) from a convenience sample from the same geographical region, matched on age and gender. The inclusion criterion for the study group was a medical diagnosis of ADHD according to Diagnostic and Statistical Manual of Mental Disorders (4th ed., text rev.; American Psychiatric Association, 2000) criteria, as assessed by a certified pediatric neurologist–psychiatrist. A Conners’ Parent Rating Scales–Revised (CPRS–R; Conners, Sitarenios, Parker, & Epstein, 1998) score ≥65 was used to provide a baseline level of symptom severity. Exclusion criteria were serious psychiatric or neurological disorder or special education setting. For the typically developing group, ADHD was excluded on the basis of a CPRS–R score ≥65. Ethical approval was obtained from the Hebrew University of Jerusalem institutional review board; parents signed informed consent, and children provided assent. Children were individually administered the PIC–ME by six trained occupational therapists, and parents completed the PIC–ME and Behavior Rating Inventory of Executive Function (BRIEF; Gioia et al., 2000) in a separate room. All assessments were conducted in Hebrew using valid translations.
Instruments
Behavior Rating Inventory of Executive Function.
The BRIEF is an 86-item ecological rating scale for children ages 5–18 yr designed to reflect executive functioning in everyday situations. It consists of eight scales, two indices, and a global executive composite (GEC). Raw scale scores are converted into t scores, with scores of ≥65 indicating clinical impairment. Internal consistency, test–retest reliability, and discriminant validity have been established for children with ADHD.
PIC–ME.
The PIC–ME is a 44-item ecological pictorial interview consisting of 34 items representing EF challenges grouped into six scales (Effort, Emotion, Working Memory, Activation, Focus, Action) and 10 items representing strengths. Administration includes several stages depending on the child’s response. The child is asked whether the situation depicted in each item (pictorial script) happens to him or her as well (yes = 1, no = 0). A positive answer is followed by more in-depth questioning regarding frequency (sometimes, often), context (home, school, or community), and whether it bothers the child and whether the child wants to do something about it (to change it).
The number of yes answers are summed for all EF challenges and for each EF scale, divided by the number of items on the scale, and multiplied by 100 to obtain a score ranging from 0 to 100. The Strengths scale is scored in the same manner. A Goal scale is calculated by summing the number of EF items that the child wants to change, ranging from 0 to 34. The other responses regarding context (whether EF challenges occur at home, at school, or in the community) are not scored but provide qualitative information. The PIC–ME includes a corresponding parent questionnaire, in which the parent is asked to rate the child on the same items (1 = similar to my child, 0 = not similar to my child) with scale and total scores calculated as described for the child ratings. The interview lasts between 30 and 45 min, and administration of the PIC–ME requires expertise in child-centered practice.
Data Analysis
Data were analyzed with IBM SPSS Statistics (Version 19; IBM Corp., Armonk, NY). Internal consistency for the PIC–ME scales and for the entire measure was examined using Cronbach’s α analysis. PIC–ME variables did not meet criteria for normal distribution (Kolmogorov–Smirnov p < .05). Therefore, data were analyzed with nonparametric statistics. Spearman rank order correlation analysis was used to examine the relationship between variables, and the Mann–Whitney U test was used to compare scores between groups.
Parent–child agreement scores were calculated for each dyad by counting the number of items on which each dyad agreed. In addition, overestimation scores (representing the number of items that parent scored as an EF problem and the child did not, meaning the child overestimated his or her functioning) and underestimation scores (representing the number of items the child scored as an EF problem and the parent did not, meaning the child underestimated his or her functioning) were calculated. Finally, frequencies were calculated for the Goal scale in the ADHD group. There were 3 missing values in the parent CPRS data. Missing values were replaced with the mode value of the typically developing group for each item.
Results
Analyses of group characteristics demonstrated no significant differences between the ADHD (n = 100) and typically developing (n = 44) groups on age (Ms = 7.96 and 7.73, SDs = 1.06 and 1.33, respectively), t(142) = −1.11, p = .266, 95% confidence interval (CI) for mean difference [−0.645, 0.179], or gender (64% and 50% boys, respectively), χ2(1) = 2.49, p = .115. The mean CPRS ADHD index scores were significantly higher (more symptoms) in the ADHD group than in the typically developing group (Ms = 70.75 and 47.26, SDs = 10.41 and 5.77, respectively), t(142) = −13.59, p = .000, 95% CI for mean difference [−26.89, −20.06], and all BRIEF scale scores were significantly higher (more impaired EF) in the ADHD group than in the typically developing group (see Table 1).
Comparison of BRIEF Scores Between Children With ADHD and Typically Developing Children
Note. ADHD = attention deficit hyperactivity disorder; BRIEF = Behavior Rating Inventory of Executive Function; CI = confidence interval; M = mean; SD = standard deviation.
p < .001.
Internal Consistency Reliability and Correlation With the Executive Function Measure
Cronbach’s α was high for the total PIC–ME EF score for both child and parent ratings (α = .953 and .914, respectively). Values of Cronbach’s α for the EF scales were in the acceptable range for most of the parent ratings (α = .767–.871), except for the Effort scale (α = .641), yet they were lower and questionable for most of the child ratings (α = .541–.775). A high correlation was found between the total EF PIC–ME parent score and the BRIEF GEC score (r s = .73, p < .001).
Comparison Between Groups
Results of the Mann–Whitney tests revealed significantly higher parent rating scores (more deficits) in the ADHD group than in the typically developing group on all PIC–ME scales except the Strengths scale. An opposite trend was found for child ratings; no significant between-groups differences were found on all PIC–ME scales except the Strengths scale, on which children with ADHD rated their strengths significantly lower than did typically developing children (Table 2). The children in both groups pointed to the relevant contexts (home, school, community) as areas in which executive dysfunctions occur. A similar pattern emerged in both groups, whereby children more frequently pointed out deficits at home and in school than in the community. The EF challenges most frequently allocated to the home context were “doesn’t know what to do when he is bored” (97%) and “needs to be told again and again to do something” (90%). Those most frequently allocated to the school context were “forgets things he needs to bring home from school” (100%), “doesn’t finish things because he runs out of time” (69%), and “does silly things in groups” (67%). In the community context, EF challenges were most frequently allocated to “difficulty calming down” (69%), “waiting my turn” (63%), and “doing dangerous things” (57%).
Comparison of Parent and Child PIC–ME Scores in ADHD and Typically Developing Groups
Note. ADHD = attention deficit hyperactivity disorder; IQR = interquartile range; Mdn = median; PIC–ME = Pictorial Interview of Children’s Metacognition and Exercise Functions; WM = Working Memory.
Note that many challenges were allocated to multiple contexts, such as “forgets what he wants to say if he has to wait,” “distracted by things around him,” and “people tell him to stop daydreaming.” Regarding age and gender, correlation analyses revealed a low correlation between age and child ratings on the PIC–ME Action scale (r s = .198, p = .048) in the ADHD group. All other correlations of PIC–ME scales with age were nonsignificant (r s = −.055 to .164). In addition, no significant differences were found between boys and girls on all PIC–ME scales (p > .1).
Parent–Child Agreement
Similar rates of agreement in parent–child dyads were found for both groups on the 34 PIC–ME EF items. The median dyad agreement scores, representing the number of EF challenges on which children and parents agreed, were 18 (interquartile range [IQR] = 12–23) in the ADHD group (100 dyads) and 17 (IQR = 13–23) in the typically developing group (44 dyads; Mann–Whitney U = 2,089, p = .630). However, significant differences were found between groups in the number of EF items on which children overestimated and underestimated their performance in comparison with their parents’ ratings.
The median overestimation scores, representing the number of EF items on which children overestimated their competence compared with their parents’ ratings, were 15 (IQR = 8–22) in the ADHD group and 7 (IQR = 3–15) in the typically developing group (U = 1,317, p < .001). The highest percentages (>55%) of overestimation in the ADHD group were found on the items “Daniel answers a question before he has finished reading it,” “Daniel doesn’t notice if he made a mistake or missed something when he is performing a task,” and “when Daniel gets excited it takes him a long time to calm down.” The median underestimation score, representing the number of EF items on which children underestimated their competence compared with their parents’ ratings, was 0 (IQR = 0–1) in the ADHD group compared with 4 (IQR = 2–10) in the typically developing group (U = 706, p = .000). The highest percentages (>30%) of underestimation in the typically developing group were found on the items “When Daniel wants to say something and he has to wait he forgets what he wanted to say,” “Doesn’t know what to do when he is bored,” and “does things and regrets them afterwards.” The median rates of agreement between dyads regarding the 10 strength items were much higher overall (Mdn = 8, IQR = 6–9, for the ADHD group, and Mdn = 8, IQR = 8–10, for the typically developing group) but significantly different between groups (U = 1,583, p = .006).
Goal Scale
The Goal scale score was calculated for the children with ADHD. Frequency analyses revealed that the median number of items that children wanted to change was 8 (IQR = 3–16). All EF items were identified by at least 10% of the children as treatment goals. The items most frequently identified by children were “distracted by things around” (57%), “can’t find his/her things” (57%), “needs to be told again and again to do things” (48%), “forgets what he/she wants to say” (47%), “doesn’t finish things” (44%), “gets very upset” (40%), and “does things and is sorry afterwards” (39%).
Discussion
In this study, we sought to evaluate the preliminary psychometric properties of the PIC–ME. Findings indicated promising results for the PIC–ME as a child-centered assessment and as a means of engaging children in collaborative goal setting regarding their EF challenges in daily life.
The high internal consistency that we found for the total EF score supports the PIC–ME’s reliability as a measure. Moreover, the large and significant correlation of the total EF score with the BRIEF GEC score supports the PIC–ME’s concurrent validity as a measure of EF in daily life (Gioia et al., 2000). Concerning the EF scales, the reliability of the parent ratings was in the acceptable range for all scales except for the Effort scale (<.7). However, the results for the child ratings were less consistent. This lack of consistency may be attributable to underdeveloped cognitive abilities (e.g., memory and time estimation). Insufficient self-monitoring processes may also explain inconsistencies in their reports (Valla, Bergeron, & Smolla, 2000). This finding was in line with other studies that have shown low reliability of children’s self-assessment in complex areas of functioning such as social and behavioral functioning (Riley, 2004), as opposed to higher reliability in specific and more concrete areas, such as motor functioning (Missiuna et al., 2006).
Comparison of parent PIC–ME EF ratings between groups revealed significant differences on all scales and total EF score. Parents of typically developing children identified EF challenges in approximately 50%–60% of the items on each scale, which was expected because EFs continue to mature well into adolescence (Taylor, Barker, Heavey, & McHale, 2015). However, parents in the ADHD group identified significantly higher percentages (∼90%) of EF items, in accordance with the literature identifying EFDs as a central mechanism of ADHD (Barkley, 2012; Brown, 2013). These findings support the PIC–ME’s construct validity in identifying EF challenges among known groups with EFDs.
Despite the large and significant between-groups differences in parent ratings, these differences were not replicated in the child ratings. The lack of difference between the child groups may represent the PIB among children with ADHD that is extensively documented in the neurological literature, showing that self-ratings of children with ADHD do not differ from those of typically developing children, yet significantly differ from those of their parents or teachers, who identify more challenges than do the children (Jia, Jiang, & Mikami, 2016; Owens et al., 2007; Steward et al., 2017; Whitley et al., 2008). In addition, we found no significant gender or age effect on PIC–ME self-ratings, except for a small and significant correlation of age with the Action scale that suggests that self-report of impulsivity may slightly increase with age. The absence of gender and age effects is in line with reported findings on the consistency of the PIB in children with ADHD across age (from childhood to adolescence) and gender (Krueger et al., 2011; Volz-Sidiropoulou, Boecker, & Gauggal, 2016).
An interesting finding was that we did not observe a trend of overestimation of abilities compared with parents and a lack of difference from peers on the Strengths scale. No significant between-groups differences were found on parent ratings on the Strengths scale (both were high), yet children with ADHD identified significantly fewer strengths than typically developing children. The similar parent ratings among groups may provide support for the validity of this scale in identifying strengths among children with ADHD. The scores of children with ADHD were also high, but they were significantly lower than those of typically developing children, which may reflect these children’s tendency to underestimate their strengths. This finding is in line with the literature that indicates low self-esteem and self-concept among children with ADHD (Palaniappan, Seshadri, Girimaji, & Srinath, 2015). The comparison between child–parent dyads revealed a complex pattern whereby 50% agreement was found in both groups, but on the remaining items the children with ADHD overestimated their abilities, whereas the typically developing children underestimated them. A recent study comparing the EF ratings of adolescents with ADHD and their parents on the BRIEF questionnaire demonstrated a similar pattern of overestimation (Steward et al., 2017). Regarding the higher rates of underestimation among typically developing children, we did not find any other studies on child self-report of EF. However, other studies have found that schoolchildren underestimate their academic abilities compared with parent and teacher reports (Cole, Martin, Peeke, Seroczynski, & Fier, 1999; Phillips, 1987; Pomerantz & Rudolph, 2003). Additional studies are needed to examine this issue further.
The findings regarding the Goal scale indicate that children identify different EF challenges and that all items were valuable in facilitating collaborative goal setting. In sum, these findings demonstrate that children with ADHD can identify some of their EF challenges, even if they cannot identify all of them. Moreover, they can define EF challenges and engage in goalsetting. These findings support the evidence on child-centered practice (Cermak & Bissell, 2014; Costa, 2014; Missiuna et al., 2006).
Limitations
The sample of typically developing children was a convenience sample and smaller than the sample with ADHD. It may not be representative of the general population. In addition, more representation of younger and older age groups is needed to examine the developmental trends in self-perception of executive functioning in occupational contexts across different ages. A larger sample will also enable factor analyses to be conducted to explore dimensionality and determine the extent to which the subscales fit the hypothesized EF constructs. The ADHD sample represents children with EFDs who were referred for Cog–Fun treatment and may overrepresent EF challenges in the population with ADHD. Further studies are needed with a more heterogeneous sample with ADHD.
Regarding the PIC–ME, the pictures may not be sensitive to cultural differences, and further research and item development are needed to examine this issue. Moreover, the length of the assessment may limit its applicability in clinical settings, and different administration methods (such as computerized formats) or abbreviated versions need to be investigated. Finally, the PIC–ME was created for children with ADHD with EF deficits who are referred to occupational therapy to improve their occupational performance. The EF items were selected according to EF models in ADHD, and it is not clear whether the PIC–ME will capture the daily EF challenges of other populations with EFDs, such as children with acquired brain injury (Levin & Hanten, 2005) or autism spectrum disorder (Kenworthy, Yerys, Anthony, & Wallace, 2008). Additional research is needed among populations of children with EFDs and representative samples of typically developing children to further explore the psychometric properties of the PIC–ME.
Implications for Occupational Therapy Practice
This study provided preliminary support for some aspects of the PIC–ME’s reliability and validity:
The PIC–ME may be a valuable tool to support child-centered practice among occupational therapists treating children with ADHD and EFDs that affect their occupational performance.
Children with ADHD were not completely aware of their EF profile, yet they should not be deemed irrelevant to the goal-setting therapeutic process because they demonstrate considerable awareness of approximately 50% of their challenges and an ability to identify treatment goals.
Children’s self-reports should be supplemented with parent reports because of the children’s demonstrated bias in overestimating EF abilities and underestimating strengths.
The PIC–ME may be valuable in reflecting discrepancies between parent–child perception and promoting realistic communication around EF challenges in the family.
Conclusion
To conclude, results support the PIC–ME’s initial reliability and validity among children with ADHD. Despite the finding that children with ADHD underreported their EF challenges compared with their parents’ reports, they were able to identify several challenges and engage in goal setting. Therefore, the PIC–ME may be a valuable tool in a family-centered intervention process including parents and children with ADHD.
Footnotes
Acknowledgments
We thank the students who participated in this project: Stav Dekel, Gat Barami, Hadar Hoori, Zohar Dil, and Oriya Amsily. We are grateful to all the children and parents who generously shared their self-assessments with us.