Derivation,Replication,and Validity Analyses of a Screener for the Behavioral Assessment of Executive Functions in Young Adults

Abstract

Ecologically valid indicators of executive functions are designed to capture dysfunction not easily measured in a lab setting. Here, we present two studies on the development and validity analyses of a behavioral screener for executive functions among young adults. In Study 1, we derived a four-factor (problem solving, attentional control, behavioral control, and emotional control) behavioral screener using a sample of 765 individuals. We used invariance analyses to evaluate the screener’s measurement reliability across sex. In Study 2, we replicated the screener derivation analyses using an independent sample of 197 undergraduates. To further examine the screener’s validity, we evaluated it against a well-known executive functions rating scale. The four-factor model was supported in both samples and analyses provided support for this screener as a valid and reliable measure for everyday executive functions among young adults.

Keywords

executive function behavior rating scale confirmatory factor analysis convergent validity invariance analysis

Over the past two decades, there has been an increasing interest in the development of executive functions (Müller & Kerns, 2015). Generally, executive functions are regarded as the product of numerous complex interactions within and between prefrontal–parietal and corticothalamic–striato–cerebellar areas that are involved in the control of actions, cognitions, and emotions (Duggan & Garcia-Barrera, 2015; Fuster, 2000; Miller & Cohen, 2001; Otero & Barker, 2015). The combination and integration of these interactions gives rise to socially appropriate behavior that can be characterized as volitional and planned, resulting in purposeful action and effective performance (Lezak, Howieson, Bigler, & Tranel, 2012). The growing popularity of the construct of executive functions can largely be attributed to findings that measures of executive functions are associated with, and often also predictive of, a host of aspects of psychological functioning such as social understanding (Devine & Hughes, 2014), academic achievement (Müller & Kerns, 2015), and psychological adjustment (Schoemaker, Mulder, Deković, & Matthys, 2013). Furthermore, executive functions deficits are implicated in a variety of developmental and acquired disorders such as autism and fetal alcohol spectrum disorder (e.g., Fuglestad et al., 2015; Pelicano, 2010). The essential role of executive functions is captured well by Diamond’s (2013, p. 137) conclusion that executive functions “are important to about every aspect of life.” While researchers and clinicians acknowledge that early detection of executive functions impairments is crucial for successful treatment or remediation (Müller & Kerns, 2015; Torralva, Roca, Gleichgerrcht, Bekinschtein, & Manes, 2009), effective detection of such disruption remains a significant challenge in neuropsychology (Garcia-Barrera, Karr, & Kamphaus, 2013; Gioia & Isquith, 2004).

Currently, three general approaches are available for the assessment of executive functions: laboratory-based performance tests, ecologically based tests, and rating scales. Laboratory-based tests have served as the traditional method for assessing executive functions. While these tests have numerous advantages, they are not always sensitive to everyday manifestations of executive dysfunction outside the test setting, or they sometimes indicate executive impairments in individuals not demonstrating any day-to-day functional difficulties (Pennington & Ozonoff, 1996). In response to the limitations of traditional approaches, researchers have developed two alternative approaches designed to capture everyday executive functions (i.e., executive behavior) in a more ecologically valid way. One approach involves the use of ecologically based tests, such as the Multiple Errands Test (Shallice & Burgess, 1991) or the Behavioural Assessment of the Dysexecutive Syndrome (Wilson, Alderman, Burgess, Emslie, & Evans, 1996) that require an individual to interact directly with a given situational context. These instruments offer numerous advantages including eliciting practical, environmentally driven information helpful for improved diagnostics, predicting real-world performance, and treatment planning. However, current measures also have substantial limitations, particularly with regard to norms, administration demands, test sensitivity, and they have generally been developed for the assessment of older individuals and clinical populations (cf. Burgess et al., 2006; Manchester, Priestley, & Jackson, 2004; Müller & Kerns, 2015; Wilson et al., 1996).

A second approach has been to gather information directly as it pertains to an individual’s everyday experiences through executive functions behavioral rating scales (Isquith, Roth, & Gioia, 2013; Meltzer & Krishnan, 2007; Silver, 2014). Although rating scales were initially intended to complement performance-based tests, recent research indicates rating scales have their own value, providing unique, but complementary and corroborative information about different aspects of executive functions as they pertain to an individual’s everyday experience (Isquith et al., 2013; McAuley, Chen, Goos, Schachar, & Crosbie, 2010; Silver, 2014; Toplak, West, & Stanovich, 2013). Executive functions rating scales are in their early development compared with other neuropsychological assessment measures, having only gained research support and considered as an integral component of executive functions assessment within the past decade (Gioia, Kenworthy, & Isquith, 2010; Silver, 2014). Despite this support, assessment of executive functions via rating scales is still associated with a number of challenges including individual bias, personal and cognitive characteristics of raters, environmental influences affecting ratings, the operationalization of the construct, and difficulty in parsing out specific executive functions deficits (Gioia, Isquith, & Kenealy, 2008; Grace & Malloy, 2001; Isquith et al., 2013). While some of these challenges are inherent to the rating scale approach, others may be more readily addressed and ameliorated (e.g., definition and operationalization of executive functions).

Despite considerable agreement on the general attributes of executive functions, there is no consensus on the component structure of this construct (Goldstein, Naglieri, Princiotta, & Otero, 2015). One influential approach to this issue applied latent variable analysis to demonstrate both the integration (unity) and multidimensionality (diversity) of three components of executive functions: inhibitory control, updating working memory, and mental shifting (Miyake et al., 2000). This seminal work has significantly informed our understanding of the nature of executive functions, and provided a key framework for subsequent research using an assessment approach based on computer-based performance instruments (Frazer, 2012; Miyake & Friedman, 2012). Given the multidimensionality of executive functions, behavior rating scales of this construct should capture: (a) the diversity of its interrelated functions; (b) the unity of these interactions—organizing, controlling, and directing cognitive activity, behavior, and emotional responses; and (c) the environmental context of everyday life (Gioia et al., 2008; Roth, Isquith, & Gioia, 2015).

A number of instruments are now available for the behavioral assessment of executive functions; however, different definitions and approaches underlie their development and rating scales vary in the number of factors they purportedly measure (Garcia-Barrera et al., 2013). Furthermore, to our knowledge, there is currently no measure specifically developed for the assessment of executive functions in the midadolescence to early-adulthood period. Instead, scales appear to have been developed for age groups either in the preschool to late adolescence range (2-18 years old) or in the whole adulthood range (18-90 years), but not specifically for the age period from 18 to 25 years (emerging adulthood), an important period for the assessment executive functions (Reynolds & MacNeill Horton, 2008). It is worth noting that while these scales typically provide norms for the emerging adulthood age group, their item content does generally not capture characteristics uniquely associated with this period (Naglieri & Goldstein, 2015a, 2015b). Age-appropriate items would probe for behaviors such as increased novelty and sensation seeking, social exploration, and risk taking (Crews, He, & Hodge, 2007; Naglieri & Goldstein, 2015a, 2015b), environmental factors such as educational attainment and increased risk for substance abuse (Blakemore & Choudhury, 2006; Crews et al., 2007), and clinical factors such as the emergence of psychopathologies associated with executive impairments (e.g., schizophrenia, obsessive–compulsive disorder; Hersen & Beidel, 2012).

One approach to address some of these issues, proposed by Garcia-Barrera, Kamphaus, and Bandalos (2011), consists in the development of a behavioral screener for the assessment of executive functions derived from the Behavioral Assessment System for Children (BASC; Reynolds & Kamphaus, 1992). The BASC, now in its third edition (BASC-3; Reynolds & Kamphaus, 2015), is a multidimensional broadband rating scale designed for the assessment of the behaviors and self-perceptions of preschoolers (ages 2-5), children (ages 6-11), adolescents (ages 12-21), and college students (ages 18-25). Garcia-Barrera et al. (2011) originally developed their BASC executive functions screener by selecting 25 items from the BASC Teacher Ratings Scale–Child (BASC-TRS-C) form that were related to four executive factors: Problem Solving, Attentional Control, Behavioral Control, and Emotional Control. The Problem Solving factor refers to “the ability to plan, problem-solve, make decisions, and organize information towards the execution of a goal,” and it includes eight items (Cronbach’s α = .805; Garcia-Barrera et al., 2011, p. 67). The Attentional Control factor refers to “the ability to focus, sustain, and shift attention systems according to task demands” (p. 67), and it includes seven items (α = .890). The Behavioral Control factor refers to the self-regulation of behavior, including inhibition/impulse control, and it includes six items (α = .842). Finally, the Emotional Control factor refers to “the ability to self-regulate emotional response to environmental and internal cues” (p. 68), and it includes five items (α = .845).

This multidimensional model demonstrated adequate fit over and beyond a unidimensional model and invariance for two age groups (6-8 years vs. 9-11 years) and sex (Garcia-Barrera et al., 2011). A number of subsequent studies have replicated the derivation of this behavioral screener for executive functions, providing support for its reliability and validity. Specifically, Garcia-Barrera et al. (2013) were able to replicate the screener and provided evidence for its measurement reliability across time using longitudinal invariance analyses and latent growth modeling in a large sample of children (N = 1,237). Sadeh, Burns, and Sullivan (2012) independently showed that the four-factor model had adequate fit in a sample of kindergarteners at risk for developing behavior disorders (N = 220). Empirical support has also been provided for the use of this instrument in preschoolers (Karr et al., 2013), and in clinical and cross-cultural samples (i.e., attention deficit hyperactivity disorder, Colombian children; Garcia-Barrera, Karr, Duran, Direnfeld, & Pineda, 2015). Garcia-Barrera et al.’s (2011) multidimensional model may offer an effective and practical method for the screening of executive functions. Further research, however, is needed to replicate and evaluate this instrument in adolescents and young adults. Additionally, prior studies have not examined how this instrument compares with other executive functions behavioral rating scales; therefore, analysis of its convergent validity is necessary to better understand its potential utility.

The BASC Second Edition Self-Report of Personality–College (BASC-2-SRP-COL) form contains 185 items, 68 of which consist of dichotomous (true/false) items, and 117 items to be rated on a 4-point Likert-type scale (Reynolds & Kamphaus, 2004). While most other ratings scales for individuals ages 18 to 25 are designed for use in all adults (e.g., 18-90 years), the items on the BASC-2-SRP-COL are specifically designed to capture many behavioral, emotional, self-concept formation issues that are specific to this developmental period (representing the transition from adolescence into early adulthood) and are highly relevant to normative executive functions (Reynolds & Kamphaus, 2004). Given the support for Garcia-Barrera et al.’s (2011) four-factor model (as derived from other forms of the BASC), a similar screener derived from the BASC-2-SRP-COL may serve as an effective executive functions measure for young adults.

Considering the variability of executive functions structures reported across ages and the absence of rating scales with content specifically designed to capture the unique factors critical to emerging adulthood, we were led to the present article. We present two studies that were aimed at developing and examining the reliability and validity of an executive functions ratings scale for young adults.

Study 1: Derivation of the Executive Functions Screener

The main objectives of Study 1 were to answer the following questions: Can an executive functions screener for young adults be derived from the BASC-2-SRP-COL using the methods previously applied in the derivation of different versions of the BASC for younger populations? What are the statistical properties of the BASC-2-SRP-COL executive functions screener? What model best describes the data (e.g., a unidimensional model or a multidimensional model)?

Given that previous studies successfully derived an executive functions screener with other versions of the BASC (Garcia-Barrera et al., 2011, 2013, 2015; Karr et al., 2013; Sadeh et al., 2012), we anticipated we would also be able to derive the target instrument in this study. In evaluating the statistical properties of the derived screener, we aimed to determine which factor structure best explained the data. Based on previous research with the BASC executive functions screener, we hypothesized that a four-factor model would provide the best fit to the data and yield an instrument with the strongest statistical properties.

A further goal of Study 1 was to examine whether the BASC executive functions screener measures executive behavior in the same way across sex. While this instrument has shown measurement invariance for sex in 6- to 11-year-olds (Garcia-Barrera et al., 2011, 2013, 2015), research has not clearly established whether this measurement invariance is sustained across ages on this or other executive functions rating scales. For example, significant sex differences were found on four out of five scales on the Barkley Deficits in Executive Functioning Scale for Children and Adolescents (BDEFS-CA; ages 6-17), but only two of five scales on the BDEFS (ages 18-81; Barkley, 2015). Similarly, more sex differences were found on preschool, child, and adolescent versions of the Behavior Rating Inventory of Executive Function (BRIEF) than the adult versions; however, these findings also appeared to be influenced by the individual providing the ratings (e.g., self, parent, teacher) and the specific scale being examined (e.g., boys are typically rated to have greater problems on the Inhibit scale, but girls are typically rated to have more problems on the Emotional Control scale; Roth et al., 2015). Furthermore, the presence of sex differences across traditional performance-based tasks of executive functions is widely variable (cf. Strauss, Sherman, & Spreen, 2006) and there is emerging evidence for sex differences in the development and function of neural structures theorized to support executive functions (Naglieri & Goldstein, 2015b). In Study 1, we assessed the screener’s measurement equivalence across sex by using configural, weak, and strong measurement invariance testing. Given prior research supporting sex measurement equivalence of executive function for other age groups, we hypothesized that our derived screener for college-aged students would demonstrate similar invariance.

Method

Participants

We conducted the screener derivation process using the original BASC-2-SRP-COL standardization database, provided with permission from Pearson Assessment (hereafter referred to as the derivation sample). This sample includes a diverse group of 765 individuals representative of the U.S. college-aged population. A summary of the derivation sample’s demographic characteristics is provided in Table 1. Further information about this sample and the data collection procedures is found in the BASC-2 manual (Reynolds & Kamphaus, 2004). Use of these data was approved by the Human Research Ethics Board at the University of Victoria (Protocol Number 14-335).

Table 1.

Demographics.

	Derivation sample	Replication sample	Replication BRIEF-A subsample
Sample size	765	197	97
Age (M ± SD in years)	20.57 + 1.61	20.46 ± 1.80	20.58 ± 1.81
Representation by developmental age (%)
18	16.21	15.74	15.46
19	27.19	18.78	15.46
20	19.08	17.77	19.59
21	17.78	20.81	18.56
22	11.11	12.69	15.46
23	5.49	7.11	8.25
24	1.44	5.58	6.19
25	1.70	1.52	1.03
Representation by sex (%)
Female	56.60	79.19	73.20
Male	43.40	20.81	26.80

Note. BRIEF-A = Behavior Rating Inventory of Executive Function–Adult version.

Screener Derivation Process

The entire set of 185 items from the BASC-2-SRP-COL was carefully reviewed and items potentially “executive” in nature were isolated. This designation was based on previous studies that had established the screener measures the following four executive functions: problem solving, attentional control, behavioral control, and emotional control (Garcia-Barrera et al., 2011, 2013). Overall, 50 items were initially extracted, serving as latent indicators of the construct. It is important to note that, while large, this number reflects several sets of very similar or near-duplicate items (e.g., I have trouble paying attention to what I am doing and I have trouble paying attention to lectures), which were optimally refined during the item screening process (see below).

Item screening included tests for multivariate normality (Mahalanobis distance), analysis of frequency distributions, and examination of potential collinearity and item content (as described in Garcia-Barrera et al., 2011). Validity indices derived from the BASC (i.e., the F, V, and L indices) were also examined to screen for any potential problematic patterns of responding. Internal consistency reliabilities for each group of items (factors) were estimated using Cronbach’s α. In consideration of the complexity of the four executive functions factors, along with the small number of indicators being used to estimate each factor, internal consistency reliability values greater than 0.6 were established as acceptable and values greater than 0.7 established as good (Hair, Black, Babin, & Anderson, 2010; Kline, 2013). Refining the remaining pool of items was guided by several principles typically used for reviewing item content and optimizing reliability coefficients. This included ensuring each factor consisted of a balanced set of items that capture both the underlying core construct, as well as the complexity each construct is theorized to represent. Furthermore, when possible, items were to belong to different original scales of the BASC to ensure factors are not merely abbreviated BASC scales. All descriptive statistics, frequency distributions, correlations, tests for multivariate normality, and reliability coefficients were calculated using IBM SPSS Version 18.

Confirmatory factor analysis (CFA) was used to evaluate the a priori four-factor model of executive functions serving as the target model for this study (referred to as Model 4). This included both an evaluation of the model’s fit to the observed data, as well as an evaluation of the model in comparison with three alternative CFA models (referred to as Models 1-3; see Figure 1). The alternative CFA models were chosen based on the strongest theoretical variations consistent with the current conceptualization of executive functions, in consideration of the item content of the selected indicators, and in line with the alternative models evaluated by Garcia-Barrera et al. (2011). The first alternative model (Model 1) was a one-factor (unidimensional) model with 23 indicators. Since this model contained all of the isolated executive functions indicators (conceptualized as contributing to one unitary construct), the latent construct was called Executive Function. The second alternative model (Model 2) was a two-factor model comprising a Problem Solving latent construct (6 items) and a behavioral self-regulation latent construct (17 items), which combines items originally comprising the Attentional Control, Behavioral Control, and Emotional Control factors. The third alternative model (Model 3) was a three-factor model, corresponding to three latent constructs: Problem Solving (6 items), Attentional Control (6 items), and Behavioral/Emotional Control (11 items; formerly Behavioral Control and Emotional Control).

Figure 1.

Construct validity analyses of the BASC-2-SRP-COL executive functions screener: Unidimensional and multidimensional models tested.

Model fit was evaluated using the χ² test, the comparative fit index (CFI), the Tucker–Lewis index (TLI), and the root mean square error of approximation (RMSEA). For this study, the cutoff criterion for all fit indices were set to those typically regarded as indicative of “good fit” (Hu & Bentler, 1999; Jackson, Gillaspy, & Purc-Stephenson, 2009; Marsh, Balla, & McDonald, 1988; Browne & Cudeck, 1993 as cited in Schermelleh-Engel, Moosbrugger, & Müller, 2003) and used in prior research with different derivations of the BASC screener (Garcia-Barrera et al., 2011, 2013). All analyses of factor structure and latent means were conducted using Mplus Version 7.11 (Muthén & Muthén, 1998-2012) using the specification for categorical data (which employs the weighted least squares with mean and variance adjusted estimator), and the missing completely at random assumption for each analysis.

Analyses of Measurement Invariance

Once the screener was derived, we conducted a series of stepwise analyses to examine if this instrument had the same meaning across sex. Configural, metric, and scalar invariance tests were performed in accordance with methods previously described in the evaluation of a similar instrument for children (Garcia-Barrera et al., 2013). The primary metrics selected for model comparison included Δχ² and ΔCFI. Since using the χ² metric can be problematic with nonnormal data, ΔCFI served as the primary difference testing metric, following the typically recommended cutoff of ≥.01 (Cheung & Rensvold, 2002; Yu, 2002).

Results

Data and Item Screening

Inspection for multivariate normality revealed no significant outliers (α = .05) and review of the BASC validity indices did not indicate any cases for exclusion from further analyses. The preliminary pool of 50 potential indicator items (14 Problem Solving items, 10 Attentional Control items, 15 Behavioral Control items, 11 Emotional Control items) was refined during item screening. Four items exceeded the established skewness and kurtosis criteria and were removed (Items 42, 45, 54, and 58). Review of intra- and interfactor item correlations resulted in the removal of seven items due to poor correlations (Items 7, 49, 120, 136, 149, 151, and 178; Supplemental Table S1 includes the final item correlation matrix [all supplemental materials are available online at http://asm.sagepub.com/content/by/supplemental-data]). Finally, review of item content and optimizing reliability coefficients for each factor resulted in the removal of 15 items (Items 57, 66, 88, 98, 101, 102, 105, 108, 109, 113, 117, 128, 129, 137, and 179). Table 2 summarizes the final set of 23 items and their original BASC-2-SRP-COL scale membership.

Table 2.

Distribution and Descriptive Statistics of the Final Set of BASC-2-SRP-COL Executive Function Items per Scale.

Item	Original scale membership	Derivation sample (N = 765)				Replication sample (N = 197)
Item	Original scale membership	M	SD	Skewness	Kurtosis	M	SD	Skewness	Kurtosis
Problem Solving
Item 56	Sense of inadequacy	0.26	0.44	1.10	−0.78	0.17	0.35	0.18	0.12
Item 87	Sense of inadequacy	0.60	0.62	0.69	0.35	1.51	0.68	1.26	1.53
Item 95	Test anxiety	1.33	0.82	−0.02	−0.63	2.12	0.65	0.00	−0.35
Item 107	Self-reliance	1.08	0.72	0.09	−0.56	2.20	0.71	−0.22	−0.84
Item 118	Sense of inadequacy, test anxiety	0.68	0.63	0.76	1.31	1.54	0.56	0.36	−0.89
Item 138	Self-reliance, ego–strength	1.22	0.76	0.24	−0.26	2.36	0.70	−0.16	−0.39
Attentional Control
Item 5	Attention problems	0.48	0.50	0.07	−2.00	0.40	0.49	0.41	−1.85
Item 36	Attention problems	0.27	0.44	1.06	−0.88	0.19	0.40	1.57	0.47
Item 67	Attention problems	0.36	0.48	0.59	−1.65	0.21	0.41	1.41	−0.01
Item 125	Attention problems	1.23	0.72	0.83	0.78	2.13	0.62	0.57	1.18
Item 148	Attention problems	0.49	0.65	1.17	1.03	1.37	0.50	0.80	−0.79
Item 160	Attention problems	0.91	0.67	0.74	1.49	1.77	0.54	0.08	0.88
Behavioral Control
Item 97	Hyperactivity	1.02	0.86	0.68	−0.03	1.73	0.80	0.94	0.37
Item 144	Hyperactivity, mania	1.21	0.67	0.52	0.57	2.19	0.70	0.44	0.38
Item 155	Mania	0.82	0.73	0.65	0.22	1.78	0.78	0.93	0.79
Item 159	Hyperactivity	0.86	0.57	0.20	0.91	1.75	0.56	0.35	1.55
Item 174	Mania	0.95	0.80	0.72	0.30	1.70	0.76	1.00	0.75
Item 175	Hyperactivity	0.60	0.75	1.13	0.81	1.44	0.63	1.36	1.91
Emotional Control
Item 16	Anxiety	0.27	0.44	1.05	−0.90	0.21	0.41	1.45	0.10
Item 47	Anxiety	0.50	0.50	0.02	−2.01	0.60	0.49	−0.43	−1.84
Item 126	Anxiety	1.29	0.76	0.56	0.15	2.36	0.73	0.36	−0.04
Item 156	Anger control	0.87	0.75	0.69	0.41	1.87	0.85	0.81	0.08
Item 171	Anxiety, anger control	1.16	0.72	0.43	0.24	2.25	0.68	0.72	0.79

Note. BASC-2-SRP-COL = Behavior Assessment System for Children Second Edition Self-Report of Personality–College.

Internal Consistency Reliability

The internal consistencies (Cronbach’s α) for the four executive functions factors were in the acceptable range. Specifically, the alpha coefficients were .704 for Problem Solving, .801 for Attentional Control, .658 for Behavioral Control, and .695 for Emotional Control. For the whole screener, an alpha coefficient of .843 was obtained. Alpha coefficients did not improve with the deletion of any items; therefore, all items were retained.

Confirmatory Factor Analysis

All models converged normally and the fit indexes for each are reported in Table 3. Overall, each model (Models 1-4) demonstrated increasingly better model fit in terms of all selected fit indicators. Progressively lower (less inflated) χ² values, improved (lower) RMSEA values, and larger CFI and TLI values were observed with each new model. Model 4 was the only model to demonstrate acceptable fit using these metrics (χ² = 886.708, RMSEA = 0.062, CFI = 0.912, TLI = 0.901).

Table 3.

23 Item Model Variation Analyses for the BASC-2-SRP-COL Executive Functions Screener With the Derivation Sample (N = 765).

Model	WLMSV χ²	Δχ²	df	CFI	ΔCFI	TLI	RMSEA [90% CI]
Model 1: One factor	1986.510*		80	0.768		0.745	0.100 [0.096, 0.104]
Model 2: Two factors	1644.860*	341.650	81	0.813	−0.045	0.793	0.090 [0.086, 0.094]
Model 3: Three factors	1331.624*	313.236	83	0.854	−0.041	0.837	0.080 [0.076, 0.084]
Model 4: Four factors	886.708*	444.916	86	0.912	−0.058	0.901	0.062 [0.058, 0.067]

Note. BASC-2-SRP-COL = Behavior Assessment System for Children Second Edition Self-Report of Personality–College; WLMSV = weighted least squares means and variance adjusted; df = degrees of freedom; CFI = comparative fit index; TLI = Tucker–Lewis index; RMSEA = root mean square of approximation; CI = confidence interval.

p < 0.0001.

Parameter Estimates

Figure 2 represents the final model configuration (Model 4) and indicates the correlations among factors. The correlations ranged from .251 to .629, which are considered low to moderate. The individual item factor loadings for the final model configuration (Model 4) are summarized in Supplemental Table S2. The factor loadings for Model 4 ranged from .462 to .856 overall; .583 to .674 for Problem Solving (with 4/6 above |.60|), .687 to .856 for Attentional Control (with 6/6 above |.60|), .462 to .769 for Behavioral Control (with 1/6 above |.60|), and .630 to .712 for Emotional Control (with 5/5 above |.60|).

Figure 2.

Final four-factor confirmatory factor analysis model for the BASC-2-SRP-COL executive functions screener using the derivation sample.

Analyses of Measurement Invariance

Table 4 summarizes the results of the measurement invariance analyses across sex. The select fit indices (CFI, TLI, and RMSEA) all demonstrated adequate goodness of fit. Moving from the configural invariance model to the weak invariance model resulted in a ΔCFI = −.013, while moving from the weak invariance model to the strong invariance model resulted in ΔCFI = +.003.

Table 4.

Invariance Testing Across Sex for the BASC-2-SRP-COL Executive Functions Screener Four-Factor Model With the Derivation Sample (N = 765).

Invariance	WLMSV χ²	Δχ²	df	CFI	ΔCFI	TLI	RMSEA [90% CI]
Configural	1263.677*		141	0.899		0.894	0.065 [0.061, 0.070]
Weak	1182.634*	81.043	122	0.912	−0.013	0.911	0.060 [0.056, 0.064]
Strong	1239.869*	−57.235	88	0.909	0.003	0.914	0.059 [0.055, 0.063]

Note. WLMSV = weighted least squares means and variance adjusted; df = degrees of freedom; CFI = comparative fit index; TLI = Tucker–Lewis index; RMSEA = root mean square error of approximation; CI = confidence interval.

p < 0.0001.

Summary and Discussion

We successfully derived an executive functions screener from the BASC-2-SRP-COL. While the Attentional Control factor was the only factor that did not represent a diversity of original BASC scales, only items considered as latent indicators of Attentional Control (and not simply “Attentional Problems” as indicated by the original BASC scale) were included. Overall, each factor (Problem Solving, Attentional Control, Behavioral Control, and Emotional Control) consisted of a diverse, yet parsimonious set of items that captured both the essence and the complexity each construct is theorized to represent. While the internal consistency reliabilities fell below those reported on previous studies of the executive functions screener using different versions of BASC (Garcia-Barrera et al., 2011, 2013; Sadeh et al., 2012), they all still fall in the acceptable range. Additionally, it is worth noting that the BASC-2-SRP-COL is unique in its inclusion of items with dichotomous true/false response choices, a fact which may explain the lower alpha coefficients observed in the present study.

We used CFA to examine the statistical properties of this screener by comparing four different models (ranging from one to four factors). Overall, these analyses indicated that a four-factor model (consisting of problem solving, attentional control, behavioral control, and emotional control factors) best described the data. Furthermore, the strength of the observed factor loadings explains the overall fit of the model and the adequacy of the selected indicators. This was consistent with our hypothesis that a four-factor model would provide the best fit to the data and yield an instrument with the strongest statistical properties. Using configural, weak, and strong measurement invariance testing across sex indicated that this instrument measures executive behavior in the same way across sex as the observed differences were reasonably close the recommended cutoff.

Study 2: Replication and Convergent Validity Analyses

The primary aim of Study 2 was to determine if the results from Study 1 could be replicated in an independent sample. Again, given the prior successful replications of this screener derivation process and support for the measurement invariance of the screener’s underlying four-factor model, we hypothesized that we would replicate our Study 1 results in Study 2.

The second goal of Study 2 was to examine the convergent validity of the derived BASC executive functions screener. To do this, we compared it with a well-established rating scale for executive functions, the Behavior Rating Inventory of Executive Function–Adult version (BRIEF-A; Roth et al., 2005), using a subgroup of the Study 2 replication sample. The BRIEF-A is the most commonly used and empirically supported rating scale for the assessment of behavioral manifestations of executive dysfunction in adults aged 18 to 90 years (Isquith et al., 2013). The BRIEF-A contains 75 items rated on a 3-point Likert-type scale (never, sometimes, and often), which contribute to nine scales: Inhibit, Shift, Emotional Control, Self-Monitor, Initiate, Working Memory, Plan/Organize, Task Monitor, and Organization of Materials. We hypothesized that our analyses would yield moderate to strong correlations between the BASC and the BRIEF-A and provide evidence for the convergent validity of the derived BASC-2-SRP-COL executive functions screener.

Method

Participants

One hundred and ninety-seven healthy, neurotypical university students were recruited to take part in Study 2 (hereafter referred to as the replication sample). Participants were excluded from the sample if they were not between the ages of 18 and 25 years (inclusive) or if they reported a significant history of neurologic or psychiatric disturbance (e.g., traumatic brain injury, seizures, mental illness), developmental disorder (e.g., attention deficit hyperactivity disorder, fetal alcohol spectrum disorder, autism), learning disability, or substance abuse. Table 1 provides a summary of the replication sample’s demographic characteristics. The study was approved by the Human Research Ethics Board at the University of Victoria (Protocol Number 12-466). All participants were informed of the study’s procedures, risks, and benefits and provided written consent before participating. All participants completed the BASC-2-SRP-COL (described above).

Replication Analyses

The same methods employed in screener derivation process described in Study 1 (above) were used to replicate the screener with this independent replication sample.

Convergent Validity Analyses

In an effort to examine the convergent validity of the derived screener, a subset of 97 participants from the replication sample was randomly assigned to complete the BRIEF-A (Roth et al., 2005) in addition to the BASC-2-SRP-COL. A summary of the demographic characteristics of the BRIEF-A subsample of the replication sample is provided in Table 1.

Monotrait (executive functions) and monomethod (self-rating scale) correlations (as outlined by Campbell & Fiske, 1959; Cronbach & Meehl, 1955) were examined by evaluating the four BASC executive functions factors (Problem Solving, Attentional Control, Behavioral Control, and Emotional Control) with their corresponding scales from the BRIEF-A. To do this, the conceptualization and item content of the BRIEF-A scales were carefully reviewed and the scales determined to be most similar to the four BASC executive functions factors were isolated as “target” scales. The Plan/Organize scale, which measures the “ability to manage current and future-oriented task demands within the situational context” including planning, implementing and achieving goals, and organizing to achieve an objective, was selected as the scale best representing Problem Solving (Roth et al., 2005, p. 22). The Working Memory scale, which examines the “capacity to actively hold information in mind for the purpose of completing a task or generating a response . . . [and] the ability to sustain attention and performance over time,” was selected as the one most similar to attentional control (pp. 21-22). The Inhibit scale, which measures “inhibitory control . . . and the ability to stop one’s own behavior at the appropriate time,” was isolated as most closely representing Behavioral Control (p. 20). Finally, the Emotional Control scale, which measures the “ability to modulate emotional responses,” was selected as best representing emotional control (p. 21).

In addition to these four target scales, the BRIEF-A also contains five other “nontarget” scales, considered less similar to the four BASC factors: Organization of Materials, Shift, Initiate, Task Monitor, and Self-Monitor. The Organization of Materials scale “measures organization in the adult’s everyday environment with respect to orderliness of work, living, and storage spaces” (p. 22). The Shift scale “measures the ability to move freely from one situation, activity, or aspect or a problem to another, as the circumstances demand . . . [it] is composed of items reflecting the ability to shift behaviorally and to shift cognitively” (p. 20). The Task Monitor scale measures “the extent to which the individual keeps track of his or her own problem-solving success or failure” (p. 22), and the Self-Monitor scale measures “the extent to which the adult keeps track of his or her own behavior and the effect of his or her behavior on others” (p. 21).

Comparisons between the BASC factors and the BRIEF-A scales were made using the following steps. First, the BRIEF-A validity indices (Negativity, Infrequency, and Inconsistency scales) were reviewed to screen for any potential problematic patterns of responding. Second, raw item scores of each of the four BASC executive functions factors and the nine total BRIEF-A scales were converted into T-scores. Next, each BASC factor was evaluated against its corresponding BRIEF-A target scale using pairwise Pearson’s r and Spearman’s ρ correlations (i.e., BASC Problem Solving with BRIEF-A Plan/Organize; BASC Attentional Control with BRIEF-A Working Memory; BASC Behavioral Control with BRIEF-A Inhibit; and BASC Emotional Control with BRIEF-A Emotional Control). Subsequently, each BASC factor was evaluated against the eight remaining BRIEF-A scales (e.g., BASC Problem Solving with BRIEF-A Working Memory, Inhibit, Emotional Control, Organization of Materials, Shift, Initiate, Task Monitor, and Self-Monitor). These comparisons were made in an effort to assess the specificity of the relations between the BASC factors and their target scales.

Finally, we used structural equation modeling (SEM) in a series of steps (as outlined by Hoyle & Smith, 1994) to further evaluate the construct validity of the derived four-factor BASC executive functions screener. The first step involved obtaining support for the measurement model of the latent variables underlying the scale; this was accomplished by using CFA and examining the internal consistency reliabilities (see Screener Derivation Process section). Next, the target and nontarget variables from the BRIEF-A (i.e., Plan/Organize, Working Memory, Inhibit, Emotional Control, Organization of Materials, Shift, Initiate, Task Monitor, and Self-Monitor) were modeled as latent variables within a separate nine-factor model, also using CFA. Finally, by using SEM the focal latent variables (Problem Solving, Attentional Control, Behavioral Control, and Emotional Control) were correlated with the target and nontarget latent variables using their latent means, and the standardized covariance (i.e., Pearson correlation coefficients) between the corresponding pairs of focal (BASC) and criterion (BRIEF-A) latent variables were inspected. We did this in order to assess the specificity of the correlations between the focal and target latent variables and to more broadly examine the convergent validity of the derived BASC screener (see Figure S1).

Results

Data and Item Screening

Inspection for multivariate normality identified no significant outliers and review of the BASC validity indices did not indicate any cases warranting exclusion from further analyses. Distribution and descriptive statistics (Table 2) and intra- and interfactor item correlations (Table S3) were reviewed and compared with those calculated with the derivation sample. No significant variations between statistics calculated for the derivation sample and the replication sample were identified.

Internal Consistency Reliability

Using the replication sample, the internal consistencies (Cronbach’s α) for the factors and the screener fell within the acceptable range. The alpha coefficient for the total screener was .799 (−.044 from total α for the derivation sample). The factor alpha coefficients were .658 for Problem Solving (Δα = −.046), .664 for Attentional Control (Δα = −.137), .647 for Behavioral Control (Δα = −.011), and .660 for Emotional Control (Δα = −.035).

Confirmatory Factor Analysis

Just as with the derivation sample in study 1, CFA was used in Study 2 to evaluate (and replicate) the target four-factor model of executive in comparison with a series of three alternative CFA models (Models 1-3; Figure 1). All models converged normally and the fit indexes for each are reported in Table 5. Each model (Models 1-4) demonstrated increasingly better model fit in terms of all selected fit indicators, with larger CFI and TLI values, lower (less inflated) χ² values, and improved (lower) RMSEA values. Here, Model 4 was the only model to have an acceptable CFI, TLI, and RMSEA values, all falling below the optimal cutoff criteria.

Table 5.

23 Item Model Variation Analyses for the BASC-2-SRP-COL Executive Functions Screener With the Replication Sample (N = 197).

Model	WLSMV χ²	Δχ²	df	CFI	ΔCFI	TLI	RMSEA [90% CI]
Model 1: One factor	514.426*		230	0.734		0.708	0.079 [0.070, 0.088]
Model 2: Two factors	483.857*	30.569	229	0.762	−0.028	0.737	0.075 [0.066, 0.084]
Model 3: Three factors	427.422*	56.435	227	0.813	−0.051	0.791	0.067 [0.057, 0.077]
Model 4: Four factors	316.867*	110.555	224	0.913	−0.100	0.902	0.046 [0.034, 0.057]

Note. BASC-2-SRP-COL = Behavior Assessment System for Children Second Edition Self-Report of Personality–College; WLSMV χ² = weighted least squares with mean variance adjusted chi-square; df = degrees of freedom; CFI = comparative fit index; ΔCFI = change in comparative fit index between models; TLI = Tucker–Lewis index; RMSEA = root mean square error of approximation.

p = .000.

Parameter Estimates

Figure 3 represents the four-factor model configuration and the correlations among factors for the replication sample. Here, the correlations were similar to those calculated using the derivation sample. They ranged from .292 to .683, again considered low to moderate. The individual item factor loadings calculated for the replication sample summarized in Supplemental Table S4. Overall, they ranged from .363 to .724, with 70% of the items having factor loadings higher than .60.

Figure 3.

Replication of the four-factor confirmatory factor analysis model for the BASC-2-SRP-COL executive functions screener using the replication sample.

Convergent Validity Analyses

To examine the evidence for the convergent validity of the four-factor BASC executive functions screener, Model 4 was compared with the BRIEF-A using a subsample of the replication sample.

Correlations between the BASC and the BRIEF-A

Review of the BRIEF-A validity indices, as well as distribution and descriptive statistics for the selected BRIEF-A items did not flag any cases for potential exclusion from further analyses (see Table S5). The convergent validity analyses of the BASC executive functions screener first consisted of computing correlations between the T-scores of its four composite scores and the T-scores of the BRIEF-A target scales. These results are summarized in Table 6. Overall, the strongest correlation was between Problem Solving (BASC) and Plan/Organize (BRIEF-A), with moderate to weak correlations between the other factors and scales. Correlations between BASC factors and nontarget scales were all generally lower than the target scale correlations.

Table 6.

Correlations Between the BASC Factor T-Scores and BRIEF-A Scale T-Scores Calculated Using the Replication Subsample Data Set (n = 97).

BASC factor	r	p	ρ	p
Problem Solving
Target BRIEF scale
Plan/Organize	.652	.000	.650	.000
Nontarget BRIEF scales
Working Memory	.433	.000	.439	.000
Inhibit	.330	.001	.368	.000
Emotional Control	.434	.000	.424	.000
Organization of Materials	.178	.081	.208	.041
Shift	.496	.000	.455	.000
Initiate	.463	.000	.469	.000
Task Monitor	.567	.000	.572	.000
Self-Monitor	.397	.000	.379	.000
Attentional Control
Target BRIEF scale
Working Memory	.615	.000	.622	.000
Nontarget BRIEF scales
Plan/Organize	.384	.000	.432	.000
Inhibit	.585	.000	.622	.000
Emotional Control	.265	.009	.299	.003
Organization of Materials	.075	.467	.091	.373
Shift	.261	.010	.307	.002
Initiate	.420	.000	.476	.000
Task Monitor	.542	.000	.558	.000
Self-Monitor	.274	.007	.246	.015
Behavioral Control
Target BRIEF scale
Inhibit	.489	.000	.480	.000
Nontarget BRIEF scales
Plan/Organize	.260	.010	.322	.001
Working Memory	.465	.000	.473	.000
Emotional Control	.270	.008	.237	.020
Organization of Materials	.238	.008	.261	.010
Shift	.280	.006	.327	.001
Initiate	.286	.004	.299	.003
Task Monitor	.203	.046	.216	.034
Self-Monitor	.380	.000	.318	.002
Emotional Control
Target BRIEF scale
Emotional Control	.588	.000	.520	.000
Nontarget BRIEF scales
Plan/Organize	.256	.011	.275	.006
Working Memory	.261	.010	.265	.009
Inhibit	.188	.065	.188	.066
Organization of Materials	−.057	.579	−.050	.626
Shift	.505	.000	.456	.000
Initiate	.274	.007	.273	.007
Task Monitor	.246	.015	.243	.017
Self-Monitor	.220	.029	.159	.119

Note. BASC = Behavioral Assessment System for Children; BRIEF-A = Behavior Rating Inventory of Executive Function–Adult version. Pearson (r) and Spearman’s (ρ) correlations are each followed by their respective p values.

SEM analysis of the BASC and the BRIEF-A

Subsequently, the four target BRIEF-A scales were modeled as a four-factor model with 36 indicator items using CFA. The model converged normally and the following fit indexes were observed: χ² = 773.174, p = .000, df = 588, CFI = 0.884, TLI = 0.876, RMSEA = 0.057. Next, the nine BRIEF-A scales were modeled as a nine-factor model with 71 indicator items. This model also converged normally: χ² = 5296.967, p = .000, df = 2485, CFI = 0.876, TLI = 0.871, RMSEA = 0.039. While these models do not provide acceptable fit by the conventional criterion standards, their statistics are fairly decent considering the current restricted sample (both in terms of size and variability) and the BRIEF-A was originally modeled in a two-factor, nine-scale fashion (Roth et al., 2005). The convergence of these models provided the support needed to continue with the final set of analyses.

Using a SEM approach to compute the correlations between the latent factors of the four factors of the BASC executive functions screener and the four target factors of the BRIEF-A revealed high correlations between focal-latent variables and target-criterion-latent variables. Specifically, the correlations were as follows: Problem Solving with Plan/Organize, r = .896; Attentional Control with Working Memory, r = .850; Behavioral Control with Inhibit, r = .779; and Emotional Control with Emotional Control, r = .768 (all p = .000; depicted in Figure 4).

Figure 4.

Comparison of the BASC-2-SRP-COL four-factor model to the selected corresponding target BRIEF-A scales.

To assess the specificity of these correlations, the four factors of the BASC executive functions screener were modeled simultaneously with the nine factors of the BRIEF-A (Figure S1). All latent mean correlations are shown in Table S6. Here, it is important to note that the target mean correlations shown in Figure 4 and Table S6 differ slightly as a result of increasing model complexity by moving from a four-factor BASC model versus a four-factor BRIEF-A model to a four-factor BASC model versus a nine-factor BRIEF-A model. The latent mean correlations between Problem Solving and its eight nontarget BRIEF-A scales were all significant (from p = .000 to .001), yet lower than its target latent mean correlation (from r = .289 to .789). The same trend was observed for Attentional Control (from r = .327 to .666, p = .000 to .001). All latent mean correlations between Attentional Control and its nontarget BRIEF-A scales were lower than the target correlation (from r = .124 to .831), and all correlations, with the exception of one, were significant (all p = .000 except Organization of Materials, p = .267). Similarly, all nontarget latent mean correlations for Emotional Control, except one, were significant (from p = .000 to .004 except Organization of Materials, p = .727). With the exception of Shift (r = .769), all nontarget latent mean correlations were lower than the target correlation (all others from r = −.036 to .446).

Summary and Discussion

In Study 2, we successfully replicated the derived screener from Study 1 using an independent sample. The statistical properties of the screener in Study 2 were similar to those in Study 1 and the results appear to provide further support for the robust psychometric properties of the four-factor model. While the reliability coefficients for Study 2 were lower than those observed in Study 1, this was expected considering the significant drop in sample size between the derivation and replication samples (568 fewer individuals). Similar to the derivation analyses of Study 1, in Study 2, all selected fit indicators improved with each new model (Models 1-4) and again, Model 4 (the four-factor model) was the only to have an acceptable CFI, TLI, and RMSEA values. Comparing Model 4 between the two studies, we found similar fit indices in Study 2 (comparing Model 4 between Tables 3 and 5: ΔCFI = +0.001, ΔTLI = +0.001, ΔRMSEA = −0.016). Furthermore, in comparison with the factor loadings calculated for the derivation sample (Study 1), most of those calculated for the replication sample (Study 2) were slightly lower, but still considered strong. Due to the uneven sex distribution in the replication sample used in Study 2, we were unable to replicate the measurement invariance of sex testing conducted in Study 1.

In Study 2, we also conducted a series of analyses that provided support for the convergent validity of the BASC executive functions screener. First, simple correlations between the BASC factors and their corresponding BRIEF-A scales were modest to strong and higher than the correlations between the BASC factors and their noncorresponding BRIEF-A scales. These observed target correlations were lower than the anticipated correlations, which were based on other studies examining the convergent validity of well-known executive functions rating scales (Delis, 2012; Gioia, Isquith, Guy, & Kenworthy, 2000; Grace & Malloy, 2001; Roth et al., 2005). The specificity of each target correlation was largely supported by the lower nontarget correlations. Second, the observed patterns of SEM correlations were consistent with the findings for the simple correlations, although the actual correlations were more robust and generally more consistent with the anticipated correlations. In line with our hypotheses, the results from both sets of analyses support the convergent validity of the derived BASC-2-SRP-COL executive functions screener with the BRIEF-A.

Discussion

While early adulthood has been established as a critical period of neural development for executive functions, limited data on its behavioral correlates are available (Taylor, Barker, Heavey, & McHale, 2013) and, to our knowledge, no executive functions behavior rating scale has been developed to specifically measure executive functions during this developmental period. To address this gap in the literature, we applied a series of steps to develop (Study 1) and independently replicate (Study 2) an executive functions screener from the BASC-2-SRP-COL (Reynolds & Kamphaus, 2004). This instrument was conceptually and methodologically based on the previously derived BASC-TRS-C executive functions screener for children (Garcia-Barrera et al., 2011). Using a four-factor model of executive functions, the screener was developed to capture behaviors related to planning and goal initiation (Problem Solving); focusing, sustaining, and shifting attention (Attentional Control); behavioral self-regulation (Behavioral Control); and emotional self-regulation (Emotional Control). A total of 23 items from the BASC-2-SRP-COL were isolated as indicators of these four components, serving as latent constructs of executive functions.

Results from the construct validity analyses, in which unidimensional and multidimensional models of the screener were tested using CFA, indicated that the four-factor model (Model 4) best explained the data and had significantly better fit than the other models. While all models converged normally, each subsequent multidimensional model showed substantial improvements in model fit (sequentially moving from Model 1 to Model 4) and the four-factor model was the only to demonstrate acceptable fit. These results indicate the BASC executive functions screener derivation process and the four-factor BASC executive functions screener (Garcia-Barrera et al., 2011) are replicable using the BASC-2-SRP-COL. Additionally, these results show that the screener captures the multidimensionality of executive functions, which is consistent with the developmental literature that has used a latent variable approach (Casey, Tottenham, Liston, & Durston, 2005; Huizinga, Dolan, & van der Molen, 2006; Lee, Bull, & Ho, 2013; Lehto, Juujärvi, Kooistra, & Pulkkinen, 2003; Willoughby, Wirth, Blair, & The Family Life Project Investigators, 2012).

Considering this developmental trajectory, it is interesting to note that while other executive functions rating scales typically have stronger fit indices than those reported here, their reported fit statistics are specific to different developmental contexts (e.g., Egeland & Fallmyr, 2010; Fournet et al., 2014; Garcia-Barrera et al., 2011, in children; Roth, Lance, Isquith, Fischer, & Griancola, 2013, in adults aged 18-90 years). Since this is the first study to examine the psychometric properties of an executive functions instrument targeted for the early-adult period (ages 18-25), the relatively lower fit indices reported here could be interpreted as reflecting the difficulty associated with attempting to measure characteristics of executive functions developmentally unique to this age group. Additionally, the lower interfactor correlations reported in this study may indicate more distinct executive functions components in young adults. It could also be that, when attempting to capture both the unity and diversity of executive functions, greater psychometric rigor can be achieved in samples of younger or older individuals, or in samples of wider age ranges that encapsulate the narrower early-adulthood period (e.g., the BRIEF-A, Roth et al., 2005; the FrSBe, Grace & Malloy, 2001). At the same time, the generally high loadings of the indicators on the latent factors and the acceptable model fit described here contrasts with Willoughby, Blair, Pek, and The Family Life Project Investigators’ (2013) discussion that single-factor models tend to provide the best relative fit to data in early childhood and the observed discrepancy between better global model fit and poorer quality measurement may not be specific to a particular age period.

When moving from a three-factor model to a four-factor model (in which Behavioral and Emotional Control are represented as distinct factors), the BASC-TRS-C executive functions screener demonstrated little difference in model fit (ΔCFI = −0.012; Garcia-Barrera et al., 2011), whereas the present study demonstrated a more substantial gain in fit (ΔCFI = −0.058 for the derivation sample, and ΔCFI = −0.100 for the replication sample). This comparison indicates that Behavioral Control and Emotional Control processes may fractionate more robustly in adolescence and emerging adulthood (i.e., between the age ranges of 6-11 years and 18-25 years). Components of executive functions have been shown to follow unique developmental trajectories (Anderson, Anderson, Jacobs, & Spencer Smith, 2008; Garcia-Barrera et al., 2013; Huizinga et al., 2006; Reynolds & MacNeill Horton, 2008; Romine & Reynolds, 2005). While research suggests that regulation of affectively charged executive processes (e.g., Emotional Control) emerges early in childhood, the development of these hot executive functions may lag behind others, such as those associated with abstract problem solving (Zelazo & Carlson, 2012).

As for the instrument as a whole, the obtained interfactor correlations (ranging from .251 to .629 for the derivation sample and .292 to .683 for the replication sample) were all lower than the absolute values of interfactor correlations on the BASC-TRS-C executive functions screener (ranging from .502 to .875; Garcia-Barrera et al., 2011). These findings support both the scale’s internal convergent validity (measuring four constructs unified by the same latent construct of executive functions) and divergent validity (each construct is unique and not better explained by another construct). Additionally, when considered in the context of the BASC-TRS-C screener, these findings are consistent with the developmental fractionation of executive functions theory, which would predict lower correlations between components of executive functions over time as their development progresses into early in adulthood. Furthermore, experimental studies examining the structure of executive functions across different ages using factor-analytic approaches have reported lower intercorrelations with increasing age (Lee et al., 2013; Lehto et al., 2003).

In Study 1, we further examined the scale’s psychometric properties by performing a stepwise analysis of measurement invariance, including configural, weak, and strong invariance testing across males and females. Results did not demonstrate significant factor structure differences across sex, and this was consistent with prior research demonstrating the measurement invariance across sex of a similar instrument for children (Garcia-Barrera et al., 2011, 2013). Admittedly, one potential limitation of our research was the uneven sex distribution of the replication sample used in Study 2, which prevented us from being able to replicate the invariance analyses done in Study 1. Our results appear to be less consistent, however, with other executive functions ratings scales that tend to show more sex differences in childhood and adolescence and fewer sex differences in adulthood (e.g., BDEFS and BDEFS-CA, Barkley, 2015; BRIEF, Roth et al., 2015). Research following individuals’ executive behavior self-ratings from childhood (age 11) to late adolescence (age 19) indicated that female executive behavior matures earlier than that for males; however, overall maturation trajectory and sex differences were also unique to each subcomponent of executive functions (Boelema et al., 2014). On the other hand, research examining sex-related differences in performance-based tasks of executive functioning in individuals aged 8 to 21 years indicated that male performance matches female performance by early adulthood (Roalf et al., 2014). While differences in maturational trajectory of executive functions components may explain some sex differences in the literature, other factors, such as prior educational attainment (Baars, Bijvank, Tonnaer, & Jolles, 2015) and methodological differences in the examination of specific executive functions subcomponents (e.g., behavioral control, emotional control type components, which appear to potentially have the greatest sex differences) may also influence results. Although the present study was limited in its investigation of potential sex differences in executive functions, our findings in comparison with the extant literature suggests that future research may benefit from such continued inquiry.

Overall, our analyses across both studies provided support that the derived instrument has acceptable psychometrics properties. Perhaps one of the strongest aspects to our study was the additional effort to replicate the derived screener in an independent sample. The substantial similarity between the derivation and replication analyses further substantiated the evidence for the psychometric integrity of the scale. Given the apparent stability of the four-factor model across different samples, sexes, and age ranges, the BASC executive functions screener may serve as a useful instrument for further studies investigating the developmental trajectories of its constructs—particularly Behavioral Control and Emotional Control.

While it is plausible that the acceptable, but not optimal, fit indices reported in this study may be indicative of challenges associated with measuring a complex and fractionating system of functions at or near a time of greater fractionation and developmental maturation, an alternative explanation is that the nonoptimal fit indices are due to shortcomings of the instrument itself. For instance, some of the internal consistency reliabilities (ranging from .658 to .801 for the derivation sample, and .647 to .664 for the replication sample) fell just above the acceptable range, and higher values would be ideal (Naglieri & Goldstein, 2015a). Our reliabilities and factor loadings are also lower than those obtained in studies with other versions of the BASC executive functions screener (e.g., α = .805 to .890 for the original screener derivation; Garcia-Barrera et al., 2011, 2013). Furthermore, we noted several differences between the BASC-2-SRP-COL and the BASC-TRS-C that made it challenging to develop an executive functions screener capturing the breadth and depth of the targeted constructs. These challenges included the addition of true/false items, the removal of items previously included in the BASC-TRS-C executive functions screener, and the inclusion of items that assess developmentally unique aspects of executive functions not previously addressed. Ultimately, many of these unique items were screened out due to their weak psychometric properties and due to their infrequency, as underrepresentation of particular aspects of a construct reduce the quality of the measure. Additionally, the small number of indicators selected for each factor (ranging from 5 to 6 for each factor) also likely contributes to the low internal consistency reliabilities reported in the current study.

Moreover, most rating scale standardization studies include clinical samples, and the exclusion of clinical groups from the present study may have contributed to the low internal consistency estimates. Internal consistencies of other executive functions rating scales have been shown to improve in mixed clinical/healthy adult samples (as supported in the present study with higher internal consistencies with the larger and more diverse derivation sample than the replication sample) and are higher in informant report samples than in self-report samples (i.e., BRIEF-A, Roth et al., 2005; Frontal Systems Behavior Scale, Grace & Malloy, 2001). Yet another factor to consider is that younger adults have been shown to report more executive functions difficulties than older adults (i.e., individuals aged 18-29 years on the BRIEF-A and aged 18-39 years on the Frontal Systems Behavior Scale; Roth et al., 2005 and Grace & Malloy, 2001, respectively).

The final objective of this article was to investigate the convergent validity of the derived BASC executive functions screener by comparing it with the BRIEF-A (Roth et al., 2005)—one of the most popular instruments for the behavioral assessment of executive functions in individuals older than the age of 18 years (Roth et al., 2015). Unfortunately, the widespread variability in the conceptualization and measurement of executive functions has made it particularly difficult to establish convergent validity for all executive functions rating scales (Naglieri & Goldstein, 2015a), and attempts to do so have generally been restricted to simple correlations (e.g., Delis, 2012; Gioia et al., 2000; Grace & Malloy, 2001; Roth et al., 2005).

Results from the first set of convergent validity analyses, which examined correlations between the BASC factors and selected BRIEF-A scales, demonstrated significant correlations generally in the moderate range. Since previous studies examining the correlations between established behavior rating scales for executive functions have reported moderate-to-high correlations, we hypothesized that a similar pattern of correlations would be observed in the present study. Although the observed correlations were lower than expected, they were still quite strong considering the differences between the instruments. While the isolated BRIEF-A scales were the closest in content and conceptualization to the BASC factors, it potentially would have been more realistic to expect moderate correlations that reflect both the similarities between, and differences in, content and conceptualization of the measures. Furthermore, convergent validity analyses traditionally compare composite scores, but our convergent validity analyses were based on direct factor-to-scale comparisons; thus, our originally hypothesized correlations were likely too ambitious. Finally, these comparisons may have also been complicated by more general scale differences. For instance, the BRIEF-A exclusively contains negatively worded executive functions items measured on a 3-point Likert-type scale, while the BASC screener contains items imbedded in a much broader behavior rating scale composed of positively and negatively worded items scored on dichotomous and 4-point Likert-type scales.

While examining patterns of correlation coefficients is one of the most common approaches to evaluating validity, it is limited by several factors. First, this approach is highly subjective; the observed magnitudes of correlations and their statistical significance are essentially “eyeballed” and judged on the degree to which they conceptually fit with the hypothesized patterns of association (Furr & Bacharach, 2014). Additionally, the correlation between two measures reflects both the extent to which they share an underlying construct and the extent to which they covary on other factors (e.g., measurement error, reliability; Campbell & Fiske, 1959). In an effort to address the weakness of correlation analysis and its typical application to convergent validity, we used SEM to examine relationships between the latent means of the BASC factors and select BRIEF-A scales and to disattenuate correlations for the effects of potential sources of error (Hoyle & Smith, 1994). Interestingly, this approach yielded high and significant correlations for each factor/scale pair, suggesting that the BASC executive functions screener has strong convergent validity with the BRIEF-A. The success of this approach suggests that a similar methodology may be helpful in addressing the challenges associated with establishing convergent validity for other executive functions rating scales. The approach could also be used to examine the convergent validity of executive functions rating scales in the context of multitrait–multimethod relationships (e.g., with laboratory-based tests of executive functions, or with interview-based instruments such as the Frontal Behavioral Inventory; Kertesz, Davidson, & Fox, 1997), and it has been shown to be a successful approach in other fields (e.g., see Bryant, King, & Smart, 2007).

Overall, the selection of target scales, as well as the specificity of the correlations between the BASC factors and their target scales, were supported by the data. Nearly all target correlations and latent mean correlations were higher than their nontarget counterparts. Interestingly, the latent mean correlation between BASC Emotional Control and the nontarget Shift scale was slightly higher than the target correlation (r = .769 vs. r = .768). Although this finding was not anticipated, it was not surprising considering the Shift scale contains several items that relate closely with those of the BASC Emotional Control factor (e.g., After having a problem, I don’t get over it easily). This suggests that the ability to shift focus from one mindset to another may be closely tied to emotional control processes. Alternatively, it suggests that the Shift scale on the BRIEF-A may be more measuring shifting in an emotional context (i.e., emotional shifting).

The significant correlations across nearly all of the BRIEF-A scales indicate that the BASC factors have a cohesive structure representing executive behavior. Additionally, these correlations are consistent with correlations between the BRIEF-A scales and the Dysexecutive Questionnaire (Wilson et al., 1996), which were all significant and ranged from .38 to .80 (Roth et al., 2005). Only the correlations between the nontarget scale Organization of Materials and some of the BASC factors were not significant. This is consistent with the finding by Roth et al. (2005) that Organization of Materials had the lowest correlation with the Dysexecutive Questionnaire (r = .38). The pattern of strong correlations between the BASC factors and the BRIEF-A scales represent evidence for convergence between the instruments and specificity of the BASC factors. Overall, this provides strong evidence of the validity of the BASC four-factor executive functions screener.

In summary, in the two studies presented here, we successfully derived and replicated an executive functions rating scale for young adults from the BASC-2-SRP-COL. This article is also the first to examine the convergent validity of the BASC executive functions screener. Analyses of the instrument’s psychometric properties support its construct validity and measurement invariance across sex, indicate that it is acceptably reliable, and suggest that this measure has strong convergent validity with the BRIEF-A. It is important to note that a number of factors limit the generalizability of the findings of this study, as well as the implementation of the BASC-2-SRP-COL executive functions screener in the future. First, further analyses of the screener’s convergent validity should be conducted using other executive functions rating scales, as well as traditional performance-based tests. Although correlations between executive functions rating scales and performance-based measures have been low (Silver, 2014), use of latent variable approaches similar to those described here and elsewhere (e.g., Bryant et al., 2007; Miyake et al., 2000) may contribute to improved understanding of the development and measurement of executive functions. Deriving norms and a composite measure for the screener would also facilitate additional convergent validity analyses and making comparisons between instruments. Finally, analyses examining the clinical validity of the screener would improve its assessment utility and advance our understanding of the development of executive functions in early adulthood, as well as the emergence of related executive dysfunction and psychopathologies. Overall, these findings support the model serving as the foundation of Garcia-Barrera et al.’s (2011) BASC executive functions screener. These findings also contribute to a growing body of research, which demonstrates that this instrument may inform our understanding of the development of executive functions throughout childhood and into early adulthood.

Footnotes

Authors’ Note

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: We acknowledge that Dr. Garcia-Barrera, coauthor in this article, previously received consultant fees from Pearson Assessment (publisher of the BASC—an instrument included in this article).

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Emily C. Duggan, MSc, Vanier Scholar gratefully thanks the Vanier Canada Graduate Scholarship and the Natural Sciences and Engineering Research Council of Canada for their financial support of her research.

References

Anderson

P. J.

Jacobs

Spencer Smith

(2008). Development and assessment of executive function: From preschool to adolescence. In Anderson

Jacobs

Anderson

P. J.

(Eds.), Executive functions and the frontal lobes: A lifespan perspective (pp. 123-154). New York, NY: Taylor & Francis.

Baars

M. A. E.

Bijvank

M. N.

Tonnaer

G. H.

Jolles

(2015). Self-report measures of executive functioning are a determinant of academic performance in first-year students at a university of applied sciences. Frontiers in Psychology, 6, 1131. doi:10.3389/fpsyg.2015.01131

Barkley

R. A.

(2015). Assessment of executive functioning using the Barkley Deficits in Executive Functioning Scales. In Naglieri

J. A.

Goldstein

(Eds.), Handbook of executive functioning (pp. 245-264). New York, NY: Springer.

Blakemore

S. J.

Choudhury

(2006). Development of the adolescent brain: Implications for executive function and social cognition. Journal of Child Psychology and Psychiatry, and Allied Disciplines, 47, 296-312. doi:10.1111/j.1469-7610.2006.01611.x

Boelema

S. R.

Harakeh

Ormel

Hartman

C. A.

Vollebergh

W. A. M.

van Zandvoort

M. J. E.

(2014). Executive functioning shows differential maturation from early to late adolescence: Longitudinal findings from a TRAILS study. Neuropsychology, 28, 177-187. doi:10.1037/neu0000049

Bryant

F. B.

King

S. P.

Smart

C. M.

(2007). Multivariate statistical strategies for construct validation in positive psychology. In Ong

A. D.

Van Dulmen

M. H. M.

(Eds.), Oxford handbook of methods in positive psychology (pp. 61-82). New York, NY: Oxford University Press.

Burgess

P. W.

Alderman

Forbes

Costello

Coates

L. M-A.

Dawson

D. R.

. . . Channon

(2006). The case for the development and use of “ecologically valid” measures of executive function in experimental and clinical neuropsychology. Journal of the International Neuropsychological Society, 12, 194-209. doi:10.1017/S1355617706060310

Campbell

D. T.

Fiske

D. W.

(1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105. doi:10.1037/h0046016

Casey

B. J.

Tottenham

Liston

Durston

(2005). Imaging the developing brain: What have we learned about cognitive development? Trends in Cognitive Sciences, 9, 104-110. doi:10.1016/j.tics.2005.01.011

10.

Cheung

G. W.

Rensvold

R. B.

(2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9, 233-255. doi:10.1207/S15328007SEM0902_5

11.

Crews

Hodge

(2007). Adolescent cortical development: A critical period of vulnerability for addiction. Pharmacology Biochemistry and Behavior, 86, 189-199. doi:10.1016/j.pbb.2006.12.001

12.

Cronbach

L. J.

Meehl

P. E.

(1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281-302. doi:10.1037/h0040957

13.

Delis

(2012). Delis-Rating of Executive Function (D-REF). Bloomington, MN: Pearson.

14.

Devine

R. T.

Hughes

(2014). Relations between false belief understanding and executive function in early childhood: A meta-analysis. Child Development, 85, 1777-1794. doi:10.1111/cdev.12237

15.

Diamond

(2013). Executive function. Annual Review of Psychology, 64, 135-168.

16.

Duggan

E. C.

Garcia-Barrera

M. A.

(2015). Intelligence and executive functioning. In Naglieri

J. A.

Goldstein

(Eds.), Handbook of intelligence (pp. 435-458). New York, NY: Springer. doi:10.1007/978-1-4939-1562-0_27

17.

Egeland

Fallmyr

Ø.

(2010). Confirmatory Factor Analysis of the Behavior Rating Inventory of Executive Function (BRIEF): Support for a distinction between Emotional and Behavioral Regulation. Child Neuropsychology: A Journal on Normal and Abnormal Development in Childhood and Adolescence, 16, 326-337. doi:10.1080/09297041003601462

18.

Fournet

Roulin

J.-L.

Monnier

Atzeni

Cosnefroy

Le Gall

Roy

(2014). Multigroup confirmatory factor analysis and structural invariance with age of the Behavior Rating Inventory of Executive Function (BRIEF)-French version. Child Neuropsychology: A Journal on Normal and Abnormal Development in Childhood and Adolescence, 21, 379-398. doi:10.1080/09297049.2014.906569

19.

Frazer

(2012). The integrative neuropsychological theory of executive-related abilities and component transactions (INTERACT): A novel validation study (Unpublished doctoral dissertation). University of Victoria, Victoria, British Colombia, Canada.

20.

Fuglestad

A. J.

Whitely

M. L.

Carison

S. M.

Boys

C. J.

Eckerle

J. K.

Fink

B. A.

Wozniak

(2015). Executive functioning deficits in preschool children with Fetal Alcohol Spectrum Disorders. Child Neuropsychology, 21, 716-731. doi:10.1080/09297049.2014.933792

21.

Furr

R. M.

Bacharach

V. R.

(2014). Psychometrics: An introduction (2nd ed.). Thousand Oaks, CA: Sage.

22.

Fuster

J. M.

(2000). Executive frontal functions. Experimental Brain Research, 133, 66-70. doi:10.1007/s002210000401

23.

Garcia-Barrera

M. A.

Kamphaus

R. W.

Bandalos

(2011). Theoretical and statistical derivation of a screener for the behavioral assessment of executive functions in children. Psychological Assessment, 23, 64-79. doi:10.1037/a0021097

24.

Garcia-Barrera

M. A.

Karr

J. E.

Duran

Direnfeld

Pineda

D. A.

(2015). Cross-cultural validation of a behavioral screener for executive functions: Guidelines for clinical use among Colombian children with and without ADHD. Psychological Assessment, 27, 1349-1363. doi:10.1037/pas0000117

25.

Garcia-Barrera

M. A.

Karr

J. E.

Kamphaus

R. W.

(2013). Longitudinal applications of a behavioral screener of executive functioning: Assessing factorial invariance and exploring latent growth. Psychological Assessment, 25, 1300-1313. doi:10.1037/a0034046

26.

Gioia

G. A.

Isquith

P. K.

(2004). Ecological assessment of executive function in traumatic brain injury. Developmental Neuropsychology, 25, 135-158. doi:10.1080/87565641.2004.9651925

27.

Gioia

G. A.

Isquith

P. K.

Guy

S. C.

Kenworthy

(2000). BRIEF: Behavior Rating Inventory of Executive Function. Lutz, FL: Psychological Assessment Resources.

28.

Gioia

G. A.

Isquith

P. K.

Kenealy

L. E.

(2008). Assessment of behavioral aspects of executive function. In Anderson

Jacobs

Anderson

P. J.

(Eds.), Executive functions and the frontal lobes: A lifespan perspective (pp. 179-202). New York, NY: Taylor & Francis.

29.

Gioia

Kenworthy

Isquith

P. K.

(2010). Executive function in the real world: BRIEF lesions from Mark Ylvisaker. Journal of Head Trauma Rehabilitation, 25, 433-439. doi:10.1097/HTR.0b013e3181fbc272

30.

Goldstein

Naglieri

J. A.

Princiotta

Otero

T. M.

(2015). Introduction: A history of executive functioning as a theoretical and clinical construct. In Naglieri

J. A.

Goldstein

(Eds.), Handbook of executive functioning (pp. 3-12). New York, NY: Springer. doi:10.1007/978-1-4614-8106-5_1

31.

Grace

Malloy

P. F.

(2001). Frontal Systems Behavior Scale: Professional manual. Lutz, FL: Psychological Assessment Resources.

32.

Hair

J. F.

Black

Babin

Anderson

R. E.

(2010). Multivariate data analysis. Upper Saddle River, NJ: Prentice Hall.

33.

Hersen

Beidel

D. C.

(Eds.). (2012). Adult psychopathology and diagnosis (6th ed.). Hoboken, NJ: Wiley.

34.

Hoyle

R. H.

Smith

G. T.

(1994). Formulating clinical research hypotheses as structural equation models: A conceptual overview. Journal of Consulting and Clinical Psychology, 6, 429-440.

35.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55. doi:10.1080/10705519909540118

36.

Huizinga

Dolan

C. V.

van der Molen

M. W.

(2006). Age-related change in executive function: Developmental trends and a latent variable analysis. Neuropsychologia, 44, 2017-2036. doi:10.1016/j.neuropsychologia.2006.01.010

37.

Isquith

P. K.

Roth

R. M.

Gioia

(2013). Contribution of rating scales to the assessment of executive functions. Applied Neuropsychology: Child, 2, 1-8. doi:10.1080/21622965.2013.748389

38.

Jackson

D. L.

Gillaspy

J. A.

Purc-Stephenson

(2009). Reporting practices in confirmatory factor analysis: An overview and some recommendations. Psychological Methods, 14, 6-23. doi:10.1037/a0014694

39.

Karr

J. E.

Garcia-Barrera

M. A.

Kerns

K. A.

Mueller

Baron

I. S.

Litman

(2013, April). Derivation of a behavioral executive functioning screener in preschoolers using a latent variable approach. Paper presented at the biennial meeting of the Society for Research in Child Development, Seattle, WA.

40.

Kertesz

Davidson

Fox

(1997). Frontal Behavior Inventory: Diagnostic criteria for the frontal lobe dementia. Canadian Journal of Neurological Science, 24, 29-36.

41.

Kline

(2013). Handbook of psychological testing (2nd ed.). London, England: Routledge.

42.

Lee

Bull

R. M.

(2013). Developmental changes in executive functioning. Child Development, 84, 1933-1953. doi:10.1111/cdev.12096

43.

Lehto

J. E.

Juujärvi

Kooistra

Pulkkinen

(2003). Dimensions of executive functioning: Evidence from children. British Journal of Developmental Psychology, 21, 59-80. doi:10.1348/026151003321164627

44.

Lezak

M. D.

Howieson

D. B.

Bigler

E. D.

Tranel

(2012). Neuropsychological assessment (5th ed.). New York, NY: Oxford University Press.

45.

Manchester

Priestley

Jackson

(2004). The assessment of executive functions: Coming out of the office. Brain Injury, 18, 1067-1081. doi:10.1080/02699050410001672387

46.

Marsh

H. W.

Balla

J. R.

McDonald

R. P.

(1988). Goodness-of-fit indexes in confirmatory factor analysis: The effect of sample size. Psychological Bulletin, 103, 391-410. doi:10.1037/0033-2909.103.3.391

47.

McAuley

Chen

Goos

Schachar

Crosbie

(2010). Is the Behavior Rating Inventory of Executive Function more strongly associated with measures of impairment or executive function? Journal of the International Neuropsychological Society, 16, 495-505. doi:10.1017/S1355617710000093

48.

Meltzer

Krishnan

(2007). Executive function difficulties and learning disabilities: Understandings and misunderstandings. In Melzter

(Ed.), Executive function in education: From theory to practice (pp. 77-105). New York, NY: Guilford Press.

49.

Miller

E. K.

Cohen

J. D.

(2001). An integrative theory of prefrontal cortex function. Annual Review of Neuroscience, 24, 167-202. doi:10.1146/annurev.neuro.24.1.167

50.

Miyake

Friedman

N. P.

(2012). The nature and organization of individual differences in executive functions: Four general conclusions. Current Directions in Psychological Science, 21, 8-14. doi:10.1177/0963721411429458

51.

Miyake

Friedman

N. P.

Emerson

M. J.

Witzki

A. H.

Howerter

Wager

T. D.

(2000). The unity and diversity of executive functions and their contributions to complex “frontal lobe” tasks: A latent variable analysis. Cognitive Psychology, 41(1), 49-100. doi:10.1006/cogp.1999.0734

52.

Müller

Kerns

(2015). The development of executive function. In Liben

L. S.

Müller

Lerner

R. M.

(Eds.), Handbook of child psychology and developmental science, Vol. 2: Cognitive processes (7th ed., pp. 571-623). Hoboken, NJ: Wiley.

53.

Muthén

L. K.

Muthén

B. O.

(1998-2012). Mplus user’s guide (7th ed.). Los Angeles, CA: Author.

54.

Naglieri

J. A.

Goldstein

(2015a). Assessment of executive function using rating scales: Psychometric considerations. In Naglieri

J. A.

Goldstein

(Eds.), Handbook of executive functioning (pp. 159-171). New York, NY: Springer.

55.

Naglieri

J. A.

Goldstein

(Eds.). (2015b). Handbook of executive functioning. New York, NY: Springer.

56.

Otero

T. M.

Barker

L. A.

(2015). The frontal lobes and executive functioning. In Naglieri

J. A.

Goldstein

(Eds.), Handbook of executive functioning (pp. 29-44). New York, NY: Springer.

57.

Pelicano

(2010). The development of core cognitive skills in autism: A 3-year prospective study. Child Development, 81, 1400-1416.

58.

Pennington

B. F.

Ozonoff

(1996). Executive functions and developmental psychopathology. Journal of Child Psychology and Psychiatry, and Allied Disciplines, 37, 51-87. doi:10.1111/j.1469-7610.1996.tb01380.x

59.

Reynolds

C. R.

Kamphaus

R. W.

(1992). Behavior Assessment System for Children (BASC). Circle Pines, MN: AGS.

60.

Reynolds

C. R.

Kamphaus

R. W.

(2004). Behavior Assessment System for Children (2nd ed.). Bloomington, MN: Pearson. doi:10.1002/9780470479216.corpsy0114

61.

Reynolds

C. R.

Kamphaus

R. W.

(2015). Behavior Assessment System for Children (3rd ed.). Bloomington, MN: Pearson.

62.

Reynolds

C. R.

MacNeill Horton

(2008). Assessing executive functions: A life-span perspective. Psychology in the Schools, 45, 875-892. doi:10.1002/pits.20332

63.

Roalf

D. R.

Gur

R. E.

Ruparel

Calkins

M. E.

Satterthwaite

T. D.

. . . Gur

R. C.

(2014). Within-individual variability in neurocognitive performance: Age- and sex-related differences in children and youths from ages 8 to 21. Neuropsychology, 28, 506-518. doi:10.1037/neu0000067

64.

Romine

C. B.

Reynolds

C. R.

(2005). A model of the development of frontal lobe functioning: Findings from a meta-analysis. Applied Neuropsychology, 12, 190-201. doi:10.1207/s15324826an1204_2

65.

Roth

R. M.

Isquith

P. K.

Gioia

G. A.

(2005). Behavior Rating Inventory of Executive Function–Adult version (BRIEF-A). Lutz, FL: Psychological Assessment Resources.

66.

Roth

R. M.

Isquith

P. K.

Gioia

G. A.

(2015). Assessment of executive functioning using the Behavior Rating Inventory of Executive Function (BRIEF). In Naglieri

J. A.

Goldstein

(Eds.), Handbook of executive functioning (pp. 301-332). New York, NY: Springer.

67.

Roth

R. M.

Lance

C. E.

Isquith

P. K.

Fischer

A. S.

Griancola

P. R.

(2013). Confirmatory factor analysis of the Behavior Rating Inventory of Executive Function-Adult Version in healthy adults and application to attention-deficit/hyperactivity disorder. Archives of Clinical Neuropsychology, 28, 425-434. doi:10.1093/arclin/act031

68.

Sadeh

S. S.

Burns

M. K.

Sullivan

A. L.

(2012). Examining an executive function rating scale as a predictor of achievement in children at risk for behavior problems. School Psychology Quarterly, 27, 236-246. doi:10.1037/spq0000012

69.

Schermelleh-Engel

Moosbrugger

Müller

(2003). Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Methods of Psychological Research Online, 8, 23-74.

70.

Schoemaker

Mulder

Devović

Matthys

(2013). Executive functions in preschool children with externalizing behavior problems: A meta-analysis. Journal of Abnormal Child Psychology, 41, 457-471.

71.

Shallice

Burgess

P. W.

(1991). Deficits in strategy application following frontal lobe damage in man. Brain, 114, 727-741.

72.

Silver

C. H.

(2014). Sources of data about children’s executive functioning: Review and commentary. Child Neuropsychology: A Journal on Normal and Abnormal Development in Childhood and Adolescence, 20, 1-13. doi:10.1080/09297049.2012.727793

73.

Strauss

Sherman

E. M. S.

Spreen

(2006). A compendium of neuropsychological tests (3rd ed.). New York, NY: Oxford University Press.

74.

Taylor

S. J.

Barker

L. A.

Heavey

McHale

(2013). Typical developmental trajectory of social and executive functions in late adolescence and early adulthood. Developmental Psychology, 49, 1253-1265.

75.

Toplak

M. E.

West

R. F.

Stanovich

K. E.

(2013). Practitioner review: Do performance-based measures and ratings of executive function assess the same construct? Journal of Child Psychology and Psychiatry, and Allied Disciplines, 54, 131-143. doi:10.1111/jcpp.12001

76.

Torralva

Roca

Gleichgerrcht

Bekinschtein

Manes

(2009). A neuropsychological battery to detect specific executive and social cognitive impairments in early frontotemporal dementia. Brain, 132, 1299-1309. doi:10.1093/brain/awp041

77.

Willoughby

M. T.

Blair

C. B.

Pek

The Family Life Project Investigators. (2013). Measuring executive function in early childhood: A focus on maximal reliability and the derivation of short forms. Psychological Assessment, 25, 664-670. doi:10.1037/a0031747

78.

Willoughby

M. T.

Wirth

R. J.

Blair

C. B.

The Family Life Project Investigators. (2012). Executive function in early childhood: Longitudinal measurement invariance and developmental change. Psychological Assessment, 24, 418-431. doi:10.1037/a0025779

79.

Wilson

B. A.

Alderman

Burgess

P. W.

Emslie

Evans

J. J.

(1996). Behavioural Assessment of the Dysexecutive Syndrome (BADS). Suffolk, England: Thames Valley Test.

80.

C. Y.

(2002). Evaluating cutoff criteria of model fit indices for latent variable models with binary and continues outcomes (Doctoral dissertation). University of California, Los Angeles.

81.

Zelazo

P. D.

Carlson

S. M.

(2012). Hot and cool executive function in childhood and adolescence: Development and plasticity. Child Development Perspectives, 6, 354-360. doi:10.1111/j.1750-8606.2012.00246.x