Using the Indonesian Computer-Based Game Prototype as a Computer-Based Game Inventory for Executive Function in Children and Adolescents: A Confirmatory Factor Analysis and Concurrent Validity Study

Abstract

Executive function influences children’s learning abilities and organizes their cognitive processes, behaviors, and emotions. This cross-sectional study examined whether an Indonesian Computer-Based Game (ICbG) prototype could be used as a Computer-Based Game Inventory for Executive Function (CGIEF) in children and adolescents. The study was conducted with 200 children, adolescents, and their parents. The parents completed the Behavior Rating Inventory of Executive Functioning (BRIEF) questionnaire, and the children and adolescents completed the CGIEF. Confirmatory factor analysis (CFA) and structural equation modeling (SEM) were performed using LISREL Version 8.80. The construct of CGIEF was valid/fit with normal theory-weighted least squares = 15.75 (p > .05). SEM analysis showed that the theoretical construct of the CGIEF was a valid predictor of executive function. The critical t value of the pathway was 2.45, and normal theory-weighted least squares was 5.74 (p > .05). The construct reliability (CR) for CGIEF was 0.91. Concurrent validity was assessed using the Bland–Altman plot, and the coefficient of repeatability (bias/mean) was nearly zero between the t scores of total executive functions of the CGIEF and BRIEF. This preliminary study showed that the CGIEF can be useful as a screening tool for executive dysfunction, metacognitive deficits, and behavioral regulation problems among children and adolescents in clinical samples.

Keywords

children adolescents CGIEF Indonesian Computer-Based Game construct validity executive function Indonesia

Executive function is a high-level executive process that controls and coordinates other cognitive abilities and behaviors (Gilbert & Burgess, 2008). Moreover, executive function is a “top-down control process” of human behavior that exerts “supervisory control” and includes abilities such as initiation, planning, and decision-making (Diamond, 2013; Kusi-Mensah et al., 2022; Uljarević et al., 2023). In other words, executive function helps one complete a task or perform certain mental activities, including the organization of thoughts and behavior, for optimal mental flexibility (such as problem analysis and solving), selective focus on certain activities, and regulate thoughts and working memory, emotional responses, and behaviors to achieve a predetermined goal (Blair, 2016; Diamond, 2013; Hartung et al., 2020). Optimal executive function implies that children and adolescents can make and execute plans, think, focus, remember, and multitask. Thus, executive function is an important predictor of a child’s learning ability, and evaluating their executive function is as important as evaluating the intelligence quotient (Colom et al., 2008; Grissom et al., 2015; Nouchi et al., 2013; Orban et al., 2014).

Prefrontal cortex is crucial for executive function, responsible for organizing, facilitating, and coordinating tasks and activities inside the cognitive system. It manages information in a multimodal temporary storage system, that is, the speech-based phonological loop, visual sketchpad, and episodic buffer (Swanson, 2015). The speech-based phonological loop is responsible for the temporary storage of verbal information, whereas the visual sketchpad is responsible for visual-spatial information storage over brief periods and plays an important role in the generation and manipulation of mental images. The episodic buffer incorporates information from several sources to establish an integrated memory (Diamond, 2013; Henry, 2010; Vandenbroucke et al., 2017; S. Wang et al., 2015).

The learning capacity of children and adolescents is strongly associated with executive function (Gunzenhauser & Nückles, 2021). Executive function is essential for children and adolescents to have adequate, purposeful, coordinated, focused, and self-regulated behaviors for optimal academic achievement, such as in the literature, science, and mathematics learning potential (Gunzenhauser & Nückles, 2021; Hofmann et al., 2012). Wiguna et al. (2012) found that 24.6% of elementary school children had learning difficulties and 8.04% had impaired working memory, which is one of the several components of executive function. Another study showed that children with working memory deficits experienced a higher risk of learning difficulties in mathematics (odds ratio [OR] = 3.373, 95% confidence interval = [1.349, 8.432]), science (OR = 3.075, 95% confidence interval = [1.034, 9.139]), and literature (OR = 4.935, 95% confidence interval = [1.779, 13.691]; Bangun et al., 2019). Thus, early detection of executive function in children and adolescents is important for predicting learning capacity (Albert et al., 2020; Gunzenhauser & Nückles, 2021; Moisala et al., 2018).

Currently, executive function assessment in children and adolescents is commonly performed using questionnaires, such as Behavior Rating Inventory of Executive Functioning (BRIEF), BRIEF-2, Comprehensive Executive Function Inventory (CEFI), Barkley Deficits in Executive Functioning Scale (BDEFS), and Executive Functioning Scale (EFS; Uljarević et al., 2023). However, this is mostly completed by parents, which is a useful method because they experience and perceive their child’s behavior and attitude over a long period across different situations. Nevertheless, it can lead to observational and response biases due to self-blame or discomfort completing a questionnaire, because parents consider that they should be responsible for their child’s condition, and in turn, it may cause in false-negative findings or underreported results (Hawk et al., 2013; Kronenberger et al., 2018). One commonly used questionnaire for executive function measurement in children and adolescents in Indonesia is the BRIEF which is completed by parents and has been validated in the Indonesian language but using it in clinical setting. Moreover, there is another version of BRIEF, the self-rated version (BRIEF-SR), for individuals aged 11 to 18 years (Cassidy, 2016). However, the BRIEF-SR has not yet been validated in Indonesia. The BRIEF comprises eight clinical domains that measure executive function: inhibition, shift, emotional control, monitoring, working memory, planning/organization, organization of materials, and initiation. The overall result was a global executive composite (GEC), which is the total score for executive function. The GEC is derived from two broad indices: the behavioral regulation index (BRI), which measures adolescents’ ability to regulate their behavior, and the metacognition index (MCI), which measures adolescents’ capabilities to solve problems with planning and organizational skills (Davidson et al., 2016).

In addition to questionnaires, there are other tools for assessing the executive function of children and adolescents in activity-based evaluations or game-like formats. These include the Delis-Kaplan Executive Function System (D-KEFS); the National Institutes of Health (NIH) Toolbox Cognitive Battery; Developmental Neuropsychological Assessment, Second Edition (NEPSY-2); Computerized Battery for Neuropsychological Evaluation of Children (BENCI); and Cambridge Neuropsychological Test Automated Battery (CANTAB); however, they have not been validated or adapted for Indonesian children and adolescents and are mostly commercial instruments that limit access to both clinical settings and research usage.

D-KEFS is used in several clinical settings to assess deficits in abstract, creative thinking, and various aspects of EF (e.g., visual attention, problem-solving, planning, creativity, flexibility). It consists of nine subtests: a trial-making test, verbal fluency test, design fluency test, color-word interference test, sorting test, 20-question test, word context test, tower test, and proverb test. The split-half reliability coefficient of D-KEFS was reported low for children and adolescents with the mean age of 8 to 15 years that was 0.61 for children and 0.60 for adolescents (range between 0.43 and 0.84; Pluck et al., 2021). Moreover, Crawford et al. (2008) explained the D-KEFS contrast scores with low reliability that ranged from 0.29 to 0.66 should not be used in neuropsychological decision-making. NIH Toolbox can be used to assess the neurological and behavioral functions of people aged 3 to 85 years (Denboer et al., 2014). It focuses on measuring cognitive, motor, emotional, and sensory functions and is administered using an iPad. Several aspects were measured using the cognitive portion of the toolbox, including executive function (dimensional change card sort test), attention (flanker inhibitory and attention test), episodic memory (picture sequence memory test), language (picture vocabulary test, oral reading recognition test), processing speed (pattern comparison processing speed test), and working memory (list sorting working memory test). The cognitive portion of the toolbox is further divided into two batteries: the toolbox cognition battery and the early childhood cognition battery. The developmental neuropsychological domain consisted of motor dexterity (Nine-Hole Pegboard Dexterity). Weintraub et al. (2013) examined the test–retest reliability of instruments in the NIH Toolbox Cognitive Battery and found that among children aged 3 to 15 years, executive function/Flanker intraclass correlation coefficient (ICC) was .95, executive function/dimensional change card sort ICC was .92, attention/Flanker ICC was .95, episodic memory/picture sequence memory test ICC was .76, working memory/list sorting ICC was .87, processing speed/pattern comparison ICC was .84, language/vocabulary ICC was .84, language/reading ICC was .99. The Computerized Battery for Neuropsychological Evaluation of Children (BENCI) is used to assess neuropsychological domains such as processing rate, visual–motor coordination, attention, memory, language, and executive function. Fernández-Alcántara et al. (2022) assessed the convergent validity of BENCI in children aged 9 to 11 years in Spain. The reaction times for trail making test A (rho = 0.330) and trail making test B (rho = 0.292) were positively associated with the visual–motor test results. The sustained attention test was positively correlated with Condition 3 of Stroop (rho = 0.274). Correct answers (CA) in the visual memory task positively correlated with the total token test score (rho = 0.314) and with Animal Tests A (rho = 0.336) and B (rho = 0.250). The verbal comprehension test was positively correlated with the token test scores (rho = 0.250) and between semantic fluency and token test scores (rho = 0.424). The CANTAB can measure several neurocognitive functions, including psychomotor and motor speed, reasoning and planning abilities, memory and attention, and frontal, temporal, and hippocampal dysfunctions (Syväoja et al., 2015). The test was administered using a computer or presented on a touchscreen. The tests were grouped into the following domains: induction, visual memory (pattern recognition memory and spatial recognition memory), executive function (spatial span, Stockings of Cambridge, and intra-extradimensional set shift), attention (reaction time and rapid visual information processing), verbal/semantic memory, decision-making and response control, and social cognition (Syväoja et al., 2015). In Indonesia, Hendrawan et al. (2020) developed a computer-based measurement for working memory (backward animal and shining star) for Indonesian children aged 4 to 6 years. This study also found that Shining Star may be applied as a standardized measure for working memory assessment in children at that age.

Although there are several kinds of questionnaires and activity-based assessment tools for executive function with their own benefits, strengths, and limitations, it is still important and useful to develop new executive function measurement tools, especially for children and adolescents, that are objective, fun, friendly, noncommercial, self-administered, and comprise local content and context. Therefore, computer-based game assessment is one of the several measurement tools that can be used for these purposes. Shute et al. (2017) explained that computer-based game measurement tools that involve gaming activities can be applied to assess the executive functions. The measurement usually gathers information from all the activities and tasks of the players in the game and other attributes achieved from player interactions (C. Wang & Huang, 2021). All information is then extrapolated as child/adolescent capacity, competence, and skills in certain aspects of functioning, such as cognition, executive function, mental capacity, and symptoms of illness (Shute et al., 2017). Computer-based game measurement tools have several advantages such as lowering anxiety levels, increasing motivation to perform game tasks, and improving the rate of participation (Killi & Ketamo, 2017; Mavridis & Thrasyvoulos, 2017). However, research on the psychometric properties of computer-based game measurement tools remains limited, particularly for executive function assessments.

In 2017, the Child and Adolescent Psychiatry Division, Department of Psychiatry, Faculty of Medicine, Universitas Indonesia, Dr. Cipto Mangunkusumo General Hospital, Jakarta, Indonesia, in collaboration with the Computer Science Department at Bina Nusantara International University, Jakarta, developed a computer-based game called the Indonesian Computer-Based Game (ICbG) for children. The study aimed to improve executive function in children with attention deficit hyperactivity disorder (ADHD). The research domain construct (RdoC) of the ICbG includes reward-related processing, inhibitory control, sustained attention, organizational skills, working memory, arousal regulation, and emotional regulation. The ICbG comprised four types of trials in one session: visual instruction without distractors, auditory instruction without distractors, visual instruction with distractors, and auditory instruction with distractors. During each trial, children and adolescents were instructed to deliver fruits of a certain color to houses of certain color. They are required to read or listen to instructions, control impulsive behaviors, sit still during the trials to organize and memorize the tasks, follow the game rules precisely, or risk failing to complete the tasks. During the trials, subjects were not only required to remember the specific colors of the fruits to be sent to colored houses but also to cover multiple aspects of executive function, such as planning (to overcome distractors that come in order), organizing (delivery of the fruit in order, as instructed), inhibiting impulses, regulating emotions, shifting attention between one instruction and another, and managing time to fulfill the maximum effort. When each trial was completed, the ICbG generated five variables as outcomes: (a) total number of fruits delivered correctly (FC), (b) total number of houses picked correctly (HC), (c) total number of fruits delivered incorrectly, (d) total number of houses picked incorrectly, and (e) time required to complete all instructions (both visual and auditory instructions). A feasibility study showed that after 20 trials, there were significant improvements in several domains of executive function based on the BRIEF measurement completed by parents, such as the general executive composite (GEC, Cohen’s d effects size = 0.82), MCI (Cohen’s d effect size = 1.02), and behavior regulation index (BRI, Cohen’s d effects size = 0.54; Wiguna et al., 2021). Hence, it was hypothesized that the ICbG could be applied as a Computer-Based Game Inventory for Executive Function (CGIEF).

The current study aimed to identify whether the ICbG can be used as a CGIEF by analyzing its construct validity and concurrent validity between the CGIEF and BRIEF parents rating (as a comparison measurement tool for executive function since it has been validated in Indonesia). The design of the CGIEF was based on the ICbG prototype developed in 2017. The primary research question addressed in this study is as follows: Can the CGIEF be used to screen for executive dysfunction and serve as a reliable executive function measurement tool in children and adolescents?

Method

This cross-sectional study was conducted at the Child and Adolescent Psychiatry Outpatient Clinic of Dr. Cipto Mangunkusumo General Hospital. Two hundred children and adolescents with parents who met the inclusion criteria participated in the study. The children and adolescents were selected through consecutive sampling. The inclusion criteria were as follows: (a) children and adolescents aged 6 to 17 years; (b) children or adolescents and at least one parent providing consent to participate in the study by completing and signing the informed consent and assent form; (c) children and adolescents willing to complete one session of the CGIEF measurement that consisted of four types of subtests; and (d) parents with a minimum educational background of junior high (middle school) and willing to complete the BRIEF questionnaire with an inconsistency level of less than nine and a negativity level of less than seven. The exclusion criteria were children and adolescents with (a) color blindness; (b) any severe or acute mental disorder such as an acute psychotic disorder, acute mood disorder, or moderate or severe intellectual disability; and (c) any physical condition that hindered them from completing the CGIEF measurement. All mental disorders excluded in the exclusion criteria were diagnosed based on the diagnostic criteria for mental disorders in the International Classification of Diseases 10 (ICD-10). Mental disorders were diagnosed by two child and adolescent psychiatrists at the Child and Adolescent Psychiatry Outpatient Clinic of the Cipto Mangunkusumo General Hospital, Jakarta, Indonesia. The study also included a drop out criterion: participants who were unable to complete all tasks during the CGIEF measurement were not included in the final analysis. However, all the participants completed the CGIEF; thus, there were no dropouts. The study protocol was approved by the Ethics Committee of the Faculty of Medicine at the University of Indonesia (KET-160/UN2.F1/ETIK/PPM.00.02/2022).

Instruments

The CGIEF measurements were performed using an ICbG prototype. Each subtest was performed under supervision. The CGIEF measurements were delivered using a personal computer or laptop with a browser that could run HTML5 and Java Script or in the form of execution (.exe) without a browser. The CGIEF uses a central processing unit with an average speed of 60 fps. CGIEF was performed using a desktop computer (Windows XPSP3, Dual-Core E5400 @ 2.70 GHz, 2 GB RAM; Microsoft, Redmond, WA, USA) and laptop (Windows 8.1 64-bit Edition, Core i3-2357M @ 1.30 GHz, 2 GB RAM; Microsoft). The browsers used in this study included Google Chrome (ver.60.0.3112.90; Google, Mountain View, USA), Firefox (ver.54.0.1; Mozilla Foundation, Mountain View, CA, USA), and Opera (ver.46.0.2597.32; Opera, Oslo, Norway). The total file size is approximately 450 kB for the browser or approximately 1.4 MB for the execution file (Wiguna et al., 2021).

The study protocol of the CGIEF measurement was as follows:

The measurement was completed in one session that consisted of four types of subtests (~30–45 minutes).

The children and adolescents used headphones during the measurement to reduce external noise and receive auditory instructions (Figure 1A).

One research team member accompanied the children and adolescents during the measurements.

Prior to the measurements, children and adolescents were told that they would take on the role of a fruit car driver and would deliver specific colors of fruit to specific house colors. The introduction also included the content of the CGIEF, tasks, and instructions on how to complete them during the measurement (procedure and how to use the game controller).

The first part of the session was a tutorial, in which children and adolescents were required to practice using a game controller and understand the tasks. They were instructed to complete the tasks through visual instructions (i.e., a picture of one fruit of a specific color to be delivered to a particular house appearing on a computer screen). The tutorial was completed in approximately 5 minutes. Participants were required to achieve 80% correct delivery of the entire task in the tutorial session before they could continue with the executive function measurement. If the delivery was less than 80% correct, the participants were required to repeat the tutorial session, as it was assumed that they were unfamiliar with the measurement procedure, especially when using the game controller. Each participant was limited to a maximum of three attempts during the tutorial.

The second part was an executive function measurement, which comprised four types of subtests performed in the following order: (a) visual instruction without distractors, (b) auditory instruction without distractors, (c) visual instruction with distractors, and (d) auditory instruction with distractors. Visual instructions appeared on a computer screen and showed fruits of two colors sent to houses in a particular order (Figure 1B). Auditory instructions were directly heard through headphones and participants were directed to deliver fruits of two colors to houses of two colors in order (i.e., “Please deliver purple fruit to the red house and orange fruit to the blue house”). The distractor consisted of several moving cars appearing simultaneously on a computer screen. During each subtest, constant background music was played that could be heard from the headphones, except when the auditory instructions were delivered. Participants were told to organize, plan, and find ways to overcome distractors, and simultaneously remember instructions to avoid incorrect deliveries. The fruit and house colors were set randomly to avoid memorization.

Following the completion of each subtest, the CGIEF generated five variable outcomes: (a) total number of FC; (b) total number of HC; (c) total number of fruits delivered incorrectly; (d) total number of houses picked incorrectly; and (e) time required to complete all instructions in each subtest. The four types of subtests in the executive function measurement section contained 20 variables, which were not disclosed to participants. Figure 2 shows the CGIEF measurement pathway.

Figure 1.

(A) Game Controller and Headphone Used by Subjects During the Measurement Process. (B) Visual Instructions Appearing on a Computer Screen and Showing Fruits of Two Colors That Should be Sent to Houses in a Particular Order.

Figure 2.

CGIEF Measurement.

BRIEF

The BRIEF questionnaire used in this study was parent-rated. It was first developed by Gioia et al. (2002) to evaluate executive function in children aged 5 to 18 years. The parents in our study completed the Indonesian version of the BRIEF, which was validated by the Department of Psychiatry, Faculty of Medicine, University of Indonesia-Dr. Cipto Mangunkusumo General Hospital. The Indonesian version of the BRIEF comprises 86 statements categorized into eight clinical domains: inhibition, shift, emotional control, initiation, working memory, planning/organization, organization of materials, and monitoring. It is rated on a 3-point Likert-type scale (1 = never happened, 2 = sometimes happened, and 3 = always happened). The BRIEF also yields summary index scores, including the GEC, an overall summary of all items from eight clinical domains; the MCI, an overall summary of items from the initiation, working memory, planning/organization, monitoring, and organization of material domains; and the BRI, an overall summary of items from the inhibition, shift, and emotional control domains. Raw BRIEF data were converted into t scores. The BRIEF data in this study appeared to be valid as the inconsistency level was less than nine, the negativity level was less than seven, there were no missing items in each domain, and there were no infrequency patterns. The BRIEF cutoff score in this study was set based on a t score of 65 (Wiguna et al., 2014). The inconsistency scale according to Indonesian BRIEF parent version is classified as acceptable if the inconsistency score ≤6 (cumulative percentile ≤98), questionable if the inconsistency score between 7 and 8 (cumulative percentile = 99), and inconsistent if the inconsistency score ≥9 (cumulative percentile >99). The negativity scale according to Indonesian version of BRIEF parent version is classified as acceptable if the negativity score ≤4 (cumulative percentile ≤90), questionable if the negativity score is between 5 and 6 (cumulative percentile 91–98), negative if the negativity score ≥7 (cumulative percentile >98; Riyadi, 2009).

Data Analysis

To answer the research question, three steps of analysis were performed:

Step 1: Confirmatory factor analysis (CFA) was performed using LISREL version 8.80. CFA aimed to identify the construct validity of the CGIEF for measuring executive function in children and adolescents.

Step 2: Structural equation modeling (SEM) was performed using LISREL version 8.8. The SEM analysis aimed to identify the construct interaction pathways of significant items from Step 1 toward executive function, with the BRIEF as the comparative measurement for executive function (Figure 3). SEM analysis is a strong technique for effectively addressing multicollinearity and can show a direct significant interaction pathway between the CGIEF as a predictor and executive function (BRIEF) as an outcome. The SEM analysis of the two latent variables, CGIEF and BRIEF, was fit/valid if p < .05, the critical t-value pathway was >1.96, and construct reliability (CR) was >0.7.

Step 3: Concurrent validity was performed with the Bland–Altman plot that aimed to elaborate the agreement between the t score of total executive function (TEF) from the CGIEF (total executive function/TEF raw score = FC1 + HC1 + FC2 + HC2 + FC3 + HC3 + FC4 + HC4) and the t score of GEC, MCI, and BRI from the BRIEF measurement. The Bland–Altman plot calculates the mean difference between two methods of measurement (the “bias”) and 95% limits of agreement as the mean difference (2 SD) (or more precisely [1.96 SD]; (Myles & Cui, 2007). The 95% limits were expected to include 95% of the differences between the CGIEF and BRIEF measurement methods, which would estimate a similar parameter if the coefficient of repeatability (bias/mean) nearly zero. The Bland–Altman plot analysis was performed using SPSS Version 25 for Mac.

Figure 3.

Hypothesis of the Construct Interaction Pathways Between CGIEF and BRIEF.

Results

The mean (SD) age of the participants was 11.80 (2.96) years, ranging from 6 to 18 years old, and 51% of the participants were female, while male participants accounted for 49% of the subjects. The three main ethnicities of the participants in our study were Javanese (33.0%), Betawinese (26.5%), and Sundanese (16.5%). Participants’ socioeconomic status was comparable between low-, medium-, and high-level background (38.0% vs. 34% vs. 28%). Moreover, participants included in this study were diagnosed with one or more neurodevelopmental disorders (NDD) and neuropsychiatric disorders (NPD), such as hyperkinetic disorders (28.0%), mild intellectual developmental disorders (27.5%), and borderline intellectual functioning (16.5%) (Table 1).

Table 1

Characteristics of Study Participants (n = 200).

Demographic characteristics (n = 200)	N	%
Gender
Male	98	49.0
Female	102	51.0
Age
6	1	0.5
7	11	5.5
8	14	7.0
9	25	12.5
10	24	12.0
11	31	15.5
12	16	8.0
13	21	10.5
14	13	6.5
15	11	5.5
16	16	8.0
17	16	8.0
18	1	0.5
Ethnicity
Ambonese	1	0.5
Balinese	1	0.5
Bataknese	15	7.5
Betawinese	53	26.5
Chinese	13	6.5
Javanese	66	33.0
Mandailing	1	0.5
Minahasan	4	2.0
Minang	7	3.5
Nias	1	0.5
Palembangese	4	2.0
Sundanese	34	17.0
Social-economy background
Low	76	38.0
Medium	68	34.0
High	56	28.0
Diagnosis based on ICD-10 diagnosis criteria for mental disorder
Schizophrenia	8	4.0
Schizoaffective disorders	4	2.0
Bipolar affective disorder	6	3.0
Depressive episode	13	6.5
Recurrent depressive disorder	4	2.0
Persistent mood [affective] disorders	2	1.0
Other anxiety disorders	9	4.5
Obsessive-compulsive disorder	1	0.5
Reaction to severe stress, and adjustment disorders	5	2.5
Gender identity disorders (e.g., gender dysphoria)	1	0.5
Psychological and behavioral disorders associated with sexual development and orientation	1	0.5
Mild intellectual developmental disorder	55	27.5
Specific developmental disorders of scholastic skills	9	4.5
Hyperkinetic disorders	56	28.0
Tic disorders	2	1.0
Nonorganic enuresis	2	1.0
Other symptoms and signs involving cognitive functions and awareness: Borderline intellectual functioning	33	16.5
Encounter for examination and observation for other reasons	6	3.0

Note. ICD-10 = International Classification of Diseases 10.

Step 1: CFA of CGIEF

The results of the CFA showed that only eight out of the 20 variables had a significant loading factor (loading factor > 0.5). This comprised the total number of FC and the total number of HC from the four types of stimuli. The CR of the visual instruction without distractors was 0.85, auditory instruction without distractors was 0.86, visual instruction with distractors was 0.86, and auditory instruction with distractors was 0.87. In general, the CR for the CGIEF was 0.91. The construct was valid with normal theory-weighted least squares of 15.75 (p > .05), root mean square error of approximation (RMSEA) of 0.029, normed fit index (NFI) of 1.00, comparative fit index (CFI) of 1.00, standardized root mean square residual (RMR) of 0.015, and goodness-of-fit index (GFI) of 0.98 (Figure 4).

Figure 4.

The CGIEF Construct Validity Analysis.

Step 2: SEM Analysis

SEM analysis aimed to test the CGIEF construct interaction pathways comprising eight significant items of data from the CGIEF as predictors of executive function as an outcome (BRIEF as the comparison measurement tool). SEM analysis showed that the theoretical construct of the CGIEF was a valid predictor of executive function. The critical t value of the pathway was 2.45, normal theory-weighted least squares was 5.74 (p > .05), RMSEA was 0.000, NFI was 0.99, CFI was 1.00, standardized RMR was 0.012, and GFI was 0.99 (see Figure 5).

Figure 5.

The Pattern of Construct Interaction Pathway of CGIEF (as Predictor) and Executive Function (BRIEF) (as an Outcome).

Step 3: Concurrent Validity With Bland–Altman Plot Analysis

The Bland–Altman plot analysis revealed that the coefficient of repeatability (bias/mean) was nearly zero between the t score of TEF of the CGIEF and the t scores of the BRIEF GEC, MCI, and BRI (Figures 6 –8). The study showed that a TEF t score more than 54.05 is predicted to have executive dysfunction, with sensitivity, specificity, and area under the curve (AUC) score as follows 0.78 (95% confidence interval = [0.65, 0.90]), 0.42 (95% confidence interval = [0.34, 0.50]), and 0.60 (95% confidence interval = [0.52, 0.69]), respectively. In addition, the TEF t score of more than 52.24 was considered to have behavior regulation problems (BRP) with sensitivity of 0.77 (95% confidence interval = [0.64, 0.90]), specificity of 0.41 (95% confidence interval = [0.33, 0.49]), and AUC of 0.61 (95% confidence interval = [0.52, 0.71]), and TEF t score of more than 54.76 was classified to have metacognition deficit (MCD) with sensitivity of 0.78 (95% confidence interval = [0.67, 0.89]), specificity of 0.42 (95% confidence interval = [0.34, 0.50]), and AUC of 0.60 (95% confidence interval = [0.51, 0.69]) (Table 2).

Figure 6.

The Bland–Altman Plot of the difference between the TEF t score of the CGIEF Executive Dysfunction and GEC t score of BRIEF.

Figure 7.

The Bland–Altman Plot of the Difference Between the TEF t Score of the CGIEF BRP and BRI t Score of BRIEF.

Figure 8.

The Bland–Altman Plot of the difference Between the TEF t Score of the CGIEF MCD and MCI t Score of BRIEF.

Table 2

CGIEF Sensitivity, Specificity, and Area Under the Curve.

CGIEF domain	Cutoff point	Sensitivity (95% CI)	Specificity (95% CI)	Area under the curve (95% CI)
ED	54.0472	0.78 [0.65, 0.91]	0.42 [0.34, 0.50]	0.60 [0.52, 0.69]
MCD	54.7563	0.78 [0.67, 0.89]	0.42 [0.34, 0.50]	0.60 [0.51, 0.69]
BRP	52.2381	0.77 [0.64, 0.90]	0.41 [0.33, 0.49]	0.61 [0.52, 0.71]

Note. CGIEF = Computer-Based Game Inventory for Executive Function; ED = executive dysfunction; MCD= metacognition deficit; MDD = metacognition deficit; BRP = behavior regulation problems.

Discussion

Executive functions refer to cognitive abilities related to the ability to control thoughts, emotions, and actions. Therefore, they are related to a higher level of cognitive function that develops in the dorsolateral prefrontal cortex (Ferguson et al., 2021). Hence, executive function may be involved in the learning capabilities of children and adolescents both inside the classroom and in community settings (Gunzenhauser & Nückels, 2021). Several studies have demonstrated that executive functions are related to teamwork, decision-making, problem-solving, adaptability skills, and emotional control by surrounding stimuli (Friedman & Robbins, 2022; Lawson et al., 2018; Shanmugan & Satterthwaite, 2016; Snyder et al., 2015). The eight variables, which consisted of the total number of fruits and houses correctly delivered and picked, respectively, represented the executive function theoretical background and produced a high loading factor to construct the CGIEF. This is because the eight variables fit the specific theory-derived executive function measurement model that covers the ability to complete tasks, perform certain mental activities, selectively focus on certain tasks, and memorize specific tasks to achieve a predetermined goal (Blair, 2016; Diamond, 2013; Hartung et al., 2020; Sha et al., 2019).

However, there are varieties of activity-based executive function measurement tools nowadays, such as D-KEFS, NIH Toolbox Cognitive Battery, NEPSY-2, BENCI, and CANTAB. The CGIEF can be categorized as a newer model of an executive function inventory tool developed through a computer-based game approach in the Indonesian language, and it is designed to use as a noncommercial measurement tool, whereas other executive function measurement tools are still in a foreign language and are not free to use. The CGIEF consists of four subtests: visual instruction without distractors, auditory instruction without distractors, visual instruction with distractors, and auditory instruction with distractors. CFA and SEM analyses showed that the CGIEF has fit CR and can measure several executive function domains similar to the BRIEF, as shown by the Bland–Altman plot analysis. For example, these preliminary results showed that the TEF t score may predict executive dysfunction, MCD, and BRP with fair sensitivity, but less specificity (Cicchetti et al., 1995; Mandrekar, 2010). Thus, the CGIEF may be able to identify approximately 77% to 78% of children and adolescents with executive dysfunction, MCD, and BRP, but approximately 20% of cases may be undetected (false negatives) in clinical samples with majority of cases were hyperkinetic disorders, mild intellectual development disorders, and borderline intellectual functioning. Hence, the CGIEF can be considered an executive function screening tool; however, its diagnostic value requires further research especially for other broadened child and adolescent population.

The use of computer-based games in the CGIEF may heighten children’s and adolescents’ interests, especially during executive function measurements, because they are exposed to technologies and media much more frequently than previous generations. Moreover, using game controllers in the measurement process can promote hand–eye coordination, visuospatial tracking, auditory comprehension, and spatial orientation, which support executive function measurements (Brown et al., 2010). The CGIEF RdoC comprises challenging activities, such as time limitations, specific tasks instructed by visual or auditory commands, and distractors, that need to be overcome during the measurement process (Wiguna et al., 2021). Participants were required to be attentive, emotionally controlled, assertive, and memorize every instruction to complete the entire task during the measurement process (Wiguna et al., 2021). Moreover, CGIEF tasks were delivered through visual and auditory instructions that involved the visuospatial sketchpad and phonological loop of the central executive processes. The visuospatial sketchpad is related to the processing of nonverbal information such as fruit color, obstacles that appear during tasks, and kinaesthetic aspects such as the appearance of a car driver during the delivery process (Vandenbroucke et al., 2017; S. Wang et al., 2015). The phonological loop is responsible for auditory information and uses this information to activate the Broadman area, which manages the encoding process of verbal memory (Diamond, 2013; S. Wang et al., 2015). In general, the visuospatial sketchpad and phonological loop processes are central to maintaining close attention, planning, organizing, working memory, eye tracking, and self-control to integrate tasks (colors, locations, and shapes) and activate and preserve executive function through the eight variables that are significantly associated with the executive function inventory (comprising the number of FC and the number of HC during the executive function measurement).

Although the CGIEF is a novel executive functioning inventory, it has some limitations. The CGIEF may not be used in children or adolescents with color blindness. Moreover, CGIEF may be difficult for children with hearing loss who are unable to comprehend the auditory instructions or children with developmental coordination problems due to their clumsiness, which may affect the executive function inventory results. Another limitation was that the CGIEF was less specific given the AUC was approximately 0.6. However, these results may not be optimal as diagnostic tools. This may be associated with the participants who originated from the child and adolescent outpatient psychiatry clinic and covered only three major neurodevelopmental cases, and it did not cover typical children and adolescents. Moreover, the range of participants’ age was quite large (6–17 years old). The other possibility is that the number of participants included in this study was not adequate and used the BRIEF parents’ rating scale, which is not considered comparable to the CGIEF AUC, which is a computer-based game format. Therefore, implementation and generalization of CGIEF in clinical uses need to be cautious, and future research is required to modify the method of instruction delivery or the target of delivery, such as changing the color of fruits or houses into the shape of fruits or houses, including typical children and adolescents as participants, and increasing the number of participants with more extended characteristics. In conclusion, this preliminary study showed that the CGIEF can be considered a valid and reliable measurement tool to screen for executive dysfunction, MCD, and BRP in children and adolescents in a clinical setting but with careful interpretation, and further evaluation is needed if it is positively screened.

Footnotes

Acknowledgements

The authors thank all the children, adolescents, and parents who participated in this study. They also thank Mr. Erik Wijaya for the help in reviewing the statistical methods used in this study.

Methodological Disclosure

The authors report how we determined our sample size, all data exclusions, all manipulations, and all measures in the study.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study and publication were funded by Universitas Indonesia PUTI Grant 2023 (Contract No. NKB-414/UN2.RST/HKP.05.00/2023).

ORCID iDs

Tjhin Wiguna

Sylvia Dominic Teh

Adilla Hastika Fasha

References

Albert

W. D.

Hanson

J. L.

Skinner

A. T.

Dodge

K. A.

Steinberg

Deater-Deckard

Bornstein

M. H.

Deater-Deckard

Bornstein

M. H.

Lansford

J. E.

(2020). Individual differences in executive function partially explain the socioeconomic gradient in middle-school academic achievement. Developmental Science, 8, e12937.

Bangun

S. R.

Wiguna

Ismail

R. I.

Kaligis

Winarsih

N. S.

(2019). Working memory and learning difficulties: An evidence-based study in a public primary school in Jakarta: Working memory and learning difficulties. International Journal of Child Development and Mental Health, 7(2), 11–20.

Blair

(2016). Developmental science and executive function. Current Directions in Psychological Science, 25, 3–7.

Brown

Kehoe

Kirakowski

Pitt

(2010). Beyond the gamepad: HCI and GameController design and evaluation. In Bernhaupt

(Eds.), Human-Computer Interaction Series: Evaluating user experience in games (pp. 197–219). Springer.

Cassidy

A. R.

(2016). Executive function and psychosocial adjustment in healthy children and adolescents: A latent variable modeling investigation. Child Neuropsychology, 22, 292–317.

Cicchetti

D. V.

Volkmar

Klin

Showalter

(1995). Diagnosing Autism using ICD-10 criteria: A comparison of neural networks and standard multivariate procedures. Child Neuropsychology, 1(1), 26–37.

Colom

Abad

F. J.

Quiroga

Shih

P. C.

Mendoza

C. F.

(2008). Working memory and intelligence are highly related constructs, but why? Intelligence, 36, 584–606.

Crawford

Sutherland

Garthwaite

(2008). On the reliability and standard errors of measurement of contrast measures from the D-KEFS. Journal of the International Neuropsychological Society, 14(6), 1069–1073.

Davidson

Cherry

Corkum

(2016). Validating the Behaviour Rating Inventory of Executive Functioning for children with ADHD and their typically developing peers. Applied Neuropsychology: Child, 5(2), 127–137.

10.

Denboer

J. W.

Nicholls

Corte

Chestnut

(2014). National Institutes of Health Toolbox Cognition Battery. Archives of Clinical Neuropsychology, 9(7), 692–694.

11.

Diamond

(2013). Executive functions. Annual Review Psychology, 64, 135–168.

12.

Ferguson

H. J.

Brunsdon

V. E. A.

Bradford

E. E. F.

(2021). The developmental trajectories of executive function from adolescence to old age. Scientific Reports, 11, 1382.

13.

Fernández-Alcántara

Albaladejo-Blázquez

Fernández-Ávalos

M. I.

Sánchez-SanSegundo

Cruz-Quintana

Pérez-Martínez

Carrasco-Sánchez

Pérez-Marfil

M. N.

(2022). Validity of the computerized Battery for Neuropsychological Evaluation of Children (BENCI) in Spanish children: Preliminary results. European Journal of Investigation in Health, Psychology, and Education, 12(8), 893–903.

14.

Friedman

N. P.

Robbins

T. W.

(2022). The role of prefrontal cortex in cognitive control and executive function. Neuropsychopharmacology, 47, 72–89.

15.

Gilbert

S. J.

Burgess

P. W.

(2008). Executive function. Current Biology, 18(3), R110–R114.

16.

Gioia

G. A.

Isquith

P. K.

Kenworthy

Barton

R. M.

(2002). Profiles of everyday executive function in acquired and developmental disorders. Child Neuropsychology, 8(2), 121–137.

17.

Grissom

N. M.

Herdt

C. T.

Desilets

Lidsky-Everson

Reyes

T. M.

(2015). Dissociable deficits of executive function caused by gestational adversity are linked to specific transcriptional changes in the prefrontal cortex. Neuropsychopharmacology, 40, 1353–1363.

18.

Gunzenhauser

Nückles

(2021). Training executive functions to improve academic achievement: Tackling avenues to far transfer. Frontiers in Psychology, 12, 624008.

19.

Hartung

Engelhardt

L. E.

Thibodeaux

M. L.

Harden

K. P.

Tucker-Drob

E. M.

(2020). Developmental transformations in the structure of executive functions. Journal of Experimental Child Psychology, 189, 104681.

20.

Hawk

B. N.

Wright

Julian

M. M.

Rosas

J. M.

Merz

E. C.

McCall

R. B.

(2013). Potential selective responding in a parent questionnaire study of post-institutionalized children. Adoption Quarterly, 16(2), 97–107.

21.

Hendrawan

Fatimah

H. N.

Carolina

Fauzani

Ariefa

M. A. M.

(2020). The construction of computer-based application of working memory test for early age children in Indonesia. International Journal on Advanced Science, Engineering and Information Technology, 10(5), 1806–1812.

22.

Henry

L. A.

(2010). The episodic buffer in children with intellectual disabilities: An exploratory study. Research in Developmental Disabilities, 31(6), 1609–1614.

23.

Hofmann

Schmeichel

B. J.

Baddeley

A. D.

(2012). Executive functions and self-regulation. Trends in Cognitive Science, 16(3), 174–180.

24.

Killi

Ketamo

(2017). Evaluating cognitive and affective outcomes of a digital game-based math test. IEEE Transactions on Learning Technologies, 11(2), 255–263.

25.

Kronenberger

W. G.

Castellanos

Pisoni

D. B.

(2018). Questionnaire-based assessment of executive functioning: Case studies. Applied Neuropsychology: Child, 7(1), 82–92.

26.

Kusi-Mensah

Nuamah

N. D.

Wemakor

Agorinya

Seidu

Martyn-Dickens

Bateman

(2022). A systematic review of the validity and reliability of assessment tools for executive function and adaptive function following brain pathology among children and adolescents in low- and middle-income countries. Neuropsychology Review, 32(4), 974–1016.

27.

Lawson

G. M.

Hook

C. J.

Farah

M. J.

(2018). A meta-analysis of the relationship between socioeconomic status and executive function performance among children. Developmental Science, 21(2), e12529.

28.

Mandrekar

J. N.

(2010). Receiver operating characteristic curve in diagnostic test assessment. Journal of Thoracic Oncology, 5(9), 1315–1316. https://doi.org/10.1097/JTO.0b013e3181ec173d

29.

Mavridis

Thrasyvoulos

(2017). Game-based assessment: Investigating the impact on test anxiety and exam performance. Journal of Computer Assisted Learning, 33(2), 137–150.

30.

Moisala

Salmela

Carlson

Salmela-Aro

Lonka

Hakkarainen

Alho

(2018). Neural activity patterns between different executive tasks are more similar in adulthood than in adolescence. Brain Behaviour, 8(9), e01063.

31.

Myles

P. S.

Cui

(2007). Using the Bland-Altman method to measure agreement with repeated measures. British journal of anaesthesia, 99(3), 309–311. https://doi.org/10.1093/bja/aem

32.

Nouchi

Taki

Takeuchi

Hashizume

Nozawa

Kambara

Sekiguchi

Miyauchi

C. M.

Kotozaki

Nouchi

Kawashima

(2013). Brain training game boost executive functions and processing speed in the young adults: A randomized controlled trail. PLOS ONE, 8(2), e55518.

33.

Orban

S. A.

Rapport

M. D.

Friedman

L. M.

Kofler

M. J.

(2014). Executive function/cognitive training for children with ADHD: Do results warrant the hype and cost? The ADHD Report, 22(8), 8–14.

34.

Pluck

Amraoui

Fornell-Villalobos

(2021). Brief communication: Reliability of the D-KEFS Tower Test in samples of children and adolescents in Ecuador. Applied Neuropsychology: Child, 10(2), 158–164.

35.

Riyadi

E. I.

(2009). Determination of the validity and reliability of Behaviour Rating Inventory of Executive Function instrument [Master’s thesis, University of Indonesia, Jakarta].

36.

Sha

Wager

T. D.

Mechelli

(2019). Common dysfunction of large-scale neurocognitive networks across psychiatric disorders. Biological Psychiatry, 85, 379–388.

37.

Shanmugan

Satterthwaite

T. D.

(2016). Neural markers of the development of executive function: Relevance for education. Current Opinion in Behavioural Sciences, 10, 7–13.

38.

Shute

Wang

(2017). Assessment and adaptation in games. In Wouters

van Oostendorp

(Eds.), Instructional techniques to facilitate learning and motivation of serious games (pp. 59–78). Springer.

39.

Snyder

H. R.

Miyake

Hankin

B. L.

(2015). Advancing understanding of executive function impairments and psychopathology: Bridging the gap between clinical and cognitive approaches. Frontiers in Psychology, 6, 328.

40.

Swanson

H. L.

(2015). Intelligence, working memory, and learning disabilities. In Papadopoulos

T. C.

Parrila

R. K.

Kirby

J. R.

(Eds.), Cognition, intelligence, and achievement: A tribute to J. P. Das (pp. 175–196). Elsevier Academic Press. https://doi.org/10.1016/B978-0-12-410388-7.00010-5

41.

Syväoja

H. J.

Tammelin

T. H.

Ahonen

Räsänen

Tolvanen

Kankaanpää

Kantomaa

M. T.

(2015). Internal consistency and stability of the CANTAB neuropsychological test battery in children. Psychological Assessment, 27(2), 698–709.

42.

Uljarević

Cai

R. Y.

Hardan

A. Y.

Frazier

T. W.

(2023). Development and validation of the Executive Functioning Scale. Frontiers in Psychiatry, 13, e1078211.

43.

Vandenbroucke

Verschueren

Baeyens

(2017). The development of executive functioning across the transition to first grade and its predictive value for academic achievement. Learning and Instruction, 49, 103–112.

44.

Wang

Huang

(2021). A systematic review of serious games for collaborative learning: Theoretical framework, game mechanic and efficiency assessment. International Journal of Emerging Technologies in Learning, 16(6), 88.

45.

Wang

Allen

R. J.

Lee

J. R.

Hsieh

C. E.

(2015). Evaluating the developmental trajectory of the episodic buffer component of working memory and its relation to word recognition in children. Journal of Experimental Child Psychology, 133, 16–28.

46.

Weintraub

Dikmen

S. S.

Heaton

R. K.

Tulsky

D. S.

Zelazo

P. D.

Bauer

P. J.

Carlozzi

N. E.

Slotkin

Blitz

Wallner-Allen

Fox

N. A.

Beaumont

J. L.

Mungas

Nowinski

C. J.

Richler

Deocampo

J. A.

Anderson

J. E.

Manly

J. J.

Borosh

. . .Gershon

R. C.

(2013). Cognition assessment using the NIH Toolbox. Neurology, 80(11 Suppl. 3), S54–S64.

47.

Wiguna

Guerrero

A. P.

Honjo

Ismail

N. S.

Kaligis

(2014). Executive dysfunction among children with antipsychotic treated schizophrenia. Clinical Psychopharmacology and Neuroscience, 12(3), 203–208.

48.

Wiguna

Ismail

R. I.

Kaligis

Minayati

Murtani

B. J.

Wigantara

N. A.

Pradana

Bahana

Dirgantoro

B. P.

Nugroho

(2021). Developing and feasibility testing of the Indonesian computer-based game prototype for children with attention deficit/hyperactivity disorder. Heliyon, 7(7), e7571.

49.

Wiguna

Setyawati

N. R.

Kaligis

Belfer

M. L.

(2012). Learning difficulties and working memory deficit among primary school students in Jakarta. Clinical Psychopharmacology & Neuroscience, 10, 105–109.