Abstract
The aim of this study is to validate the Attention, Memory, and Frontal Abilities Screening Test (AMFAST), a novel, 10-minute, paper-and-pencil measure developed to identify attention, processing speed, memory, and executive functioning deficits in children and adults with various conditions characterized by frontal-subcortical dysfunction. We administered the AMFAST to 186 English-speaking healthy control participants (aged 8-88 years) without reported cognitive impairment. The AMFAST was also administered to a mixed clinical sample that included 114 English-speaking individuals (aged 8-84 years) who also received comprehensive neuropsychological testing. Results indicated that total AMFAST scores in the healthy control sample were not significantly affected by education or gender. There was, however, a significant effect of age, as the 8- to 10-year-old group scored significantly lower than other age groups. Thus, only participants 11+ years were included in further analyses. The AMFAST demonstrated high test–retest and interrater reliabilities, good construct validity, and the identified optimal cutoff score of 70 had excellent sensitivity and specificity for differentiating between cognitively intact and cognitively impaired individuals. These findings demonstrate that the AMFAST is a highly effective screening test that can be used to identify attention, memory, processing speed, and executive functioning deficits in individuals from middle childhood through older adulthood.
Cognitive impairment complicates multiple conditions that occur across the life span. Indeed, cognitive deficits are routinely seen in neurologic conditions such as traumatic brain injury (TBI; Arciniegas, Held, & Wagner, 2002), multiple sclerosis (MS; Chiaravalloti & DeLuca, 2008), and Parkinson’s disease (PD; Aarsland, Brønnick, Larsen, Tysnes, & Alves, 2009), psychiatric illness such as schizophrenia (Heinrichs & Zakzanis, 1998), unipolar and bipolar depression (Rubinsztein, Michael, Paykel, & Sahakian, 2000), and attention-deficit/hyperactivity disorder (ADHD; Hervey, Epstein, & Curry, 2004), and systemic diseases such as systemic lupus erythematosus (SLE; Loukkola et al., 2003), sickle cell disease (SCD; Schatz, Finke, Kellett, & Kramer, 2002), and human immunodeficiency virus (HIV; Heaton et al., 2011). Oftentimes, the cognitive dysfunction associated with these conditions is characterized by a distinct pattern of deficits; namely, deficits are typically found in the areas of attention, processing speed, memory, and/or executive functioning (EF). Although different underlying neuroanatomic mechanisms are thought to be at play in these discrete disorders, they all have been associated with disturbances that either directly or indirectly affect the functional integrity of frontal-subcortical brain circuits. Moreover, multiple medications and treatments, including certain chemotherapies (Buizer, de Sonneville, & Veerman, 2009) and focused brain radiation (Roman & Sperduto, 1995), are also known to impact these frontal-subcortical circuits.
As it stands presently, the gold standard for the identification of cognitive impairment is formal neuropsychological assessment. While there are many benefits to these evaluations (Arffa & Knapp, 2008; Tremont, Westervelt, Javorsky, Podolanczuk, & Stern, 2002), they can be costly and time consuming, not always readily available, and require specialized and advanced training (Nelson et al., 2016; Sweet, Benson, Nelson, & Moberg, 2016). As such, there has been a movement in recent years for medical practitioners in primary care settings and specialized clinics to administer screening tests when cognitive impairment is suspected. The information obtained through these screeners can then be utilized to determine whether a referral for more comprehensive neuropsychological evaluations is indicated.
Multiple tests have been developed to screen for cognitive impairment (Cullen, O’Neill, Evans, Coen, & Lawlor, 2007). In general, these measures are brief (i.e., 20 minutes or less) and contain items sampling a wide range of cognitive domains. Many of these measures were originally developed to screen specifically for dementia and consequently place a high emphasis on memory and less of an emphasis on other areas of cognitive functioning (Segal-Gidan, 2013). One such area that is often underassessed in cognitive screeners is EF. For instance, one of the most commonly used cognitive screening instruments, the Mini-Mental State Examination (MMSE; Folstein, Folstein, & McHugh, 1975), does not contain any items relating directly to EF. On the other hand, another popular screening test, the Montreal Cognitive Assessment (MoCA; Nasreddine et al., 2005), does evaluate aspects of EF. However, it too was developed primarily to identify cognitive changes associated with dementia; as such, it contains items assessing other cognitive domains such as language, visuospatial skills, and orientation that are pertinent to a dementia differential but may not be as relevant for some of the conditions mentioned earlier. For instance, of the 30 total possible points of the MoCA, 6 (20%) relate to aspects of orientation (i.e., time and place), which, although relevant in the assessment of dementia, is not a sensitive indicator of more subtle types of impairment seen in other neurocognitive disorders. Also, the authors of the MoCA state on their website that their paper test may not always capture deficits in processing speed (Nasreddine, 2018), an aspect of cognition that is highly sensitive to psychiatric and neurologic disease. Nevertheless, the MoCA is one of the most established and widely utilized broad cognitive screening measures, and it has been used to screen for cognitive impairment in a variety of conditions characterized by frontal-subcortical dysfunction, including MS (Dagenais et al., 2013), HIV (Overton et al., 2013), SLE (Adhikari, Piatti, & Luggen, 2011), PD (Zadikoff et al., 2008), and schizophrenia (Fisekovic, Memic, & Pasalic, 2012).
Furthermore, most screening tests like the MoCA were created to assess functioning in adults only. Indeed, given that the MoCA was originally developed for older patients, it may not be the most appropriate screen for youth; outside of a few studies investigating its utility in adolescents and young adults (Pike, Poulsen, & Woo, 2017; Yang, Long, & Xiao, 2016), it has not been validated in younger populations. To our knowledge, there are only a handful of cognitive screening tests that have been developed specifically for children. Most of these are computer-based measures (Reeves, Bleiberg, Winter, & Roebuck-Spencer, 2004; Weintraub et al., 2013), and we are only aware of one measure, the School-Years Screening Test for the Evaluation of Mental Status (SYSTEMS; Ouvrier, Hendy, Bornholt, & Black, 1999), that is administered in a paper-and-pencil format. It should be noted that the SYSTEMS is essentially composed of items from the MMSE (modified for children) and, as such, does not directly assess processing speed or components of EF. Given that many of the aforementioned conditions characterized by frontal-subcortical dysfunction (e.g., SLE, SCD, ADHD, and leukemia) are present in children and adolescents as well as adults, we believe that there are clinical indications for a cognitive screener that specifically evaluates for deficits in attention, processing speed, memory, and EF and can be administered to individuals across the life span. It is our experience that a notable proportion of neuropsychological referrals from hospital-based services come from pediatric departments (e.g., pediatric rheumatology, pediatric hematology-oncology, and pediatric neurology), and we believe that such a screening tool would be invaluable for these referral sources in determining whether to send their patients for more comprehensive neuropsychological testing. It is also worth noting that, to our knowledge, no screening tests have been published with the intention of use across the life span to date, which could simplify the process for practices that treat patients of all ages. Moreover, from a research standpoint, such an instrument would also be useful to allow for longitudinal studies of conditions across the life span.
With the above limitations of existing screening measures in mind, we developed the Attention, Memory, and Frontal Abilities Screening Test (AMFAST). The AMFAST is a novel, paper-and-pencil screening test that was specifically designed to assess for the presence of attention, processing speed, memory, and/or EF deficits, which are commonly found in individuals with various medical and psychiatric conditions. The beta version of the AMFAST was created by the first author, a board certified neuropsychologist with more than 15 years of clinical experience. Items were refined over a period of several months with input from several neuropsychologists and specialists in other fields (e.g., neurology and rheumatology) as well as from feedback from participants who were administered beta versions of the AMFAST. The goal was to develop an easily administered and scored instrument that takes less than 10 minutes to complete (which was achieved for all participants during beta testing). Item/task selection was based on the clinical experiences of the first author regarding areas of impairment commonly encountered in patients with frontal-subcortical dysfunction as well as the information garnered through a review of the literature on memory, attention, and EF across the life span (Burgess, 2003; Lezak, Howieson, & Loring, 2004; Squire & Butters, 1992). In the end, six subtests were selected for the final version of the AMFAST (see Table 1). In line with other screening instruments, all of the AMFAST tasks were intended to be relatively simple such that only individuals with actual cognitive impairment would exhibit diminished performance.
AMFAST Subtests.
Three of the tasks of the AMFAST were designed to screen for memory deficits and are simplified versions of well-established, commonly used paradigms. These are (a) a word-list learning and recall task, similar to the memory task of the MoCA, involving the presentation of five words over two trials (List Memory); (b) a story/prose learning and recall task involving the presentation of a short story (read aloud once) that is composed of seven main details (Story Memory); and (c) a figure reproduction and recall task in which examinees are required to copy and later reproduce (free recall) a figure that has five details/elements (Spatial Memory). In the selection of these tasks, importance was placed on assessing memory through different modalities (i.e., auditory and visual), and the decision to include both list learning and story learning tasks was based on research findings demonstrating distinct patterns of performance across these tasks in different patient groups. For example, studies have shown that individuals with frontal-subcortical dysfunction (in contrast to temporal lobe dysfunction) often perform better on story as compared with list learning tasks because they benefit more when information is presented in an organized context (Cummings, 1990; Perri, Fadda, Caltagirone, & Carlesimo, 2013). To our knowledge, there are no other cognitive screening tests that assess memory through all three of these paradigms.
The other three tasks of the AMFAST were designed to screen for deficits in aspects of attention, processing speed, and EF and also utilize simplified versions of commonly employed paradigms. These are (a) a timed divided attention/cancellation task in which examinees are instructed to draw a slash through all the 2s and 4s in a series of written numbers as fast and as accurately as possible (Number Cancellation); (b) a timed set-shifting/mental flexibility task, similar to Part B of the Oral Trail Making Test (Ricker & Axelrod, 1994), in which examinees are required to alternate between saying numbers and letters aloud in numerical and alphabetical order (“1, A, 2, B, 3, C,” etc., until “13”) as fast and as accurately as possible (Number–Letter Switching); and (c) an auditory attention/response inhibition task in which examinees are provided a sheet of paper with four numbered circles on it and are instructed to tap the number 4 when the examiner says 2, tap 2 when he or she says 4, and do not tap when he or she says 1 or 3 (Number Inhibition). In the selection of these tasks, importance was placed on assessing multiple aspects of attention and EF (e.g., divided attention, set-shifting, mental flexibility, and inhibitory control). Moreover, the timed aspect of two of these tasks was emphasized, given the importance of processing speed in the assessment of conditions characterized by frontal-subcortical dysfunction. We are unaware of any other cognitive screening tests that assess EF to the extent that the AMFAST does, other than perhaps the Frontal Abilities Battery (Dubois, Slachevsky, Litvan, & Pillon, 2000), which is a brief measure that solely evaluates aspects of EF (but not attention or memory) and was designed for more acutely impaired populations at bedside.
The aim of this study was to investigate the psychometric properties of and validate the AMFAST. Toward this goal, we examined (a) the effects of demographic variables (i.e., age, education, and gender) on AMFAST scores in a healthy control sample; (b) the test–retest reliability of the AMFAST in a subset of participants in the healthy control sample; (c) the interrater reliability of the Spatial Memory subtest in a subsample of the healthy control sample given that there is some subjectivity inherent in scoring systems of this type; (d) the construct validity of the AMFAST in a mixed clinical sample, by assessing how performance on the AMFAST relates to performances in corresponding areas on a comprehensive neuropsychological test battery; and (e) the diagnostic utility of the AMFAST in a mixed clinical sample, as assessed by its sensitivity and specificity at identifying deficits in attention, processing speed, memory, and EF on a comprehensive neuropsychological test battery. Ultimately, we aimed to identify an empirically derived AMFAST cutoff score that optimally distinguishes impaired and nonimpaired participants.
Method
Participants
Healthy Control Sample
One hundred eighty-six English-speaking individuals, aged 8 to 88 years, comprised the healthy control sample. Of these 186 subjects, 24 (12.9%) were included in the test–retest normative sample. Participants were recruited among the following groups: family members/caregivers of investigators’ outpatients, support staff at Montefiore Medical Center (MMC) located in the Bronx, New York, acquaintances of investigators, medical inpatients without neurological/cognitive symptoms admitted to MMC, and inpatients’ healthy relatives. Participants were excluded from the study if they had uncorrected visual impairment, uncorrected hearing loss, were not fluent in English, were primarily nonverbal or uncommunicative, had an upper extremity disability that would affect motor performance, were currently taking medicine that might depress performance (e.g., anticonvulsants, antipsychotics, and anxiolytics), were currently undergoing treatment for alcohol or drug abuse, had a history of any memory or other cognitive complaints, or were previously diagnosed with any physical or mental condition or illness that might depress test performance, including but not limited to stroke, epilepsy, brain tumor, TBI, brain surgery, encephalitis, meningitis, MS, SLE, PD, Huntington’s chorea, Alzheimer’s disease, schizophrenia, bipolar disease, major depression, HIV, intellectual disability/mental retardation, specific learning disability, and/or ADHD. Individuals were also excluded if they were ever placed in special education, if they ever repeated a grade, and/or if they ever had a 504 plan or individualized education program.
Clinical Sample
One hundred fourteen English-speaking individuals, aged 8 to 84 years, comprised the clinical sample. Clinical patients were referred to the investigators via general hospital referral base, including neurology, neurosurgery, hematology/oncology, rheumatology, psychiatry, primary care, pediatric, and outpatient psychology clinics. Individuals within this clinical sample were diagnosed with a wide variety of conditions: Alzheimer’s disease (6.1%), vascular dementia (3.5%), unspecified dementia (8.8%), mild cognitive impairment (16.7%), SLE (7.9%), other rheumatologic illness (2.6%), leukemia (2.6%), lymphoma (0.9%), brain tumor (1.8%), other cancer (0.9%), major depression(4.4%), bipolar disorder (0.9%), anxiety disorder (0.9%), adjustment disorder (7.9%), schizophrenia (0.9%), schizoaffective disorder (1.8%), somatoform disorder (1.8%), ADHD (7.9%), autism spectrum disorder (0.9%), SCD (5.3%), epilepsy (2.6%), MS (0.9%), TBI (1.8%), stroke (1.8%), and learning disability (0.9%).
Institutional review board (IRB) approval was obtained at MMC/Albert Einstein College of Medicine (AECOM). Healthy control participants provided oral consent prior to participation, as per the MMC/AECOM IRB protocol. Clinical participants also provided oral consent prior to the administration of the AMFAST, which was included as part of a comprehensive neuropsychological battery. Attempts were made to minimize interference with tests administered as part of the comprehensive battery; in particular, whenever possible, the AMFAST was either administered prior to the administration of the comprehensive battery, after a long break, or on a different date altogether (e.g., at the feedback session).
Measures
The AMFAST
The AMFAST is a novel, paper-and-pencil, 10-minute screening test that was specifically designed to assess for the presence of attention, processing speed, memory, and/or EF deficits, which are commonly found in individuals with various medical and psychiatric conditions. The AMFAST is composed of six subtests (see Table 1): three attention/EF tasks and three memory tasks. The total score is calculated by summing up the scores of the six subtests, with a maximum of 100 points.
Each of the three memory tasks has a maximum of 15 points. Only the delayed memory portions of each task are scored (not the immediate memory/learning components). On the List Memory task, three points are allotted for each spontaneously recalled word, and one point is allotted when a word is recognized following a semantic cue that is provided in the event that the word is not spontaneously recalled (maximum = 15 points). On the Story Memory task, 2 points are given for each of seven spontaneously recalled details (specified target words), and a bonus point is awarded for recalling the story perfectly (maximum = 15 points). Of note, if participants exhibit interference between the details of the list and story (e.g., they recall specific details of the list when asked to recall the details of the story, such as “Saturday” instead of “Sunday,” “dog” instead of “cat,” and “father” instead of “mother”), 2 points per interference error are subtracted from the total AMFAST score. On the Spatial Memory task, 1 point is given for accuracy and 1 point is given for the correct placement of each component based on objectively defined criteria, with a bonus point given if a detail is both correctly drawn and placed (maximum = 15 points).
With regard to the attention/EF tasks of the AMFAST, the Number Inhibition task has a maximum of 15 points. On this task, 5 points are subtracted for each error (omission, commission, or perseverative) made by participants. If participants commit more than two errors, they receive a score of zero. The Number–Letter Switching and Number Cancellation tasks have a maximum of 20 points each, with 10 points allotted for speed and 10 points allotted for accuracy of performance. Given the importance of processing speed in the assessment of conditions that the AMFAST was designed to assess, we decided that the Number–Letter Switching and Number Cancellation tasks would each contribute a slightly greater percentage toward the total score (i.e., 20%) than other tasks (i.e., 15%) in order to be able to separately and adequately account for both speed and accuracy. It is important to note that the scoring system for these two tasks with a processing speed component was specifically designed to account for normal variability and only penalizes for impaired performance. This was accomplished by examining the first 50 healthy control participants and designing a system in which examinees only began to lose points when their completion times were 1 standard deviation or more below the mean.
Comprehensive Clinical Battery
Multiple neuropsychological measures were utilized as part of the comprehensive clinical battery. In line with the composition of the AMFAST, only measures assessing components of attention, processing speed, memory, and EF were analyzed in this study. Measures administered as part of the comprehensive battery that assessed cognitive domains other than those assessed by the AMFAST (e.g., language, visuospatial function, etc.) were not included. Memory measures used in this study included subtests from the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS; Randolph, 1998), subtests from the Neuropsychological Assessment Battery (NAB; White & Stern, 2001), subtests from A Developmental Neuropsychological Assessment–Second edition (NEPSY-II; Korkman, Kirk, & Kemp, 2007), subtests from the Wechsler Memory Scale–Revised edition and Fourth edition (WMS-R/IV; Wechsler, 1987; 2009), subtests from the Children and Adolescent Memory Profile (ChAMP; Sherman & Brooks, 2015), components from the California Verbal Learning Test–Second edition and Child version (CVLT-II/CVLT-C; Delis, Kramer, Kaplan, & Ober, 1994, 2000), and components from the Brief Visuospatial Memory Test–Revised edition (BVMT-R; Benedict, 1997). Attention/EF measures used in this study included subtests comprising the working memory and processing speed indices from the Wechsler Adult Intelligence Scale–Fourth edition (WAIS-IV; Wechsler, 2008) and Wechsler Intelligence Scales for Children–Fifth edition (WISC-V; Wechsler, 2014), subtests from the Delis–Kaplan Executive Function System (D-KEFS; Delis, Kaplan, & Kramer, 2001), subtests from the RBANS, subtests from the NAB, subtests from the NEPSY-II, the Trail Making Test (TMT; Reitan, 1979), the Stroop Color and Word Test (Golden, 1978), and the Brixton Spatial Anticipation Test (BSAT; Burgess & Shallice, 1997). Measures were selected based on clinical appropriateness on a case-by-case basis, so not all participants received the same set of tests.
Statistical Analysis
Healthy Control Sample
The effects of age, gender, and education on total AMFAST scores in the healthy control sample were examined. For the purposes of these analyses, participants were stratified by age (8-10, 11-15, 16-19, 20-29, 30-39, 40-49, 50-59, 60-69, and 70+ years), gender, and level of education (less than high school, high school/some college, bachelor’s degree, and postbachelor’s degree). The designated age groups for the children/adolescents (aged 8-19 years) were selected to approximate late childhood (8-10 years), preteen/early adolescent (11-15 years), and late adolescent (16-19 years), and this grouping arrangement is generally consistent with the age cutoffs used by other studies investigating child and adolescent brain development (e.g., Shaw et al., 2006; Sowell, Thompson, Holmes, Jernigan, & Toga, 1999). One-way analysis of variance test was utilized to separately investigate the effects of age and education on total AMFAST scores, while the two-sample t test was used to test the effects of gender on the total AMFAST scores. Of note, only participants older than 30 years were included in the education analyses, so as to not penalize younger individuals for not yet having the opportunity to attain college and postgraduate degrees. To investigate the test–retest reliability of the AMFAST, the bivariate Pearson’s correlation was calculated between total AMFAST scores at Time 1 and total AMFAST scores at Time 2 for a subset of 24 healthy control participants. The minimal time interval between test administrations was set at 28 days. In order to assess for possible selection bias between the test–retest participants and the rest of the healthy control participants, a two-sample t test was used to compare total AMFAST scores at Time 1 between the two groups. To assess interrater reliability for the Spatial Memory subtest, the intraclass correlation coefficient was calculated between the scores obtained by three independent raters across 20 randomly selected cases.
Clinical Sample
To assess the construct validity of the AMFAST, Pearson’s correlation was calculated between the sum of scores of the three memory subtests from the AMFAST and the percentage of impaired scores on memory measures from the comprehensive neuropsychological test battery. For example, if 10 memory measures were administered in the comprehensive test battery and 4 of those measures resulted in impaired scores, the impairment index would be 0.4 or 40%. A similar analysis was conducted for the three attention/EF subtests of the AMFAST and the attention, processing speed, and EF measures from the test battery. Our neuropsychological test batteries consisted of no fewer than seven measures in each of the above domains. We defined impairment on a particular test as standard scores that fell at least 1.5 standard deviations below the normative mean (i.e., ≤7 percentile). It should be noted that we opted to use this method of assessing construct validity instead of simply comparing scores on the AMFAST with traditional neuropsychological measures, because (a) our participants did not receive identical comprehensive clinical batteries and (b) ultimately, the objective of the AMFAST is to identify impairment in attention, processing speed, memory, and EF. This is in contrast to formal neuropsychological testing, which provides a more nuanced and comprehensive assessment of an individual’s strengths and weaknesses in these domains.
The diagnostic utility of the AMFAST was assessed by examining its ability to identify overall impairment on the comprehensive neuropsychological test battery. Participants were considered to be overall impaired on comprehensive neuropsychological testing when at least 25% of their scores on measures of attention, processing speed, memory, and EF (in total) fell at least 1.5 standard deviations below the normative mean (i.e., ≤7 percentile). This standard of defining impairment has been previously utilized and supported in the literature (e.g., Beauchamp et al., 2015). We specifically selected this method of defining impairment to account for normal variability and the fact that previous studies have demonstrated that it is not uncommon for cognitively intact participants to obtain one or more impaired scores within a comprehensive neuropsychological battery (Binder, Iverson, & Brooks, 2009; Brooks, Iverson, & White, 2007; Schretlen, Munro, Anthony, & Pearlson, 2003). A receiver operating characteristic (ROC) with area under the curve (95% CI) analysis was conducted between total AMFAST scores and impairment on the comprehensive neuropsychological test battery. The sensitivities, specificities, and overall classification accuracies were tabulated for various total AMFAST scores. The optimal AMFAST cutoff score was identified by maximizing the Youden index scores (Youden, 1950).
Results
Healthy Control Sample
Demographic information for the healthy control sample is summarized in Table 2. The mean age of the healthy control participants was 31.3 years (SD = 21.5). Of the healthy control participants, 62% of participants were female, and the mean number of years of education (children included) was 11.4 (SD = 5.1). In terms of race/ethnicity, 36.8% of participants identified as Hispanic/Latino, 32.4% identified as Caucasian, 20.0% identified as African American, 2.1% identified as Asian, 1.6% identified as Afro-Caribbean, 0.5% identified as Native American, and 6.5% identified as other.
Participant Demographics.
An analysis of variance revealed significant differences in total AMFAST scores based on participant age, F(8, 176) = 7.64, p < .01. Post hoc comparisons using the Bonferroni correction indicated that the mean score for the 8- to 10-year-old group (M = 77.8, SD = 8.8) was significantly lower than the mean scores for all other age groups (see Figure 1). In light of this difference, along with the fact that that our clinical sample consisted of few impaired children in the 8- to 10-year-old range that would allow us to investigate whether a scoring change or other modifications to the AMFAST could be made, this group was excluded from further analyses, and only participants 11 years and older were included in the analyses of both the normative (n = 157) and clinical (n = 106) samples. No other significant differences were found based on age. Therefore, the age groups were collapsed for further analyses. No significant gender differences were found in total AMFAST scores, t(155) = 0.84, p = .40. Similarly, no significant differences were found in total AMFAST scores based on level of education, F(3, 76) = 1.97, p = .14. Accordingly, these groups were collapsed in the subsequent analyses. In the end, after excluding the 8- to 10-year-old, the mean score on the AMFAST was 88.38 (SD = 6.93).
Mean total AMFAST scores by age in the healthy control sample.
The test–retest reliability of the AMFAST, as measured by the Pearson’s correlation between total AMFAST scores at Time 1 and total AMFAST scores at Time 2 for a subset of 24 normative participants, was .87 (p < .01). On average, total AMFAST test scores increased by only 1.9 points (SD = 2.7) over the test–retest interval (M = 98.8 days, SD = 91.6, range = 29-309 days). There were no significant differences found in total AMFAST scores at Time 1 between the test–retest participants and the rest of the normative participants, t(156) = 1.75, p = .08.
The interrater reliability of the Spatial Memory subtest, as measured by the intraclass correlation coefficient of the three independent raters across 20 randomly selected cases, was .95 (p < .01).
Clinical Sample
Demographic information for the clinical sample is summarized in Table 2. The mean age of the clinical participants was 44.2 years (SD = 24.4). A total of 65% of participants were female and the mean number of years of education (children included) was 12.0 (SD = 4.3). In terms of race/ethnicity, 30.1% of participants identified as Hispanic/Latino, 24.8% identified as Caucasian, 20.4% identified as African American, 14.1% identified as Afro-Caribbean, 1.8% identified as Asian, and 8.8% identified as other.
In terms of the construct validity analyses, Pearson’s correlation between the sum of scores of the memory subtests of the AMFAST and the percentage of impaired memory measures in the comprehensive neuropsychological test battery was −.81 (p < .01). Similarly, Pearson’s correlation between the sum of scores of the attention/EF subtests of the AMFAST and the percentage of impaired attention, processing speed, and EF measures in the comprehensive neuropsychological test battery was −.82 (p < .01).
According to the case processing summary of the ROC analysis, 36 of the 106 clinical subjects (34%) were classified as cognitively impaired according to results of comprehensive neuropsychological testing. The ROC curve between total AMFAST scores and impairment on the comprehensive neuropsychological battery is depicted in Figure 2. The area under the curve was 0.98 (95% CI [0.96, 1.00]). In addition, the sensitivities, specificities, overall classification accuracies, and Youden index scores for various AMFAST cutoff are presented in Table 3. An analysis of the Youden index scores revealed that the best balance of sensitivity (97.2%) and specificity (95.7%) was at an AMFAST cutoff score of 70/100. Using this cutoff score, there was a 96.2% overall classification success at identifying impairment in the clinical battery. Using a higher cutoff score of 73 resulted in 100% sensitivity, albeit with a lower specificity (87%) and lower overall classification accuracy (91.4%).
ROC analysis between total AMFAST scores and impairment on the comprehensive neuropsychological battery in the clinical sample.
Identification of Cognitive Impairment in the Clinical Sample by Total AMFAST Scores.
Note. AMFAST = Attention, Memory, and Frontal Abilities Screening Test.
Discussion
The AMFAST is an easy to administer, 10-minute screening test that can be used to assess for the presence of attention, processing speed, memory, and EF deficits in individuals from middle childhood through older adulthood. We found that average AMFAST scores did not significantly differ across age groups (aged 11+ years) and were not significantly affected by education or gender. Furthermore, the AMFAST was shown to have high test–retest and interrater reliabilities, good construct validity, and the identified optimal cutoff score of 70 had excellent sensitivity and specificity for differentiating between cognitively intact and cognitively impaired individuals in a diverse clinical sample across a large portion of the life span.
One interesting finding from the current study was that total AMFAST scores were significantly lower in our healthy control sample of 8- to 10-year-olds as compared with all other age groups. This suggests that children younger than 11 years are performing in a qualitatively distinct manner than the older participants. However, this finding is not completely unexpected given the extant data on neuromaturation, particularly within the frontal lobes, which are intimately involved in higher order thinking skills (i.e., complex attention and EF abilities). Indeed, many studies have demonstrated remarkable brain development during adolescence and, in some cases, well into early adulthood (e.g., Johnson, Blum, & Giedd, 2009; Sowell et al., 1999). Of note, data have specifically shown that gray matter volumes peak between the ages of 11 to 14 years in the frontal lobes (Giedd & Rapoport, 2010; Lenroot & Giedd, 2006), which may potentially help explain the jump in AMFAST scores in our sample in participants at age 11. Our findings are also consistent with a study by Bornholt, Ajersch, Fisher, Markham, and Ouvrier (2010) in which scores on the SYSTEMS, another cognitive screening measure for children and adolescents (aged 5-15 years), increased steadily during childhood but plateaued at age 11. Importantly, this finding does not necessarily preclude use of the AMFAST in children younger than 11 years old. Indeed, we are currently investigating whether a scoring correction can be implemented or other modifications can be made to the AMFAST that make it suitable for younger participants.
In addition, it was somewhat surprising that the older adults in our healthy control sample did not differ significantly from the younger groups given that certain domains evaluated within the AMFAST (e.g., processing speed) are often affected by aging (Eckert, Keren, Roberts, Calhoun, & Harris, 2010; Lu et al., 2011). One possible explanation for this finding is that our healthy control sample was relatively small (n = 186); in particular, our oldest group (70+ years) was underrepresented in comparison with other groups (n = 12). Thus, insufficient statistical power could explain the lack of significant differences between some of our age groups. Another potential explanation for why AMFAST scores were not reduced in our older adult groups is that the cutoffs used for scoring are so lenient that they only pick up on truly aberrant performances that are beyond the scope of normal aging. Indeed, the scoring system of the AMFAST (particularly for tasks with a processing speed component) was specifically developed to account for normal variability and only penalizes impaired performance. It was by design that respondents only begin to lose points after hitting a certain impairment threshold, and overall performance outcomes (i.e., total AMFAST scores) are intentionally only affected after performances fall outside the bounds of this lenient threshold. Thus, while there may have been relative declines in certain domains such as processing speed in our older participants, this would not have been reflected in their AMFAST scores unless their performances were truly aberrant. This is akin to the subtle declines associated with normal aging (referenced above) and their limited impact on certain functional outcomes such as driving ability. For example, while processing speed may decline mildly with age, the vast majority of older adults continue to drive without negative consequence. In fact, studies have consistently shown that older groups (aged 50-79 years) are among the safest drivers on the road and have been found to have lower crash rates than younger groups (Tefft, 2012, 2017). Thus, despite the fact that their processing speed/reaction times may be slightly reduced compared with younger adults, there is no concern for identifying older drivers as being significantly different and/or less capable than younger drivers until there is evidence of actual impairment.
Although the overall findings are encouraging, several limitations of the present study warrant mentioning. First, it is important to note that in the present study, we considered participants to be impaired on comprehensive neuropsychological testing when at least 25% of their scores on measures of attention, memory, processing speed, and executive function fell at least 1.5 standard deviations below the normative mean (i.e., <7 percentile or T < 35). While there is a precedent for this method of defining cognitive impairment (e.g., Beauchamp et al., 2015), it is possible that this standard did not capture participants with more subtle deficits. Given that one of the intended applications of the AMFAST is to identify persons who would benefit from neuropsychological consultation, it may be worth investigating whether a more inclusive definition of impairment is indicated, and this is something that we intend to examine in the future. After all, the benefits of ultimately identifying cognitive impairment far outweigh the risks of additional testing. For the same reason, it may also make sense for clinicians to utilize a higher AMFAST cutoff score if the purpose of administration is to decide whether to refer their patients for more comprehensive assessment. In such cases, we would recommend using an AMFAST score of 73, which, although less specific than the optimal cutoff of 70, had 100% sensitivity at identifying cognitive impairment in our study.
A second limitation of the present study is that the neuropsychological measures utilized within the comprehensive clinical battery were not consistent across participants. This is partly due to the fact that distinct cognitive measures have been validated for different age groups (i.e., pediatric, adult, and geriatric). For example, in the realm of intellectual functioning, the WAIS-IV (Wechsler, 2008) is marketed for adults, while the WISC-V (Wechsler, 2014) is specifically indicated for children. Similarly, the NAB (White & Stern, 2001) is a comprehensive neuropsychological battery developed for adults, while the NEPSY-II (Korkman et al., 2007) is a similar comprehensive battery for children. In addition, there were clinically driven reasons for utilizing different measures across participants (e.g., clinical referral question and using consistent norms throughout the battery). Moreover, in general, a consensus within the field has been established that geriatric batteries are shorter by nature than other batteries and are specifically tailored to forming an efficient determination of differential diagnoses for conditions typically seen in older adults (e.g., Alzheimer’s disease, vascular dementia, and geriatric depression). As such, the total number of tasks administered within geriatric batteries is usually smaller than the number of tasks administered within a pediatric/adult battery. With that in mind, attempts were made to keep the clinical batteries as comparable across participants as possible.
A third limitation of this study is that our clinical sample was heterogeneous and was composed of individuals with a multitude of neurological, psychiatric, and other medical conditions across the life span. Given that our sample sizes for many of these conditions were small, we did not have sufficient power to determine whether the AMFAST is more effective at identifying impairment in certain conditions versus others. In the future, we aim to investigate the AMFAST’s diagnostic efficacy in more homogenous populations, particularly in specific conditions in which attention, processing speed, memory, and EF are especially affected (e.g., HIV, MS, SCD, SLE, and ADHD). We also intend to examine different profiles within the AMFAST (e.g., more impairment in memory versus more impairment in attention/EF), as we believe that certain profiles may be diagnostically revealing.
It is important for practitioners to keep in mind that the AMFAST was specifically designed to screen for attention, processing speed, memory, and EF deficits. While we believe it will be a useful tool to screen for cognitive impairment in a wide variety of neurological, psychiatric, and general medical conditions across the life span, particularly those that are characterized by frontal-subcortical dysfunction, it should not be viewed as a substitute for comprehensive neuropsychological testing and was not intended to quantify or characterize specific attention, processing speed, memory, and EF profiles. Instead, it was designed to help clinicians identify appropriate cases to refer for further evaluation. This is important because not all referrals are appropriate for full assessments (e.g., not all ADHD cases require neuropsychological evaluation for diagnostic purposes), and the AMFAST is meant to assist in triaging cases (e.g., cases in which is ADHD diagnosed based on Diagnostic and Statistical Manual of Mental Disorders–Fifth edition symptoms plus clearly documented impairment on the AMFAST). In terms of implications specific to pediatric populations, it is our experience that children/adolescents with medical conditions that place them at risk for cognitive impairment are often referred for neuropsychological testing when it becomes known that they are experiencing difficulties in school. We believe that the AMFAST could help practitioners determine whether underlying neurocognitive dysfunction (i.e., problems in attention, processing speed, memory, and EF) is contributing to their patients’ academic struggles. In the event that the AMFAST reveals impairment, a referral for more extensive neuropsychological testing would certainly be indicated in order to tease apart whether their problems are secondary to neurocognitive deficits or whether they might be due to other factors (e.g., speech/language disorder, learning disability, and/or emotional/behavioral causes).
Furthermore, the AMFAST may not be the most appropriate screening measure for disorders in which impairment in other cognitive areas (outside of attention, memory, and EF) is prominent and/or expected. In fact, deficits in these other cognitive domains may actually confound AMFAST test results. For instance, an examinee with extensive visuospatial impairment may struggle on the Spatial Memory subtest of the AMFAST due to a problem with visual perception/visuoconstruction rather than memory, per se. Similarly, although the language burden required for the AMFAST is relatively low, individuals with severe receptive or expressive language deficits may have trouble understanding task instructions and/or providing verbal responses.
In the same vein, the AMFAST, like other screening measures, may not be the best tool for detecting clinically significant impairment individuals with long-standing premorbid intellectual or developmental disabilities. Theoretically, it could be predicted that those with low intellectual functioning at baseline would score below a 70 on the AMFAST, but low scores (technically below the established cutoff) would not necessarily be indicative of a change in ability nor would it necessarily be a reason for concern. On the flip side, a score in the 70s, while technically above the determined cutoff, may represent a clinically significant decline in an individual with above-average premorbid functioning. As such, it is critical to utilize clinical judgment when choosing to administer and subsequently interpreting AMFAST scores, particularly in populations that were not specifically included in the normative sample in this study.
With the aforementioned caveats in mind, the present study demonstrated that the AMFAST is a reliable and effective measure that can be used as a brief screening tool to help identify the presence of attention, processing speed, memory, and EF deficits in diverse patient populations across a large portion of the life span. Going forward, it will be important to replicate these findings in other samples in order to ensure that they are generalizable. Moreover, a goal for future research is to develop alternative forms of the AMFAST so that it can be used for serial evaluations over short periods of time without concern of practice effects (e.g., preoperatively and postoperatively; pretreatment and posttreatment). Additionally, another target for future research is to directly compare the AMFAST with other available screening measures that are commonly utilized (e.g., MoCA) to evaluate their respective sensitivities/specificities in specified clinical populations. Finally, we aim to translate the AMFAST into multiple languages and then validate these versions in distinct clinical samples.
In conclusion, given the AMFAST’s quick and easily accessible properties, this novel tool can be implemented in a variety of settings such as helping clinicians determine whether referrals for more comprehensive neuropsychological evaluations are indicated. The AMFAST, including administration and scoring instructions (Supplemental Material), can be downloaded for free (https://www.amfast.org/).
Supplemental Material
AMFAST_Scoring_and_Instructions – Supplemental material for Validation of the Attention, Memory, and Frontal Abilities Screening Test (AMFAST)
Supplemental material, AMFAST_Scoring_and_Instructions for Validation of the Attention, Memory, and Frontal Abilities Screening Test (AMFAST) by Bryan M. Freilich, Nicole Feirsen, Elise I. Welton, Wenzhu B. Mowrey and Tamar B. Rubinstein in Assessment
Footnotes
Acknowledgements
The authors would like to thank Cristina Fernandez-Carbonell, Daniel Saldana, Amanda Bono, and Amanda Zwilling for their assistance with data collection.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
