A Novel Method for Direct Assessment of Everyday Competence Among Older Adults

INTRODUCTION

The number of adults aged 65 and older has increased to about 45 million people, and is projected to more than double by 2060. The numbers of those aged 85 or more is projected to triple from approximately 6 million in 2013 to nearly 15 million in 2040 [1]. Population aging, especially the increase in the “oldest old,” has resulted in a tremendous interest in understanding age-related changes in cognition and the implications for everyday functioning. As evidenced by the recent Institute of Medicine (IOM) Report, Cognitive Aging: Progress in Understanding and Opportunities for Aging [2], cognitive aging is an important public health issue. The genesis of this concern relates to individual and societal fears that normative age-related changes in cognition (e.g., declines in abilities such as working memory) and other types of cognitive changes such as mild cognitive impairment (MCI) will impede a person’s ability to perform activities essential to independent living, increase the need for care and support of older people, and diminish the quality of life for both older adults and families [2]. Although some controversy remains surrounding the construct of MCI, the prevalence of MCI is increasing globally [3].

Living independently requires the ability to carry out basic Activities of Daily Living (ADLs) such as bathing and eating; as well as Instrumental Activities of Daily Living (IADLS) such as managing finances; medications and other health-related tasks (e.g., following a doctor’s orders); finding one’s way; and maintaining a household. Mynatt and Rogers [4] discussed the importance of performance on enhanced activities of daily living (EADLs), which refers to the ability to adapt to changes in the environment and one’s routine and learn new ways of doing things (e.g., scheduling a doctor’s appointment using a patient portal). EADL types of activities are becoming increasingly important to independence in today’s technology driven environment.

A person’s ability to perform activities essential to daily living is generally referred to as everyday competence [5, 6]. We recognize that everyday competence is complex and involves an interaction among an individual’s physical, cognitive, emotional, and social functioning [5] and importantly, that behaviors occur within a physical and social environment. Our focus is on everyday cognitive competence. The literature [e.g., 7–12] clearly indicates that cognitive abilities are important to the performance of daily activities, especially those that are complex or involve new learning such as learning to use new technologies. In fact, studies have shown that declines in cognitive abilities place older adults at greater risk than younger adults for experiencing decrements in making financial decisions [7, 8]; medication management [9]; driving performance [10]; performance on technology-based tasks [11, 12]; and using patient portals of electronic medical records [13].

Based on the recent consensus definition [14], MCI is generally recognized as a pathological condition with varied forms (e.g., amnestic, multiple-domain), distinct from dementia, where there is some level of cognitive impairment and subjective memory complaints but performance on functional activities remain largely intact. However, although the results of outcome studies have been variable, depending on the assessment strategies and the selection of participants, recent findings indicate that MCI does result in impairments in functional performance. For example, Burton and colleagues [15] found that individuals with MCI performed more poorly than cognitively intact older adults on IADLs. Similarly, Aretouli and Brandt [16] found that individuals with MCI had more difficulties performing everyday activities than older adults without cognitive impairments. Developing strategies to enhance the everyday competence of older adults (both with and without cognitive impairments), and their ability to perform activities essential to daily living requires understanding the types of challenges they experience when performing these activities.

It is generally known that traditional measures of cognitive abilities are not adequate in terms of predicting a person’s ability to perform everyday tasks. Most assessment strategies for everyday competence are typically in paper and pencil format, are self-report, involve an informant (e.g., relative or clinician), or require in-person administration by a trained professional. For example in the Burton study [15], participants completed self-report measures of IADL performance [17] and the Everyday Problem Solving Test [18], a paper and pencil measure of IADL activities. Performance was also assessed by informants’ ratings. Ratings of participants’ performance were also obtained from relative informants. In the Aretouli and Brandt study [16], two informant-reported ratings of daily functioning were used, The Activities of Daily Living-Prevention Instrument [19] and The Informant Questionnaire on Cognitive Decline in the Elderly [20]. While these measures provide insight into a person’s performance abilities, there are documented limitations with these types of measures such as rater bias, difficulty in identifying informants, and additional costs related to the the inclusion of informants [21]. To overcome these shortcomings, performance-based measures such as the Revised Observed Tasks of Daily Living (OTDL-R) [22] which includes nine tasks, representing medication use, telephone use, and financial management, have been developed. Performance on the OTDL-R is related to performance on paper and pencil measures of everyday abilities and measures of cognitive function. Similarly, the Naturalistic Action Test (NAT) [23] is a standardized performance-based measure that evaluates ADL performance across several tasks: preparing toast and coffee, wrapping a gift, and packing a lunch box and a school bag. The NAT was developed for use with brain injured and stroke patients; however, Giovannetti and colleagues [24, 25] have used the NAT to understand performance deficits in patients with mild dementia and to guide interventions to remediate these deficits. While performance-based measures, such as the OTDL-R and the NAT are more ecologically valid, they require administration by trained observers and may not capture the complexity of current everyday activities or activities that are necessarily relevant with respect to older adults and everyday functioning.

This study evaluated the psychometric properties of three major subtests of a new computerized functional capacity assessment battery in older adults with aMCI, the UM Computer-Based Functional Assessment Battery (UMCFAB) (developed by Czaja and colleagues) which assesses performance on a variety of everyday activities including: ATM Banking/Financial Management, Prescriptions Refill via Telephone/Voice Menu System, and a Doctor’s visit (understanding medication and follow-up visit instructions). The UMCFAB has previously shown good-moderate test-test reliabilities, concurrent and discriminative abilities among older adults with persistent mental illness and cognitively normal elders [26]. In the current study, we examined: criterion validity by comparing performance on this battery relative to in-person standardized assessments of cognition and sensitivity to group differences between individuals with aMCI and non-cognitively impaired older adults. We also report test-retest reliability for sample of aMCI patients. Test-retest reliabilities for non-impaired older adults are reported elsewhere (see Czaja et al. [26]).

METHODS

Measures

2.2.1 UM computer-based functional assessment battery

The functional tasks developed for the project included a: money management (Automatic Teller Machine, ATM); an interactive telephone menu/prescription refill task; and a doctors task where the participant had to interpret and remember medication information and instructions for a follow-up visit provided by a physician. The ATM task was a replication of a current operational ATM system. The prescription refill task was a simulation of a local chain pharmacy. The tasks are available in English and Spanish, in multi-media format and include graphic representations, video, text, and concurrent speech instructions. They are delivered in person on a touch screen or mouse format in a PC environment. Sample screens for the ATM task are presented in Fig. 1. Each task consisted of subtasks with sequential demands. For the ATM task, participants were sequentially instructed to check the balance in their savings account, transfer money between their checking and savings accounts, and to withdraw cash from their savings account. For the Prescription task, they had to call the pharmacy (using a telephone keypad on the screen), refill two different prescriptions (the pill bottles appeared on the screen), and request a pickup time and date. The Doctor’s task involved participants watching a video of an actor playing the role of a physician providing medical information, information about required medications, and appointment information. The participants viewed two different vignettes. The first vignette involves medication management where participants are advised about a medication regime for five fictitious medications and then required to correctly dispense pills into a weekly medication organizer and answer queries related to dosage and timing of medications and special instructions (e.g., food). The second vignette focuses on scheduling a follow-up appointment. Following delivery of instructions by the physician, participants respond to a set of queries about preparing for the next visit.

OPEN IN VIEWER

Fig.1

Sample screens for the ATM task.

Performance metrics for all three tasks include a rate measure (total correct/time), which reflected a measure of task efficiency. Performance metrics also included errors, total correct and task completion time for each of the tasks.

Prior to implementation, in each study the tasks were piloted tested with CN and MCI older adults using standard usability protocols developed by Czaja and colleagues for older adults and technology-based tasks [40].

RESULTS

DISCUSSION

The current investigation represents the first study to investigate the use of of a computerized performance-based functional assessment battery in older adults with aMCI. The subtasks represented in the battery were actual representations of real world systems and thus assess a person’s ability to actually perform these tasks. We choose these subtasks for inclusion in the study as the ability to perform financial and medication management tasks is critical to independence.

Overall, we found that performance on our battery (The ATM, Doctors Task, and Prescription Refill Task of the UMCFAB) successfully discriminated between aMCI and cognitively normal older adults. Moreover, UMCFAB tasks that were studied demonstrated good test-retest reliabilities and adequate concurrent validity with cognitive measures in those with aMCI. In addition, administration of this type of battery was acceptable with this population. All of the participants were able to use the software to perform the tasks.

The best measures on the UMCFAB battery were those of efficiency calculated as a rate measure by the formula Rate = (Correct Responses/Total Time). We found that better scores on Trails A and B, tests of simple and more complex visual scanning and psychomotor speed were most highly related to both rate on the ATM and Prescription Refill tasks. In contrast, a higher rate of efficiency on the Doctor’s Task was related to better Trails B and Block Design Scores. While these are both speeded tests, Block Design also taps visuospatial skills and non-verbal reasoning abilities. Interestingly, none of the memory or language measures were related to performance on the CFAB functional measures.

This expands upon the work of Czaja et al. [26] that indicates that the UMCFAB has good to moderate test-retest reliabilities and good concurrent validity among cognitively normal older adults and older adults with persistent mental illness such as schizophrenia.

Although MCI has been traditionally defined as the lack of functional deficits that interfere with everyday life, this is often a clinical judgment in which the practitioner does not have the ability to to directly observe higher order skills such as that assessed by the UMCFAB. Although MCI diagnostic criteria exclude substantial deficits [32] that presumably affect independent daily living; research has identified that mild but significant functional deficits are present in individuals diagnosed with MCI across a number of domains [33–35]. Previous research has suggested that deficits in remembering appointments and medications, and organizing tax records or other business affairs, appear to be present in approximately 40 to 55% of individuals with aMCI [36]. This is despite the fact that these individuals are deemed functionally independent by examining clinicians. Clearly, direct assessment of functional ability is essential for both early diagnosis of MCI and longitudinally tracking the progression of disease [37].

There are various ways to assess functional abilities. Commonly used measures of functional abilities typically rely on inventories completed by patients and their informants or paper and pencil approaches. Although there is value in these approaches, it is important to note that self-reports and even the reports of collateral informants in AD and other neurodegenerative may be biased [38]. There are instances when a collateral informant has had limited opportunity to observe the patient performing the activities that are being assessed, has a cognitive or psychiatric disability themselves, or is governed by unrecognized biases that can lead to inaccurate evalautions. Thus, their report may be of little value in providing useful functional information.

Paper and pencil measures of functional abilities such as the Performance-Based Skills Assessment [39], the Everyday Problem Solving Test [18] are also commonly used and have associated strengths and weaknesses. While these types of measures are convenient and relatively easy to administer, they require individuals to perform in tasks in a role-play format that does not capture the complexities that exist in the everyday world. Further, these measures do not reflect how many of the tasks assessed by the measure are currently performed in today’s technology oriented world [40]. Another method of assessing functional capacity is direct observation of the patient performing behaviors in a clinical setting using highly trained and reliable observers. This assessment method also yields important insight into a person’s capacity but generally requires trained professionals, which can be costly and logistically problematic. In addition observing behaviors in a clinical setting does not capture the complexity of the real world or tap actual real-world functional behaviors that may be aided by compensatory strategies and rely contextual cues within the environment.

For these reasons, actual computer-based simulations of real-world functional tasks comprising the UMCFAB such as using a similar ATM machine used by banks, a Prescription Refill task that is analogous to telephone automated systems increasingly employed by pharmacies across the country and an actual real-world simulation of a Doctor’s task, which requires an individual to comprehend physician instructions, makes the UMCFAB battery a valuable mechanism for identifying and tracking real-world functional deficits over time. In addition to the excellent discriminative ability shown in the current aMCI and cognitively normal older adult samples, recent research has also demonstrated that the UMCFAB is highly sensitive to deficits in older adults with severe mental illness such as schizophrenia [26]. Of note, our findings that 40% of older adults with aMCI had difficulties with commonly encountered tasks such as an ATM machine and over a quarter of individuals had difficulties comprehend instructions on the Doctor’s Task indicate that subtle performance-based cognitive deficits can be detected utilizing technology-based direct functional assessment. The UMCFAB battery has also been translated into Spanish and can also be easily expanded to include other tasks and translated into other languages.

In addition, the UMCFAB employs a sophisticated computer interface with simulations of actual real world activities, is well tolerated by older adults and does not necessitate a skilled examiner. Since the battery records responses in real time (in milliseconds) and is completely computer-based, these functional measures can be delivered and automatically scored over the Internet remotely. Thus this battery can be easily administered to older adults who live in remote areas and do not have access or transportation to remote tertiary medical centers. Further, by employing virtual reality and video technology, the CMCFAB tasks have been developed that realistically simulate everyday functions. This is particularly relevant given older adults are increasingly required to interface with technology such as financial transactions with an ATM machine, online bill paying, using increasing automated prescription refill services at pharmacies, and engaging in Internet-based information-seeking and purchasing of goods and services.

In summary, the UMCFAB computer-based functional tasks were well tolerated by older adults with and without aMCI and demonstrated excellent discriminative ability. These results and previous studies of older adults with chronic mental health conditions (see Czaja et al. [26]) indicate that performance-based computer simulations of real-world tasks may be important tools with respect to assessing functional performance. This is especially true in cases where there is not a reliable informant who has an opportunity to observe functional performance or may have biased estimates of performance or the older adult is unable to access a clinical setting. Overall, the results of this study confirm the potential importance of performance-based computerized functional assessment and these measures are worthy of further research.

There are some limitations associated with the current investigation. In this study, we had relatively modest numbers of participants, especially those who were non-impaired, who were followed cross-sectionally. We plan on following these and additional participants over a longer time period to determine the extent to which biological markers (such as MRI) and novel and traditional cognitive measures can track functional decline using CMCFAB measures. We are also studying the properties of CMCFAB tasks in different ethnic and cultural language groups. The battery included in this study was also limited to three subtasks. We are expanding the array of CMCFAB tasks available to include a wayfinding task and an online banking task.