Scoping review of the Multiple Errands Test: Is it relevant to youths with acquired brain injury?

Abstract

Introduction

For youths who sustain acquired brain injuries, distinguishing typical development of executive function from the impairment(s) can be a challenging but critical assessment consideration. Occupational therapists working with individuals after brain injury can use the Multiple Errands Test as a performance-based assessment of the effect of executive dysfunction in the real world. Although numerous test versions exist for different settings and diagnostic populations, their relevance to youths is unclear. We surveyed the non-virtual reality literature for test versions to determine the measurement properties and clinical utility for assessing youths in a community setting.

Method

A scoping review was completed to summarize study purpose/design, Multiple Errands Test structure, assessment environment, sample characteristics, psychometric properties, clinical utility and main findings of the test versions.

Results

We included 10 quantitative studies and found the strongest reliability and validity evidence for male adults with moderate to severe acquired brain injury, in a hospital setting. Multiple Errands Test versions can distinguish people with brain injury from controls and identify challenges in the home and community. No studies examined the test performance of younger participants.

Conclusion

This review highlights the research of several Multiple Errands Test versions and identifies gaps in that research, including the need for development of a test version for youths in a community setting.

Keywords

Occupational therapy executive function performance-based assessment brain injuries adolescent young adult youths

Introduction

Worldwide, brain injury is a leading cause of death and disability, with incidence of pediatric traumatic brain injury (TBI; age ≤18 years), ranging from 47 to 280 per 100,000 (Dewan et al., 2016). TBI is one type of acquired brain injury (ABI), and the overall statistics are not known (Toronto ABI Network, 1999). ABI can result in a range of physical, cognitive, emotional and behavioral impairments, and is associated with permanent disability, learning disabilities, chronic mental illness, homelessness and incarceration (Colantonio et al., 2014; Dams O’Connor et al., 2014; Ewing Cobbs et al., 2004). High rates of TBI occur in adolescence and the incidence is greater in males (Dewan et al., 2016).

Late adolescence through early adulthood is known to be a critical developmental stage for youths (the term ‘youths’ encompasses late adolescence into young adulthood, defined as under 24 years of age) and of the frontal lobe regions of the brain supporting executive function development (Sowell et al., 2004). Executive function includes higher order cognitive processes that coordinate and integrate lower order cognitive processes to enable goal-directed behavior (Cicerone et al., 2006; Stuss and Alexander, 2007). Lower order cognitive functions include planning, organizing, problem-solving, decision-making, impulse control, cognitive flexibility and goal selection (Hunt et al., 2013). New and complex task performance (in areas such as academic, work and driving) required for adult life roles, independent living and employment are reliant on executive function. Unlike adults, youths have yet to fully develop their cognitive functions, including executive function (Taylor et al., 2013), and have limited pre-injury experience using intact executive function to draw upon after brain injury.

The confluence of developmental phase and executive dysfunction from brain injury creates a disadvantage for youths and has implications for the remainder of their adult life. An additional challenge with this population is that when a brain injury occurs in a youth, it can result in deficits that are difficult to discern from brain functions that are still developing. As such, the consequences of brain injury in youths are often underrecognized, misunderstood by professionals and family members and misinterpreted in the context of continued growth and development (Savage et al., 2005). It is crucial that rehabilitation professionals, working with this subset of brain injury survivors, are able to accurately evaluate and treat these deficits. Of primary concern is generating and using high-quality evidence to inform professional judgements concerning functional outcomes.

When evaluating executive function in youths, a clear understanding of typical development is needed, so that the problems that arise from brain injury can be identified and distinguished from normal development. Researchers studying the development of executive function during typical adolescence and young adulthood have found overall improvement in executive function ability on traditional tests of executive function, but the evidence also indicates that development of executive function is variable, with some aspects maturing at younger ages and others later on. A few studies have examined executive function in neurologically intact youths. Taylor et al. (2013, 2015) used traditional neuropsychological tests of executive function with 17- to 19-year-olds and found non-linear development with periods of improvement, stability and decline in function within this age range. Toglia and Berg (2013) established a baseline of performance for typical youths (age 16–21 years) using the Weekly Planning and Calendar Activity, a performance-based assessment of executive function. These results, although informative, have yet to be correlated with real-world executive functioning.

Evaluation of residual cognitive deficits and impact on function for real-world performance is an essential part of brain injury rehabilitation and within the scope of occupational therapy practice. Evaluation of executive function has typically used traditional neuropsychological tests done in an environment free of distraction or interruption. Explicit tasks are completed in short duration and do not tax the executive processes in a similar manner to real life, which is inherently more complex (Chan et al., 2008; Dawson et al., 2009; Shallice and Burgess, 1991). In contrast, The Multiple Errands Test (MET) is a naturalistic measure of the impact of executive function on everyday life through the observation of an individual’s performance of an activity in a real-world setting. The ability to capture contextual influences contributing to task performance makes the MET a useful tool for occupational therapists.

The MET takes place in real-world settings (shopping mall, hospital lobby) and therefore several site-specific versions of the MET have developed over time. Alderman et al. (2003), in the early stages of MET research, identified that ‘others should be able to adapt it (the MET) for their own environment’ (p. 32). Site-specific versions of the MET differ in the testing environment, specific task components and scoring.

To date, there has been no published review of the site-specific versions of the MET to summarize the extent, range and nature of research activity, or to guide therapists wanting to use or add to the development of a version of the MET in practice. Additionally, use of the MET with the community dwelling, youth population and gaps in the MET were of interest. The objective of this literature review is to determine if the MET is relevant for use with the youth population in a community setting through evaluation of its clinical utility, measurement properties and study population characteristics.

Method

A scoping review was selected as a suitable methodology as defined by Colquhoun et al. (2014), given it ‘is a form of knowledge synthesis that addresses an exploratory research question aimed at mapping key concepts, types of evidence, and gaps in research related to a defined area or field by systematically searching, selecting, and synthesizing existing knowledge’ (p.1292–1294).

The scoping literature review was completed following the Arksey and O’Malley (2005) framework and the recommendations of Levac et al. (2010). The five stages of the scoping review were followed: identify the research question, identify relevant studies, study selection, charting the data and collating, summarizing and reporting the results (Levac et al., 2010).

Stage 1: Identifying the research question

The research question: What does the research tell us about the non-virtual reality versions of the MET and their clinical utility and measurement properties for measuring executive function in the youth population in a community setting?

Stage 2: Identifying relevant studies/literature search

The following scientific literature databases were searched: PubMed, CINAHL and PsychINFO. Additionally, the following grey literature databases were searched: BASE, CogPrints and Open Grey. The search terms: Multiple Errands Test or, brain injuries* AND executive function* AND clinical assessment tools or outcome assessment health care* were used. The asterisk indicates that the term is a MeSH term. For this review, the Toronto ABI Network’s definition of ABI was used and is defined as is an injury to the brain occurring after birth, which is not hereditary, congenital or degenerative; it can be caused by a traumatic blow or injury to the head (TBI), anoxia or a tumor. Hand-searching of the reference lists of included articles were reviewed to identify additional relevant articles. The search yielded 476 articles across the six databases, which was reduced to 423 when duplicates were removed.

Stage 3: Study selection

The literature search strategy is outlined in Figure 1 using the preferred reporting items for PRISMA flow diagrams. The original study of the MET by Tim Shallice and Paul Burgess was published in 1991 (Shallice and Burgess, 1991), therefore all articles published after 1991 were included. Additional inclusion criteria: peer reviewed, journal articles published in English, specific to a sample with ABI that focused on MET tool development, and validation. Two thesis dissertations within the grey literature search were excluded as no related studies published in a peer-reviewed journal were found. One published poster was excluded as insufficient details were provided to answer the research question. Studies were excluded if the MET was used for qualitative research, to study treatment effectiveness, the sample population was not ABI, the virtual reality MET version was used and focus was on alternate uses of the MET other than to assess executive function.

Figure 1.

PRISMA flow diagram (Moher et al., 2009).

Records were screened for inclusion/exclusion criteria using titles and abstracts, duplicate articles were removed and a total of 19 full-text articles were remaining. After hand-searching the reference lists of the 19 full-text articles, three additional related articles were located. When inclusion/exclusion criteria were applied none of these articles was retained. Full-text articles were then screened for inclusion/exclusion criteria, yielding 10 articles eligible for the review. The second author was involved in determining study inclusion and exclusion and verified the final study selection.

Stage 4: Charting the data/data extraction

The 10 eligible articles, all quantitative studies, were reviewed and relevant study information was extracted in to Table 1. Table 2 (online only at https://journals-sagepub-com.web.bisu.edu.cn/doi/suppl/10.1177/0308022618791714) provides a summary of the psychometric properties and methodological quality per study reviewed. The Data Extraction Form and Guide for Studies Evaluating the Clinical Measurement Properties of Outcome Measures (Law and MacDermid, 2014) was used to identify measurement properties and relevant statistics. Standards for reliability and validity were taken from criteria provided in Law and MacDermid (2014) and Portney and Watkins (2009). The consensus-based standards for the selection of health measurement instruments (COSMIN) checklist (Terwee et al., 2011) was used to review the methodological quality of each psychometric property in the scoping review. The four-point rating scale (excellent, good, fair, poor) quality was applied to each measurement property in each study. The following COSMIN measurement properties are not in Table 2 (online only at https://journals-sagepub-com.web.bisu.edu.cn/doi/suppl/10.1177/0308022618791714) because no evidence of the measurement properties was found in the included studies: measurement error, criterion validity, cross-cultural validity and responsiveness. Also, a single study reported on the ceiling effect, which COSMIN classifies an interpretability issue, not a psychometric property. COSMIN guidelines for overall measurement property score is obtained by taking the lowest score for any of the criteria (worst score counts method). Table 3 (online only at https://journals-sagepub-com.web.bisu.edu.cn/doi/suppl/10.1177/0308022618791714) provides a summary of MET main findings, clinical utility, study strengths and implications from the scoping review synthesis.

Table 1.

Descriptive summary of articles reviewed (in date order).

Authors	Design	Purpose	MET version	Test location	MET requirements and scoring	Sample/generalizability
Authors	Design	Purpose	MET version	Test location	MET requirements and scoring	Participants	Participant residence	Diagnosis severity	Mean age (range)	Males /total sample
Shallice and Burgess (1991)	Case study	Develop a performance-based assessment of executive function with open-ended, multiple subgoal requirements for patients with TBI	MET	Shopping mall	11 subtasks: – six simple tasks (such as buy items) – meet at designated time/location after 15 min – four pieces of information Six rules Scoring: four error subcategories defined; tally of frequency of errors in each error category	Dx = 3 CG = 9	C	TBI	Dx = ^a (27–55) CG = 40 (24–63)	Dx = 2/3 CG^a
Knight et al. (2002)	Cross-sectional	Develop and explore the validity of a hospital version of the MET	MET-HV (Hospital Version)	Hospital	12 subtasks: – buy three items – collect an item – use the telephone – mail an item – four pieces of information – meet at designated time/location and – tell assessor the time Nine rules Scoring: four error subcategories defined; tally of frequency of errors in each error category	Dx = 20 CG = 20 (hospital staff)	H	TBI = 12 CVA = 5 CVA and TBI = 3 Severe to very severe	Dx = 36 (20–53) CG = 36 (23–55)	Dx = 17/20 CG = gender matched with Dx
Alderman et al. (2003)	Cross-sectional	Develop and explore the validity of a simplified version of the MET Examine relationship between different types of errors on MET-SV to determine the subcomponents underlying multitasking performance	MET-SV (Simplified Version)	Shopping mall	12 subtasks: – buy six items – four pieces of information – meet at designated time/location and – tell assessor the time Nine rules Scoring: four error subcategories defined; tally of frequency of errors in each error category	Dx = 50 CG = 46 (hospital staff and associates)	B	TBI = 39 CVA = 9 Cerebral tumor resection = 2 very severe	Dx = 35 (18–59) CG = 29 (21–58)	Dx = 41/50 CG = 23/46
Tranel et al. (2007)	Cross-sectional	Replicate and extend the finding of Shallice and Burgess (1991) using a closely adapted version of the MET with a group with focal, stable damage to the VMPC contrasted with PFC, non-PFC and CG	MET (adapted by authors)	Shopping mall	Same 11 subtasks as MET but site- specific items and information substituted – six rules Scoring: four error subcategories defined; tally of frequency of errors in each error category – 5 summary scores calculated	Dx VMPC = 9 Dx PFC = 8 Dx non-PFC = 17 CG = 20	a	CVA = 6 VMPC CVA = 5 Benign tumor = 4 PFC Resection of arteriovenous malformation = 1 Benign tumor = 1 non-PFC CVA = 11 resection = 3 temporal lobectomy = 2 encephalitis = 1 Severity not reported	Dx VMPC = 51^a Dx PFC = 51^a Dx non-PFC = 51^a CG = 54^a	Dx VMPC = 5/9 Dx PFC = 5/8 Dx non-PFC = 9/17 CG = 8/20
Manes et al. (2009)	Cross-sectional	Investigate everyday executive function abilities in patients with focal cerebellar lesions using assessments sensitive to damage to the prefrontal cortex including real-life tasks (MET)	MET-HV (adapted by authors)	Hospital	12 subtasks – buy three items – collect an item – use the telephone – mail an item – three pieces of information – phone test proctor at designated time and tell assessor the time – inform proctor when tasks are completed Nine rules Scoring: four error subcategories defined; tally of frequency of errors in each error category and total of all four subcategories	Dx = 11 CG = 11	^a	Cerebellar lesion = 11 Severity not reported = 11	Dx = 51 (28–69) CG age matched to Dx	Dx = 9/11 CG gender matched to Dx
Dawson et al. (2009)	Cross-sectional	Further establish the psychometric properties of the MET, clinical utility, inter-rater reliability and develop a Baycrest site-specific version Standardize scoring of the instrument for increased ease of use Determine a summary score to discriminates between community-dwelling ABI group and matched controls	BMET (Baycrest site)	Hospital	12 subtasks – buy three items – collect an item – use the telephone – mail an item – four pieces of information – meet at designated time/location and tell assessor the time – tell assessor when finished Eight rules Scoring: errors identified and defined ahead of time in the scoring 11 summary scores calculated	Stroke Dx = 14 CG = 13 TBI Dx = 13 CG = 12	C	Stroke mild to moderate TBI moderate to severe Severity unknown for 2 TBI and 5 stroke participants	Stroke Dx = 59 (33–80) CG = 57 (27–81) TBI Dx = 43 (26–58) CG = 46 (22–57)	Stroke Dx = 8/14 CG = 7/13 TBIDx = 11/13 CG = 10/12
Maeir et al. (2011)	Prospective/cohort	Examine the relationship of the MET at discharge from a rehabilitation hospital to participation in the community 3 months later in a group of individuals with ABI	MET-HV	Hospital	12 subtasks – buy two items – borrow 1 item – collect an item – use the telephone – mail an item – four pieces of information – meet at designated time/location and tell assessor the time – tell assessor when finished Nine rules Scoring: four error subcategories defined; total errors ranging from 0 to 30 possible errors; strategy score: tally of number of strategies used	Dx = 30	H	Stroke = 19 TBI = 8 Tumor = 3 Severity not reported	Dx = 54 (24–75)	Dx = 20/30
Morrison et al. (2013)	Prospective/ cohort	Examine discriminant validity of the MET-R among a group with mild stroke/CVA; provide an objective scoring system	MET-R (Revised)	Hospital	Not specified; based on MET and MET-HV Scoring: total time, number of locations visited, number of tasks completed and total rule breaks	Dx = 25 CG = 21	H	stroke mild	Dx = 60^a CG = 60^a	Dx = 10/25 CG = 6/21
Cuberos Urbano et al. (2013)	Cross-sectional	To examine the ability of the MET scores to predict symptoms of apathy, disinhibition and disorganized behavior in everyday life as reported by relatives on the Frontal Systems Behavior Scale and the DEX and test the reliability of the Spanish HRT-Granada MET-HV. (Spanish MET)	Spanish MET	Hospital	12 subtasks (more similar to MET-HV than BMET) Nine rules Scoring: four error subcategories defined; tally of frequency of errors in each error category and total of all four subcategories	Dx = 30	C	TBI = 19 CVA = 8 Tumor = 2 Anoxia = 1 Severity not reported	Dx = 35 (19–60)	Dx = 25/30
Clark et al. (2015)	Cross-sectional	Create a revised version of the BMET to improve its ability to distinguish performance patterns and errors in individuals with ABI versus healthy individuals; to create an alternate version for use as an outcome measure for determining efficacy of rehab approaches for impairments of executive function	BMET-R (Revised)	Hospital	BMET-R Version A – similar to BMET BMET-R Version B – minor variations to Version A in items to purchase and information to find out Revisions: 1) stopped by examiner after 12 min, asked to find a specified flyer and keep it with them; 2) find a specific room and its name; 3) specify time as 2:15 pm rather than amount of time elapsed (after 15 min) Scoring: task completion classified as completed, partially completed or omitted; total errors (frequency of task omissions, frequency of partially completed tasks and frequency of inefficient behavior); frequency of rule breaks, frequency of task-unrelated behavior, frequency of strategy use	Dx = 15 CG = 16	C	TBI = 8 moderate to severe CVA = 8	Dx = 63^a CG = 61^a	Dx = 10/15 CG = 10/16

Missing information.

ABI: acquired brain injury; BMET: Baycrest Multiple Errands Test; BMET-R: Baycrest Multiple Errands Test-Revised; C: community; CG: control group (normal); CVA: Cerebrovascular Accident (CVA); DEX: Dysexecutive Questionnaire; Dx: diagnosis group; H: hospital; HRT: Hospital de Rehabilitaciony Traumatologia; MET: Multiple Errands Test; MET-HV: Multiple Errands Test Hospital Version; MET-R: Multiple Errands Test Revised; MET-SV: Multiple Errands Test Simplified Version; PFC: prefrontal comparison; non-PFC: nonprefrontal comparison group; TBI: traumatic brain injury; VMPC: ventromedial prefrontal cortex.

Stage 5: Collating, summarizing and reporting the results

A descriptive, numerical summary was developed for each area of extracted data, followed by a thematic analysis to identify and report patterns of strength and gaps in the evidence. The results were interpreted in regard to the original research question and implications of the findings to research and practice discussed.

Results of the Scoping Review

Study purpose and design

The 10 articles reviewed were published between 1991 and 2015. The purpose of all studies related to developing and establishing the properties of various site-specific versions of the MET, two studies also examined the effect of brain lesion location on executive function. The purpose for two studies was to develop an objective scoring method to improve clinical utility (Dawson et al., 2009; Morrison et al., 2013). The original study by Shallice and Burgess (1991) used a case study design and two studies used a prospective cohort design (Maeir et al., 2011; Morrison et al., 2013). The other seven studies used a cross-sectional design focused on the clinical measurement properties of several versions of the MET.

MET requirements

The original MET was completed in a shopping mall and included 11 subtasks: six simple tasks including buying items, meeting at a designated location and time, obtaining four pieces of information and following six rules. The MET Hospital Version (MET-HV) included 12 subtasks and nine rules, with some subtasks as in the original MET (items to buy and information to find). The MET-HV specified a broader range of subtasks that were not specified in the original MET, including mailing an item and telephone use. The other site-specific versions that use a hospital test environment have been closely modeled on the MET-HV subtasks, with site-specific adaptations but with consistency in the requirements. These four versions include the MET-HV used by Manes et al. (2009), Baycrest Multiple Errands Test (BMET), MET-HV used by Maeir et al. (2011), and Spanish MET. For the Multiple Errands Test-Revised (MET-R), the requirements are not provided but identified to be based on both the MET and MET-HV. The Baycrest Multiple Errands Test-Revised (BMET-R) introduced three revisions to the task requirements made following expert consensus. The Multiple Errands Test-Simplified Version (MET-SV) that took place in a shopping mall, expanded the original MET subtasks from 11 to 12 and rules from six to nine.

Sample characteristics/age

The testing environment varies across versions of the MET, with the assessment taking place in either a hospital or shopping mall. The majority of MET studies (n = 7) identified testing in the hospital environment: lobby, main floor or designated boundaries within a hospital complex. Three studies used a shopping mall as the test environment.

There was an array of participant living arrangements reported. Four studies identified different community settings (alone, family homes, group homes), three studies identified living in the hospital or inpatient setting, one study included both inpatient and outpatients and two studies did not provide this information.

ABI severity was not consistently reported or absent. The first MET study (Shallice and Burgess, 1991) described a severe TBI classification although no formalized measure of severity was referenced. Three studies consisted of mixed etiology ABI of moderate to severe severity (Alderman et al., 2003; Clark et al., 2015; Knight et al., 2002). The BMET study included a greater range of participant severity including stroke and TBI, but some participants could not be classified (Dawson et al., 2009). The MET-R is the only study that focused on those with mild stroke (Morrison et al., 2013).

Across the 10 studies the age range was 18–81 years, with the mean age for both brain-injured participants and controls being over 29 years of age. In the majority of articles reviewed (n = 9), more than 50% of the study sample were male. The Morrison et al. (2013) study was the only study with a primarily female sample, with mild stroke.

Psychometric properties

Available psychometric properties will be summarized collectively across the articles and then reported for individual MET versions.

Reliability

Reliability refers to the ability of a measure to consistently measure the underlying concept (real-word executive function)(Portney and Watkins, 2009). Inter-rater reliability was the most commonly reported. Adequate to excellent inter-rater reliability has been reported for the MET-HV (Knight et al, 2002), MET-R (Morrison et al., 2013), Spanish MET (Cuberos Urbano et al., 2013) and the BMET- R version B (Clark et al., 2015). Conversely, the inter-rater reliability of the MET-SV has not been determined statistically. Adequate inter-rater reliability was reported for the BMET (Dawson et al., 2009) and mixed results were found for BMET-R Version A (Clark et al., 2015). Test–retest reliability was mentioned in two studies (Alderman et al., 2003; Knight et al., 2002) but the data were reported as unpublished. Knight et al. (2002) reported adequate internal consistency for MET-HV.

Validity

Validity refers to the accuracy of the measure in measuring in underlying construct (Portney and Watkins, 2009). Content validity is the extent to which the construct (real-world executive function) measured is adequately reflected by the items/structure of the measure (Law and MacDermid, 2014). It can vary across populations, so it should be established for the population with which the assessment will be used (Haynes et al., 1995). Face validity is an aspect of content validity and considers if the measure is acceptable and relevant to those who are tested by it (Portney and Watkins, 2009). The developers of COSMIN consider content validity to be the most important measurement property (Terwee et al., 2018). Content validity of the MET is evident as two experts in the field, Tim Shallice and Paul Burgess, developed the format and procedures of the original MET (Shallice and Burgess, 1991) and these authors helped to develop three additional versions: the MET-HV (Knight et al., 2002), the MET-SV (Alderman et al., 2003) and the BMET (Dawson et al., 2009). Exploratory factor analysis of the MET-SV supported a two-factor structure, rule breakers and task failers (Alderman et al., 2003), indicating that executive function is likely a theoretical construct that can be subdivided.

Construct validity has been considered across several studies, most often known groups, divergent and convergent validity. In all the studies that looked at known groups validity (n = 6), the frequency of errors made by brain-injured participants was significantly higher than those of controls for several error categories (Alderman et al., 2003; Clark et al., 2015; Knight et al., 2002; Manes et al., 2009; Morrison et al., 2013; Tranel et al., 2007).

To establish convergent validity the MET was compared to other ecologically valid test of executive function (intended to measure the same construct). Fair to moderate correlations were found between the MET-HV/MET-SV/BMET and the Behaviour Assessment of the Dysexecutive Syndrome battery and Dysexecutive Questionnaire (DEX) (Alderman et al., 2003; Dawson et al., 2009; Knight et al., 2002) and between the MET-R and the Executive Function Performance Test (Morrison et al., 2013). The MET (Spanish version) was found to correlate with the Frontal Systems Behavior Scale and the DEX. Ecological validity has been examined and strong correlations found between the BMET and tests intended to measure real-world functioning like the Assessment of Motor and Process Skills and the Sickness Impact Profile (Dawson et al., 2009).

In four studies the version of the MET demonstrated a lack of correlation (divergent validity) with traditional test of frontal lobe function and traditional neuropsychological tests of memory, IQ, and so on (Alderman et al., 2003; Knight et al., 2002; Manes et al., 2009; Tranel et al., 2007). Some correlations were found, particularly with traditional tests of frontal lobe function, such as the Wisconsin Card Sorting Test and the Modified Wisconsin Card Sorting Test (Knight et al., 2002; Manes et al., 2009), and Trail Making Test (Tranel et al, 2007). In general, the MET was found to be more sensitive that traditional methods of evaluation to executive function deficits in brain-injured individuals. Predictive validity was examined in one study (Maeir et al., 2011).

Responsiveness

Responsiveness refers to the ability of a measure to detect change over time (Law and MacDermid, 2014). The BMET-R has been developed to have two versions (Clark et al., 2015) for repeat measurement over time, but no evidence of responsiveness of the versions has been reported. The MET responsiveness was not reported in the articles reviewed.

Main findings and clinical utility

Across the majority of studies, performance on the test was able to adequately discriminate between people with and without brain injury. Generally, individuals with brain injury make more errors and different types of errors than controls. They omit more tasks, break more rules and achieve fewer tasks, less efficiently, on the MET. Overall, using the MET to evaluate executive function in the real world can strengthen an occupational therapist’s ability to identify people with executive function deficits who may otherwise go undetected on traditional tests of executive function. The MET can qualitatively and quantitatively measure the impact of executive function problems at the level of everyday functioning; social behavior and contextual influences can be observed, and performance can more directly link to rehabilitation interventions.

Administration time for the MET ranges from 45 min (Alderman et al., 2003) to 1 h (Clark et al., 2015), and some limited equipment is required and transportation to the shopping mall is assumed for the MET-SV. In all the articles reviewed the training requirements for the MET are not clearly reported. A manual is available from the authors for the BMET (Dawson et al., 2009), but for other versions clinicians are required to read the research article.

Scoring for most MET studies is based on error subcategories defined by Shallice and Burgess (1991) as (a) inefficiencies, (b) rule breaks, (c) interpretation failure and (d) task failure and on a tally of the frequency of errors in each subcategory determined by the assessor after MET performance. A cut-off score has been published for the MET-HV and for the MET-SV. The BMET scoring is different in that errors in each subcategory are defined ahead of time to reduce subjectivity; also, the MET-R scoring removed error subcategories of inefficiency, interpretation failure and task failure. Instead scoring on the MET-R is defined as total time, number of locations visited, number of tasks completed and total rule breaks, as well as performance efficiency (the ratio of tasks completed to total number of locations visited).

Methodological quality

The quality rating for most of the measurement properties in the 10 articles reviewed was either fair or poor. Most studies reported reliability, hypothesis testing/construct validity and content validity, whereas internal consistency and structural validity were covered by single studies only.

Discussion

Overall MET strength

This scoping review supports the MET as a sensitive, ecologically valid and useful tool for occupational therapists to evaluate the impact of executive function on real-world performance in adult males (mean age 29 years) with moderate to severe ABI in a naturalistic context (hospital setting). Primarily, for this population, the MET has been found to aid in the identification of executive function deficits in real-world situations and to discriminate between people with and without a brain injury.

MET and youths

There have been no specific studies to examine how the adolescent/young adult population perform on the MET, or its ability to discriminate between healthy and individuals with ABI in this age range. Research is needed to describe and evaluate executive function using the MET in neurologically intact teens and young adults (ages 16–24 years). As previously mentioned, this developmental stage is unique for executive function maturation amidst increasing contextual demands related to life stage. Traditional tests of executive function have shed some light on the changes occurring in executive function during late adolescence/early adulthood. However, having MET norms for this population could offer a performance-based assessment of executive function and provide greater understanding of typical executive function development at the performance level within contextual influences. The MET research needs to identify executive function problems that arise from youth brain injury and can be distinguished from normal development. Identification of executive function issues allow for targeted rehabilitation interventions to address these impairments, which is critical information for the youth’s support network of the impact of executive dysfunction on real life.

MET and community setting

The majority of test development and research has occurred on MET versions intended for use in a hospital testing environment (the BMET, BMET-R, MET-R and Spanish MET Hospital Version). The benefits of the various hospital versions include their utility for assessing patients with mobility issues and ease of an accessible testing environment for occupational therapists. The MET hospital version(s) task requirements include a broader range of subtasks (telephone use, mailing an item) compared to the shopping mall versions, allowing the assessor to observe a greater variety of life tasks. Overall, much less research using the MET to test executive function in a real-world community setting has been done. Aside from the original case studies, only two studies used (Alderman et al., 2003; Tranel et al., 2007) a shopping mall as a test environment. For occupational therapists working in the community, routine access to a hospital environment is neither feasible nor applicable to the assessment of community-dwelling individuals. Using a shopping mall to complete multiple errands is inherently more ecologically valid than simulating a shopping environment in a hospital lobby. As typical inpatient hospital stays become shorter and community reintegration is an expedited goal of rehabilitation, there is a need to develop community versions of the MET. Site-specific versions are also of limited use to occupational therapists without access to the sites, so generating versions of the MET that can be used more widely, like in a home environment, would improve clinical utility for community-based occupational therapists. Finally, there is no evidence the MET can evaluate change in executive function over time (treatment effectiveness, maturation or decline in status) or to predict future status (diagnosis or outcome). Research is needed to establish the responsiveness of the various versions of the MET as this will be needed to discriminate change in real-world executive functioning with rehabilitation interventions for youths and adult populations.

MET clinical utility

The clarity and standardization of scoring for an assessment tool has a large effect on clinical utility. The scoring of the MET-SV (Alderman et al., 2003) and MET-HV (Knight et al., 2002) lacked clarity and the potential for systematic error was evidenced due to the subjectivity of some areas of the scoring. Several studies (Cuberos Urbano et al., 2013; Maeir et al., 2011; Manes et al., 2009; Tranel et al., 2007) based their scoring on these early studies, therefore standardization of measurement in these studies was less than optimal. Although several revisions to scoring and administration procedures have been developed (Dawson et al., 2009; Morrison et al., 2013) to improve the standardization of the measure for hospital versions (BMET, MET-R), these revisions have not yet been integrated with the MET-SV that takes place in the community. Therefore, there is need for further MET development for community settings with integration of a revised scoring system.

Most of the studies pertaining to the measurement properties of the MET include a mostly male sample, accurately reflecting the TBI population statistics. Clinicians evaluating executive function in females should be aware of this trend in the sample characteristics of the research findings as it affects generalizability.

MET psychometric properties

This review identified areas of strength and several areas for future development pertaining to the psychometric properties of the MET. There is strong evidence for known groups validity and convergent validity with other ecologically valid tests across multiple versions. Divergent validity is evidenced across several studies with lack of correlation between the MET and traditional test of executive function, which is expected given the limitation of most traditional tests of executive function at predicting real-world performance. Minimal correlations between the MET and test of frontal lobe function have been reported and is not entirely unexpected as impaired executive function has been found in individuals with lesion to the frontal lobes.

Content validity across MET versions has been established by expert consultation, but further evidence by other methods such as consultation with the population to be tested is needed. Application of the COSMIN checklist in this review confirmed poor evidence across studies for face validity. The target population of this review is youths, therefore it will be important for future research to consult with youths regarding the requirements of the MET to establish the content and face validity of the MET for this population.

The BMET-R has introduced the most significant recent changes to the subtasks requirements as well as contributing a version A and version B for use as an outcome measure. The changes to the BMET-R were made to improve construct validity; however, results indicated that alternate versions did not identically assess real-world executive function. The authors of the BMET-R aptly expressed the challenge of future studies is to ‘develop, clear, novel, equally challenging, comparable tasks and environments.’ (Clarke et al., 2015: p.14)

The inter-rater reliability of several hospital-based versions of the MET is adequate but evidence is lacking for the community-based MET-SV, limiting the confidence that different raters using this version will determine the same results when testing the same person. Further research using a community-based version of the MET should examine inter-rater reliability. The internal consistency of the MET should also be examined further in future studies.

Although some articles identify that the time and effort required to administer the MET is a possible barrier to its use (Tranel et al., 2007; Morrison et al., 2013), the MET provides a measure of executive function in the real-world, which is often the missing information that occupational therapists need to make accurate judgements concerning the functional outcomes of ABI. Additional gaps in the research identified include that further development and publication of version-specific manuals is still needed for standardization of the MET and to aid therapists wanting to use the MET in clinical practice, as well a need for additional cultural specific versions to be developed.

Finally, most measurement properties ratings in the included articles fell in the fair or poor rating primarily due to the COSMIN checklist requirements for sample size. Further research on the MET with larger samples will improve methodological quality of the measurement properties and enable greater confidence in the MET psychometric properties.

Review limitations

This review was limited to published journal articles and therefore unpublished information concerning the MET was not included. A published poster on MET-Home Version preliminary results (Burns 2016) was excluded due to insufficient detail. Virtual versions of the MET were also excluded as not all occupational therapists in the community have access to this technology. Although beyond the scope of this review, comparing the similarities and differences between the virtual MET measurement properties and those versions of the MET in this review could add to the understanding of the strengths and limitations of the various versions. Qualitative studies were not included as these would not assist with examining the psychometric properties of the MET; however, clinical utility of the MET has been examined from a qualitative perspective by Nalder et al. (2015).

Conclusion

This scoping review examined the current evidence regarding the non-virtual reality versions of the MET, and aimed to help occupational therapists understand the strengths, shortcomings and utility of various versions of the MET in relation to the characteristics of the participants and the clinical setting. This review found no evidence that the MET can measure real-world executive function in the youth population in a community setting. Rather, the review found the MET was most effective at distinguishing adult male individuals with moderate to severe ABI from neurologically intact controls in a hospital testing environment. Although the MET is a complex assessment, it correlates strongly with real-world function and can strengthen an occupational therapists’ ability to identify people with executive function deficits who may otherwise go undetected on traditional tests of executive function. The MET can inform treatment planning and patient/family education by identifying the problems that could occur in resuming life roles and the need for support. The review also identifies that the methodological quality of included studies was low, which provides guidance for methodological considerations in future research.

Key findings

The MET can identify the impact of executive function on real-world performance in adults with ABI, mainly in a hospital setting; there is limited evidence of its effectiveness in community settings

No studies have been done specifically on youths in either hospital or community settings

What the study has added

This scoping review of the non-virtual MET versions identifies 10 site-specific versions of the MET, the strengths, weaknesses and gaps in the research, and areas for future research.

Footnotes

Research ethics

Ethics approval was not required for this scoping review as per section 2.4 of the Tri-council Policy Statement on Research Involving Human Subjects (TCPS2).

Consent

Informed consent was not required as human subjects were not involved.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Contributorship

Vanessa L Hanberg, Diane E MacKenzie and Brenda K Merritt contributed to the study design and methodology of the scoping review. Vanessa L Hanberg conducted the literature search and, with Diane E MacKenzie, conducted the study selection. Vanessa L Hanberg completed the data extraction from selected articles. All authors contributed to analysis, synthesis and interpretation of the scoping review. Vanessa L Hanberg wrote the first draft of the manuscript. All authors reviewed and edited the manuscript and approved the final version of the manuscript.

References

Alderman

Burgess

Knight

et al. (2003) Ecological validity of a simplified version of the multiple errands shopping test. Journal of the International Neuropsychological Society 9(1): 31–44.

Arksey

O’Malley

(2005) Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology 8(1): 19–32.

Burns S (2016) Assessing the impact of executive dysfunction in the home environment using the multiple errands test home version (MET-Home): preliminary results. Archives of Physical Medicine and Rehabilitation 97(10): e79.

Chan

RCK

Shum

Toulopoulou

et al. (2008) Assessment of executive functions: Review of instruments and identification of critical issues. Archives of Clinical Neuropsychology 23(2): 201–216.

Cicerone

Levin

Malec

et al. (2006) Cognitive rehabilitation interventions for executive function: Moving from bench to bedside in patients with traumatic brain injury. Journal of Cognitive Neuroscience 18(7): 1212–1222.

Clark

Anderson

Nalder

et al. (2015) Reliability and construct validity of a revised Baycrest Multiple Errands Test. Neuropsychological Rehabilitation 27(5): 667–684.

Colantonio

Kim

Allen

et al. (2014) Traumatic brain injury and early life experiences among men and women in a prison population. Journal of Correctional Health Care 20(4): 271–279.

Colquhoun

Levac

O’Brien

et al. (2014) Scoping reviews: Time for clarity in definition, methods, and reporting. Journal of Clinical Epidemiology 67(12): 1291–1294.

Cuberos-Urbano

Caracuel

Vilar-López

et al. (2013) Ecological validity of the Multiple Errands Test using predictive models of dysexecutive problems in everyday life. Journal of Clinical and Experimental Neuropsychology 35(3): 329–336.

10.

Dams O’Connor

Cantor

Brown

et al. (2014) Screening for traumatic brain injury: Findings and public health implications. The Journal of Head Trauma Rehabilitation 29(6): 479–489.

11.

Dawson

Anderson

Burgess

et al. (2009) Further development of the Multiple Errands Test: Standardized scoring, reliability, and ecological validity for the Baycrest version. Archives of Physical Medicine and Rehabilitation 90(11 Suppl): S41–S51.

12.

Dewan

Mummareddy

Wellons

et al. (2016) Epidemiology of global pediatric traumatic brain injury: Qualitative review. World Neurosurgery 91: 497–509.e1.

13.

Ewing Cobbs

Barnes

Fletcher

et al. (2004) Modeling of longitudinal academic achievement scores after pediatric traumatic brain injury. Developmental Neuropsychology 25(1–2): 107–133.

14.

Haynes

Richard

Kubany

(1995) Content validity in psychological assessment: A functional approach to concepts and methods. Psychological Assessment 7(3): 238–247.

15.

Hunt

Turner

Polatajko

et al. (2013) Executive function, self regulation and attribution in acquired brain injury: A scoping review. Neuropsychological Rehabilitation 23(6): 914–932.

16.

Knight

Alderman

Burgess

(2002) Development of a simplified version of the multiple errands test for use in hospital settings. Neuropsychological Rehabilitation 12(3): 231–256.

17.

Law

MacDermid

(2014) Evidence-Based Rehabilitation – A Guide To Practice, 3rd ed. Thorofare, NJ: SLACK Incorporated.

18.

Levac

Colquhoun

O’Brien

(2010) Scoping studies: Advancing the methodology. Implementation Science 5: 69–69.

19.

Maeir

Krauss

Katz

(2011) Ecological validity of the Multiple Errands Test (MET) on discharge from neurorehabilitation hospital. OTJR: Occupation, Participation and Health 31(1): S38–S46.

20.

Manes

Villamil

Ameriso

et al. (2009) “Real life” executive deficits in patients with focal vascular lesions affecting the cerebellum. Journal of the Neurological Sciences 283(1–2): 95–98.

21.

Moher

Liberati

Tetzlaff

et al. (2009) Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Med 6(7): e1000097.

22.

Morrison

Giles

Ryan

et al. (2013) Multiple Errands Test-Revised (MET-R): A performance-based measure of executive function in people with mild cerebrovascular accident. American Journal of Occupational Therapy 67(4): 460–468.

23.

Nalder

Clark

Anderson

et al. (2015) Clinicians’ perceptions of the clinical utility of the Multiple Errands Test for adults with neurological conditions. Neuropsychological Rehabilitation 27(5): 685–706.

24.

Portney

Watkins

(2009) Foundations of Clinical Research Applications to Practice, New Jersey: Pearson Prentice Hall.

25.

Savage

DePompei

Tyler

et al. (2005) Paediatric traumatic brain injury: A review of pertinent issues. Pediatric Rehabilitation 8(2): 92–103.

26.

Shallice

Burgess

(1991) Deficits in strategy application following frontal lobe damage in man. Brain 114(2): 727–741.

27.

Sowell

Thompson

Toga

(2004) Mapping changes in the human cortex throughout the span of life. The Neuroscientist 10(4): 372–392.

28.

Stuss

Alexander

(2007) Is there a dysexecutive syndrome? Philosophical Transactions of the Royal Society B Biological Sciences 362(1481): 901–915.

29.

Taylor

Barker

Heavey

et al. (2013) The typical developmental trajectory of social and executive functions in late adolescence and early adulthood. Developmental Psychology 49(7): 1253–1265.

30.

Taylor

Barker

Heavey

et al. (2015) The longitudinal development of social and executive functions in late adolescence and early adulthood. Frontiers in Behavioral Neuroscience 9: 252.

31.

Terwee

Mokkink

Knol

et al. (2011) Rating the methodological quality in systematic reviews of studies on measurement properties: A scoring system for the COSMIN checklist. Quality of Life Research 21(4): 651–657.

32.

Terwee

Prinsen

Chiarotto

et al. (2018) COSMIN standards and criteria for evaluating the content validity of health-related patient reported outcome measures: A Delphi study. Quality of Life Research 27(5): 1159–1170.

33.

Toglia

Berg

(2013) Performance-based measure of executive function: Comparison of community and at-risk youth. American Journal of Occupational Therapy 67(5): 515–523.

34.

Toronto ABI Network (1999) Definition of ABI. Available at: www.abinetwork.ca/for-professionals (accessed December 28 2017).

35.

Tranel

Hathaway Nepple

Anderson

(2007) Impaired behavior on real-world tasks following damage to the ventromedial prefrontal cortex. Journal of Clinical and Experimental Neuropsychology 29(3): 319–332.