Face validity of the youth Multiple Errands Test (yMET) in the community: A focus group and pilot study

Abstract

Introduction

During late adolescence and early adulthood, maturation of cognitive functions including executive functions are occurring. The multiple errands test is an assessment of real-world executive functions and, to date, non-virtual reality multiple errands test research has focused primarily on adults with acquired brain injury in hospital settings. There is poor evidence across multiple errands test studies for content and face validity and limited studies in the community. This study aimed to explore multiple errands test face validity for typically developing youth (age 16–24 years) and describe their community setting performance on a youth multiple errands test.

Methods

A youth focus group (N = 5) was conducted to explore perceptions of the multiple errands test. From their input, the youth multiple errands test was developed and pilot tested (N = 9) in a shopping mall.

Results

Two themes emerged from focus group analysis and limited changes, relevant to youth, were made to develop the youth multiple errands test. The focus group and pilot study found the youth multiple errands test was acceptable and cognitively challenging for youth, with older youth performing better than younger youth. Overall youth multiple errands test performance suggests similarities to healthy adults in previous studies.

Conclusion

Findings must be interpreted with caution since the sample was small, but preliminary results indicate that future studies with the youth multiple errands test are feasible and warranted.

Keywords

Executive function adolescent young adult multiple errands test face validity occupational therapy

Introduction

Late adolescence and young adulthood are a critical period of brain development for cognitive functions, including executive function (Taylor et al., 2013). Executive function (EF) is an overarching set of cognitive functions that coordinate and integrate subordinate cognitive components to enable goal-directed behaviour (Cicerone et al., 2006; Stuss, 2009). These cognitive components include task-setting and task-monitoring, goal-selection, planning, organizing, problem-solving, decision-making, impulse control and cognitive flexibility (Hunt et al., 2013). Impairment in EF from acquired brain injury (ABI) profoundly impacts everyday life, particularly in the underlying cognitive abilities necessary for new and complex task performance that are required for adult life roles like employment and independent living. The challenge in the adolescent and young adult population is that executive dysfunction from ABI can be difficult to distinguish from normal development and has possible implications for the remainder of adult life (Beauchamp et al., 2011).

The multiple errands test (MET) is a naturalistic assessment of real-world EF used by occupational therapists and neuropsychologists to identify the impact of impaired EF. The MET is a complex shopping activity conducted in a real-world context, with the therapist observing and rating performance. The test-taker must follow a set of rules while completing a series of tasks, including buying items and finding out specific information. A scoping review of the MET previously identified that the MET is a sensitive, ecologically valid and useful tool for occupational therapists to evaluate the impact of EF on real-world performance (Hanberg et al., 2018). The scoping review found that across the non-virtual reality versions of the MET, the main evidence of its use is in adult males with ABI in a hospital setting, with limited evidence in community settings (Hanberg et al., 2018). The review also identified no evidence for the MET in the late adolescent and early adult population (age 16–24 years), termed youth hereafter, and poor evidence across studies for content and face validity.

Content and face validity are fundamental psychometric properties for assessment instruments. Content validity is the extent to which the construct (that is, real-world EF) measured is adequately reflected by the items/structure of the measure (Law and MacDermid, 2014). It can vary across populations, so it should be established for the population with which the assessment will be used (Haynes et al., 1995). The developers of the consensus-based standards for the selection of health measurement instruments (COSMIN) consider content validity to be the most important measurement property, because “it should be clear that the items of the [measurement instrument] are relevant, comprehensive, and comprehensible with respect to the construct of interest and target population” (Terwee et al., 2018: 16). To date, the content validity of the MET for the adult population has been established by expert contribution (Hanberg et al., 2018).

Face validity is an aspect of content validity and considers if the measure is acceptable and relevant to those who are tested by it (Portney and Watkins, 2009). In a review article by Auger et al. (2006), pragmatic criteria of measurement tools including the concept of acceptability are defined. Based on their discussion, acceptability is defined as a subjective perception of the level of difficulty of the assessment. Relevancy of an assessment, for our purposes, refers to the appropriateness for youth of the items/structure in current time, circumstances and contemporary culture. Thus far, no evidence for the face validity of the MET has been reported for any population.

To address several gaps in the MET literature for youth, the aim of this research was first to examine the face validity of the original MET from the perspective of typically developing youth in a suburban community setting. Then, based on their perceptions and input, integrate changes to the MET and pilot test the revised MET in a community setting with youth (using the age range of 16–24 years). Another aim was to compare the performance differences of younger to older participants. Previous work on EF in youth suggests several areas of improvement in EF ability from age 17–19 years on traditional neuropsychological tests of EF (Taylor et al., 2013). Therefore, the assumption for this study was that younger participants will make more errors and complete fewer tasks on the MET than older participants. The research questions were as follows:

What is the face validity of the original MET for the youth age group (age 16–24 years) in a community setting?

How do youth perform on the MET administered in a community setting?

What is the acceptability of the MET for youth in a community setting?

Methods

Ethics approval for the study was obtained in 2017, as was written informed consent from all participants. As the MET is a naturalistic assessment taking place in real-world environments, several site-specific versions of the MET have been developed for use in hospital or community settings (for example a shopping mall). The version used in this study is based on the MET-SV (simplified version) (Alderman et al., 2003) as it takes place in a community setting. Hereafter, the MET-SV version is referred to as the MET.

Focus group design

To explore the face validity of the MET with the youth population, a focus group was selected as a suitable design (Ivanoff and Hultberg, 2006). Given the exploratory kind of knowledge that a focus group would generate, it was determined a fitting method to gather the youth population perspective. Specifically, MET language and content was explored as a means of establishing the face validity of the MET for use with a youth population.

Focus group recruitment

The guidelines provided by Hollis et al. (2002) and Ivanoff and Hultberg (2006) were consulted for planning and delivery of the focus group. There is a range of recommendations for focus group size, ranging from four to 12 (Depoy and Gitlin, 2011; Hollis et al., 2002; Ivanoff and Hultberg, 2006). The focus group sample was based on non-random methods, convenience and snowball sampling. Youth were recruited from known agencies (for example recreation centres with youth programmes, high school, college, libraries) and through parents of youth known to the lead researcher. Youth volunteered to participate in the study. Ivanoff and Hultberg (2006) recommend homogeneity as a priority when selecting participants, but that some differences amongst group members are important to capture diversity and increase the transferability of findings (Hollis et al., 2002). Therefore, the lead researcher selected a mix of ages within the targeted age range (16–24 years) and a mix of genders, and recruited those with some experience with shopping.

Focus group procedure

Ahead of the focus group, a focus group guide was developed based on published suggestions for content, number and types of questions (Hollis et al., 2002; Ivanoff and Hultberg, 2006). Four key questions and probes were developed on the understanding they were a guide rather than a specific protocol. The open-ended questions asked about general impressions, language, items and information included in the MET instructions. The focus group participants were introduced to the MET, the role of the MET and executive functioning in an information letter provided in advance. During the focus group each participant read the MET before the questions were asked. The lead researcher was the group facilitator of the focus group. Focus group discussion was audio-recorded and recordings transcribed verbatim. The transcript was then analysed using the step-by-step thematic analysis guide as defined by Braun and Clarke (2006). There are six phases of thematic analysis: familiarizing yourself with your data; generating initial codes; searching for themes; reviewing themes; defining and naming themes and producing the report. A second researcher independently reviewed the transcript and initial codes to confirm the findings.

Pilot study methods

Following the focus group and data analysis, the MET was revised based on youth input and named the yMET (youth MET). The yMET was then used in a pilot study with community-dwelling, typically developing youth aged 16–24 years in a suburban mall setting. The pilot study participants were recruited from the same sources as the focus group by non-random sampling methods, convenience sampling and snowball sampling. There were no overlapping members between the focus group and pilot study. This was purposeful, so pilot study participants did not have previous knowledge of the MET items, which could make completing the tasks easier for them. Guidelines for the number of pilot study participants was determined by reviewing Hertzog (2008), who suggests 10 or fewer if the purpose of the study is acceptability of formatting or clarity of instructions. Pilot study participants completed the yMET at a shopping mall in western Canada, starting at a designated location, with the lead researcher observing their performance. Participants were read the instructions, then provided with a copy of instructions, a map of the mall, required money and a pen. The lead researcher was not permitted to video record in the mall environment to document each participant’s performance on the yMET; therefore, the researcher kept detailed field notes for each participant, writing down specific details of events, observations and occurrences. The yMET was scored (with permission) according to the guidelines developed by Dawson, Nalder and Clark (2016). These scoring guidelines were chosen to improve the standardization of scoring across MET studies and because errors in performance and total error scores were clearly defined. A total errors score was calculated by adding tasks omitted, partial task failures and rules broken. Partial task failures were defined ahead of time from a list of identifiable errors that prevent full completion of the task. Participants were asked after completing the yMET to rate the level of perceived difficulty when completing the yMET on a Likert scale (1=extremely easy to 5 = extremely difficulty) and to respond to an open-ended question about their experience completing the yMET.

Results

The results are presented for the focus group, followed by the pilot study. The focus group was conducted at a community library, consisted of five youth (three females and two males) aged 16–22 years old and lasted an hour. There were challenges recruiting youth related to availability and interest in participating in focus group activities. The youth that volunteered for the focus group were from high school and parent sources. The planned format of the focus group resulted in an in-depth discussion with all group members contributing, some more than others. All the focus group participants were engaged in either school or post-secondary education, making it difficult to stay in communication afterwards to share the analysis and interpretation of the findings with them, due to the start of a new school year.

Focus group

For phase 1 of the thematic analysis, the focus group facilitator was also the transcriber of the audio-recording, and had ample opportunity to become familiar with the data. For phase 2, initial codes were developed as a way of organizing the data into meaningful groups by reviewing the transcript and coding manually. Examples of initial codes from the transcript included difficulty of the task, understanding rules/instructions, questioning item relevancy, and cell phone use. The codes were further examined for patterns, and seven candidate themes identified (phase 3). These seven candidate themes were then further sorted into two main ideas or themes (phase 4) and the themes named and defined (phase 5) with sub-themes. The first theme contained the coded material supporting preserving the assessment purpose and the second theme contained coded material supporting various changes to the assessment to make it relevant for the audience (youth) and the time and place of the assessment. One area of coded material, cell phone use, identified a tension between themes, as cell phones are relevant to youth but would also impact the way youth would carry out the assessment. Ultimately, based upon participant feedback, cell phone use became a sub-theme of the theme ‘maintain assessment purpose’.

Two main themes were generated from the focus group data: (1) maintain assessment purpose, which incorporated the two sub-themes of (a) difficulty of task and (b) limiting cell phone use; and (2) rethinking assessment relevancy, which incorporated the three sub-themes of (a) clarity of rules and instructions, (b) appropriateness to person (age 16–24 years), location, circumstances and contemporary interest, and (c) item substitutions.

Theme 1: maintaining assessment purpose

Generally, youth in the focus group perceived that the MET assessment is difficult: ‘there are a lot of rules to follow’ (FG1); ‘it’s a little bit difficult to… understand everything’ (FG2); however, one member stated, ‘it’s not that difficult’ (FG5). The focus group discussion indicated several factors that contribute to making the MET assessment challenging. These include the rules and the cognitive demand associated with the rules: ‘There is a lot of multi-tasking, and I’m not very good at multi-tasking, it’s a little stressful’ (FG3); unusual/unfamiliar items and/or where to buy the item(s); understanding the assessment requirements due to language used in the MET and relevancy of the tasks to current time and place.

While the focus group participants recognized that changes would need to be made to the MET assessment to improve comprehension of instructions and make it relevant to time/place, the discussion highlighted the need to balance maintaining the integrity of the assessment purpose with changes to language and content. FG5 stated, ‘I can understand it all just fine… some of it is used in Britain and doesn’t transfer over well… such as words like packet of plasters, I frankly don’t have a clue what that is’, and FG2 stated, ‘I suppose the question then is about relevancy or difficulty’. It was important to the group that, when considering changing the assessment, the integrity of the assessment purpose was not compromised by reducing the challenge of the assessment. When discussing changing an item, one participant said, ‘I feel like you need to choose something equally challenging to find’ (FG1). Generally, the group felt that item substitutions should stay within the same category and be equally challenging to find.

Maintaining the level of challenge was also discussed as an important factor when considering cell phone use. FG2 stated:

I would have some kind of contingency in as far as cell phone, because I think they could just get all the information… they won’t move, and it kind of takes away from the spirit of the multiple errands test, when you are basically condensing multiple errands into a single… you don’t have to use a crutch, but you can use it.

While this was verbally expressed by one member, the group unanimously agreed (this was determined during the focus group by a show of hands) that while using cell phones during the assessment was relevant to their age group, cell phone use to look up information required for the assessment should be restricted as unlimited cell phone use could make finding the information too easy and limit the opportunity for the therapist to observe performance directly.

Theme 2: rethinking assessment relevancy

When considering making changes to the assessment, it was more important that those completing the assessment understand the instructions and task requirements than taking into consideration personal shopping preferences.

Yeah… one or two items could be changed, but what’s relevant to me might not be relevant to someone else, like someone might not know where to get a candle, where I’d say bath and body works, some else might not say so (FG3).

Uhmm, yeah, I’m more on the side where, I just mean like if I was given this task, I don’t think I’d care what the items were or if I’d use them, as long as I knew what they all were, I can identify them… (FG1)

Some discussion focused on possible gender differences, for example girls know about the store Forever 21, whereas the boys had never been in this store or had only limited knowledge of the store. A clear understanding of the task requirement was discussed as being related to culturally specific language in the MET, issues of contemporary interest to youth and location specific constraints. FG2 said, ‘I think I agree that the language is a bit dated, for someone from our generation, our culture… it’s a little bit difficult to… understand everything’, and FG5 said, ‘no one really goes to libraries anymore’. Substitutions or changes to reflect Canadian ways of saying things, current habits of youth and practical considerations about the specific mall the assessment would occur in were suggested.

Overall, there was more group consensus on substitutions for the information to be gathered by youth on the MET (the second task on the MET), than which items to purchase (the first task on the MET). Substitutions to information to be gathered were more directly related to issues of current youth habits and location specific constraints. While substitutions to items for purchase were suggested, this topic was more debated and generally it was more important that items to purchase were put in Canadian terms but be otherwise unchanged, to maintain the challenge of the assessment.

Changes to the MET

The revised instructions, based on focus group input, are found in the Appendix. See Table 1 for the changes made to the MET. The table includes reasons for changes to the MET based on the focus group discussion as well as practical considerations. MET changes were agreed upon during the focus group.

Table 1.

Summary of original MET changes in the yMET.

Original MET	Substitution on yMET	Reason from focus group	Practical issues
Small brown loaf	Small loaf of brown bread	Canadian word use, understanding
Packet of plasters	Package of bandaids	Same as above
Bar of chocolate	Chocolate bar	Same as above
Single light bulb	Package of matches OR candle OR bar of soap No consensus, no change	3/5 focus group members wanted to change this item, but no consensus could be reached on what the final change should be
Birthday card	No change
Key ring	No change
What is the headline from either today’s Daily Mail, Daily Mirror or The Sun newspapers?	Details of the sale at Old Navy (what is on sale and what is the sale amount) OR Special of the day at restaurant	Contemporary interest
What is the closing time of the library on Saturday?	What is the closing time of specific restaurant/the mall on Saturday?	Contemporary interest	Location specific constraint; there is no library at this mall
What is the price of 1 pound or kilogram of tomatoes?	No change
How many shops sell televisions?	How many cell phone providers are in the mall?	Contemporary interest
You must meet me under the clock 20 minutes after you have started this task and tell me the time	Meet me at the mall directory 20 minutes after I have said ‘begin the exercise’ and tell me the time. You can then continue the exercise.	The clarity of this item was discussed, and revisions suggested including explicitly stating they can continue the exercise after the meeting at the directory	This mall does not have a clock, so another designated location was determined (the mall directory)
Whilst carrying out this exercise you must obey the following rules:	While carrying out this exercise you must follow these rules:	Culturally specific language: use Canadian words/expressions instead
You must carry out all these tasks but may do so in any order	No change
You should spend no more than £5	You should spend no more than $17	Culturally specific language	Lead researcher determined all items can be purchased with $14 but this left very little room in the budget. The amount was increased to $17 (they are given $20 so they still have to budget and stay within the budget).
You should stay within the limits of the upper floor of the shopping centre	You should stay within the limits of the mall and do not exit the mall to the outdoors	Location constraints; discussion in focus group that mall is big but if you limit them to upper floor there are items/information they likely can’t locate with this limitation	You have to exit the main doors to the mall and either walk across an outdoor parking lot or underground parking area to access Walmart. If this rule stays, they can’t go to Walmart without violating a rule.
No shop should be entered other than to buy something	No change Additional explanation provided in the instructions read aloud	Needs to be clarified based on focus group feedback
You should not go back into a shop you have already been in	No change
You should not buy any item from the stalls	You should not buy any items from the kiosks OR You should not use your cell phone more than two times to search for information	Contemporary interest: it was more important to limit cell phone use than include the rule about kiosks. Rather than add in another rule about cell phone use, the cell phone limit replaced this rule.
You should buy no more than 2 items from Tesco	You should buy no more than 2 items from the Dollar Store	Location specific issues
Take as little time as possible to complete this exercise, without rushing excessively	No change
Do not speak to the person observing you unless this is part of the exercise	No change

MET: multiple errands test; yMET: youth multiple errands test.

Pilot study

Nine youth aged 16–24 years completed the yMET on a one-to-one basis with the lead researcher at a shopping mall. The pilot study volunteers came from youth agencies, high schools and parent sources. Seven of the pilot study participants were either engaged in school or post-secondary education and two were employed. Refer to Table 2, first column, for a summary of the results and descriptive statistics for the pilot study. One participant revealed, after yMET completion, a past ABI due to a sport-related concussion (that is, he met the exclusion criteria for prior neurological injury). The participant’s scores were compared to the others and given that his scores on the yMET were within the range of the other participants, the data was included in this study.

Table 2.

Descriptive statistics of youth performance on the yMET, with reference to other versions.

	yMET	yMET younger	yMET older	BMET	BMET-R
	Mean (range) N = 9	Mean N = 5	Mean N = 4	Mean (range) Stroke controls (SC)^a N = 13 Traumatic brain injury controls (TC)^a N = 12	Mean (range) Version A (VA)^c Controls^b N = 16 Version B (VB)^c Controls^b N = 16
Age (years)	18.67 (16–24)	17 (16–18)	20.75 (19–24)	SC = 56.7 ± 15.8 TC = 46.1 ± 9.6	61.4 ± 12.0
Males/females	4/5	3/2	1/3	SC = 7/6 TC = 10/2	10/6
Tasks completed (out of 12)	10.44 (8–12)	9.8	11.25	SC = 8.3 ± 2.1 TC = 6.8 ± 1.3	VA = 9.3 ± 1.7 VB = 8.1 ± 2.0
Tasks not completed/omissions	1.56 (0–4)	2.2	0.75	SC = 0.7 ± 0.8 TC = 0.8 ± 1.5	VA = 0.8 ± 1.3 VB = 1.3 ± 1.7
Partial task failure	3.44 (1–6)	3.6	3.25
Rules broken (out of 8)	2.11 (1–3)	2.4	1.5	SC = 1.6 ± 1.2 TC = 2.4 ± 1.3	VA = 2.1 ± 1.5 VB = 2.8 ± 2.2
Total error score^d	7.00 (2–9)	8.2	5.5	SC = 9.1 ± 5 TC = 12.7 ± 4.9	VA = 4.8 ± 3.4 VB = 4.5 ± 2.0
Total time to complete (min)	28.56 (20–38)	29.6	27.25
Perceived difficulty rating	2.78 (1–4)	3.2	2.25

yMET: youth multiple errands test; BMET: Baycrest multiple errands test (Dawson et al., 2009); BMET-R: Baycrest multiple errands test – revised (Clark et al., 2015)

BMET controls were defined as friends/family of ABI participants or from a volunteer pool; read, understand, speak English; mean age > 45 years; walk independently for half an hour or more; cognitive and depression screen in normal range.

BMET-R controls were defined as from a volunteer pool; mean age > 60 years; no history of neurological injury/disorder; community dwelling; fluent in written/spoken English; can ambulate independently for at least 30 minutes; no history of bereavement within six months; cognitive and depression screen in normal range.

Counterbalanced design between design exposure.

yMET defined total errors = tasks omitted, partial task failures and rules broken

Dawson et al. (2009) defined total errors = task failures, task omissions, rules broken and other inefficiencies.

Clark et al. (2015) defined total errors = task omissions, tasks partially completed and inefficient behaviour.

To examine general trends in the data and address the hypothesis, two categories with close to equal numbers in each category were determined, aged 16–18 years, termed the younger group, and aged 19–24 years, termed older group. See Table 2 for a summary of these results. General trends in the data supported that the younger participants had a higher mean total error score, compared to the older participants’ mean total error score. Younger participants also completed fewer tasks compared to older participants.

The rules most often broken by youth on the yMET were the rule ‘do not speak to the person observing you unless this is part of the exercise’ and ‘you must carry out all these tasks but may do so in any order’, with seven of the nine participants breaking these rules. The scoring of the rule to carry out all the tasks is scored broken if any of the 12 tasks are omitted. Four of the nine youth broke the budget rule ‘you should spend no more than $17’. The task most often omitted was task #12, ‘tell the person observing you when you have finished the exercise’, with six of the nine youth not completing this task.

The mode for perceived difficulty rating for the nine participants, provided after completion of the yMET, was 3 (‘neither easy or difficult’), with four participants choosing this response; one participant rated ‘extremely easy’, two rated ‘easy’ and two rated ‘difficult’.

Qualitative comments

The youth participants provided comments of their subjective experience following completion of the yMET. Several comments directly address the issue of acceptability, which was defined in this study as perceived level of difficulty. The budget and time constraints were reasons provided for perceived level of difficulty.

…hard, things are more expensive than I thought; hard to find things that fit the price; what is a key ring? (P4)

When I first got here and saw the list I thought it would be an easy task, but when I did it I found difficulty finding things within my budget (P5).

The part I found most difficult was remembering to look at the time and not go over budget (P8).

The cognitive challenge associated with decision-making and multi-tasking were other reasons provided for level of difficulty. There were also comments that showed evidence of metacognition or understanding of one’s own thought processes.

I thought it was somewhat difficult but do-able. I had to make decisions on where I would buy certain things based off if I thought I could find them at other stores (P6).

It was slightly difficult because of all the tasks that I had to complete, and the items took a bit of thought to find, but overall, it was a moderate challenge (P7).

Showed me some things about myself that I didn’t fully notice before; it was a fun challenge (P1).

Interesting exercise!… brings out themes in how I approach tasks; very curious to see results and comparatives (P2).

There were comments to indicate that for some youth the yMET was easy.

It was simple, with clear instructions and it was quick (P3).

Straight forward task and what was expected (P9).

Field notes

The majority of youth (7/9) consulted the mall directory and one youth looked at the map provided. Other environmental cues that were used to complete the tasks were the mall hours posted on the doors to the mall, asking store clerks and asking at the service desk in the mall. All the youth participants relied on their personal cell phone to keep track of the time with no instances where participants used their cell phones other than to momentarily check the time.

Discussion

The focus group identified that widespread societal changes (for example growth of technology) and culturally specific language impact the content and face validity of assessments developed a decade or more ago in a different cultural setting. The focus group insights and MET revisions to create the yMET have updated the assessment to reflect issues of contemporary interest and importance to youth in North America, enhancing the face validity of the yMET for this population. Some changes were also made due to language, culture and site-specific reasons, not unique to youth and not directly the focus of this study. Not all potential changes were included to make the yMET relevant to youth, in order to preserve the assessment purpose and challenge.

An issue relevant to youth and part of contemporary youth habits that the focus group raised is routine cell phone use and an over-reliance on cell phones. Despite their relevancy, cell phones were identified by the focus group as something that could interfere with MET performance. The cell phone could be used to look up required information without the participant having to move to a location or search for information in the physical environment. As such, cell phone use was restricted in the yMET rules to ‘no more than two times to search for information’ and, in the instructions provided in the Appendix, youth were also told they could use their cell phone to look at the time. From field note analysis, no youth broke the cell phone use rule; more specifically, no youth used their cell phone to search for information – instead, they mostly used the directory or other sources of information in the physical environment. When youth used their cell phone it was to keep track of the time, an important task requirement. Overall, the yMET is cognitively demanding enough for youth that they did not use their cell phones for non-yMET related purposes (as a distraction) and remained engaged in the yMET tasks throughout.

Another study aim was to examine how youth perform on the yMET, given the cognitive development and maturation of EF in this age group. Previous published versions of the MET provide an understanding of how adults (both adults with brain injury and non-brain-injured controls) perform on the MET, but it is not possible to make a direct comparison across MET versions due to the version differences (different testing environments, shopping mall and hospital, and item and scoring differences). While direct comparisons cannot be made, how youth did on the yMET in this pilot study relative to adult non-brain-injured controls in previously published studies of the MET was considered. We based the scoring methods of this study on those developed by the researchers of the Baycrest Multiple Errands Test (BMET), and as such, consideration is given to two of the BMET studies (Clark et al., 2015; Dawson et al., 2009) and presented in Table 2 in the grey columns to distinguish from yMET results. There are differences in the definitions of total errors between studies. There is greater conformity in how other areas of scoring were defined, including tasks completed, tasks not completed (task omissions) and rules broken. Generally, youth in the pilot study appear to perform on the yMET in a similar way to healthy adults in previous studies, with the greatest variability in total error scores attributable to differences in scoring definitions across studies.

Examining the trends in the errors made by youth on the yMET provides some indication of the cognitive challenge for this age group. The rule most often broken by youth on the yMET was the rule to not speak to the examiner unless part of the exercise. This rule requires youth to use impulse control as they carry out the yMET, an area of cognitive control still developing in this age group (Blakemore and Robbins, 2012; Peach and Gaultney, 2013). Typically, youth completing the yMET asked the examiner questions (although given time before starting to ask questions) or reported difficulties to the examiner (for example, ‘I’ve run out of money’). The task most often omitted by youth was the task to explicitly tell the examiner when they had finished the exercise. While this information is clearly printed in bold letters on the instruction sheet and read aloud to all participants before they begin the yMET, it is the last task presented. It could be that youth are already thinking about how they will approach the various yMET tasks, therefore they have ‘tuned out’ and are not paying attention at this stage of the instructions. This information is then not stored in working memory for them to recall later at the end of the yMET when they are required to do this task. Youth performance in this pilot study of the yMET support that the yMET is sensitive to identifying areas impacting EF such as impulse control and attention and can assist occupational therapists working with youth to develop strategies to improve these skills.

Multiple strategies were used to understand the face validity and specifically the acceptability of the MET for the youth population. Several focus group members indicated they thought the MET would be difficult to complete and, based on their input, aspects of the MET were modified or substituted to address location specific issues, Canadian/contemporary language and terminology as well as other issues relevant to youth. Then, the pilot study group rated their perceptions of the level of difficulty after completing the yMET on the Likert scale and results showed that the level of acceptability for the youth age group mostly fell in the ‘easy’ to ‘neither easy or difficult’ rating. The open-ended question the participants completed after the yMET concerning their experiences while completing the yMET included words like fun, interesting, simple, clear, quick, straightforward and do-able. The written feedback that indicated increased level of challenge/difficulty included terms like moderate challenge, somewhat difficult and time/budget constraints. Methodological triangulation of findings from the focus group discussion, Likert scale rating and the open-ended question were used to determine the acceptability or perceived level of difficulty of the yMET. Triangulation of findings suggests the yMET was acceptable to youth but also provides a fitting challenge.

For occupational therapists working with youth, this study suggests that the yMET can be used with typically developing youth in a community setting to identify areas impacting real-world EF. The pilot study provides preliminary evidence that younger participants in the youth age group make more errors and complete fewer tasks, but the small sample size limits the strength of this evidence.

Study limitations

The youth recruited for this study were all engaged in either school or work; therefore, the results are not based on a diverse study sample that includes at-risk youth or youth with varying abilities.

For the focus group part of this study, only one focus group, with five people in the group, was conducted. When several focus groups are conducted it allows for a broader range of perspectives and a greater likelihood of saturation, or the point at which information gathered does not provide new understandings. With only one focus group conducted in this study, with a minimum number of people, it was not possible to reach a point of saturation and a narrow range of viewpoints are considered. Another method to enhance the rigour of data collection and analysis for the focus group would be member checking, but it was not undertaken for this study due to timing as, following the focus group, the youth were starting a new school year and unavailable to provide further feedback.

While the small sample size of nine participants limits the strength of the evidence of the pilot study and the conclusions that can be made, this pilot study serves a purpose in determining the feasibility of future larger studies. A further study limitation was that in this study it was challenging to recruit the full range of ages (16–24 years), with fewer participants in the 20–24 years ages; therefore, the data is likely to be more representative of how the younger end of this age range perform on the yMET.

As a next step, more focus groups with diverse youth samples could contribute refinements to the yMET and enhance face validity. Future yMET studies with larger sample sizes, consistent scoring methods, comparison youth and performance groups, and/or a longitudinal study following youth into adulthood would provide an understanding of real-world executive functioning development occurring in this age group.

Conclusion

The results of this mixed method study suggest there is evidence for face validity of the yMET with a typically developing youth population in a community setting and that the yMET is acceptable to this population. Youth also appear to perform in a similar way to healthy adults on the yMET, although this was not demonstrated by statistical methods, but rather from general trends in relation to previously published studies with adults. Preliminary results with the youth population support that the yMET tasks are sufficiently demanding that, with clear cell phone use rules, youth do not rely on cell phones for yMET task completion; the yMET is sensitive to identifying underlying cognitive skills that impact real-world EF in youth; and that older participants in the age range perform better than younger participants. These preliminary results indicate that a larger future study with the yMET is feasible and warranted.

Key findings

yMET was developed with input from a youth focus group with minimal changes to language/requirements.

Youth found the yMET format and requirements to be acceptable and challenging.

Results of the yMET pilot study indicate that a larger study with the yMET should be conducted.

What the study has added

A focus group with typically developing, community-dwelling youth provided input to develop the yMET and, with the pilot study, suggests the yMET is challenging, acceptable and relevant for this population.

Footnotes

Acknowledgements

We thank Deirdre Dawson, Emily Nalder and Amanda Clark for providing the guidelines for scoring of the multiple errands test.

Research ethics

Ethics approval was obtained for this study from the Health Sciences Research Ethics Board at Dalhousie University on 1st September 2017 (file number 2017-4181).

Consent

All participants provided written informed consent to be audio-recorded for the focus group and to participate in the pilot study.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship and publication of this article.

Funding

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

Contributorship

Vanessa Hanberg originated the study idea and contributed to the design and methodology of the focus group and pilot study together with Diane MacKenzie and Joan Versnel. Vanessa Hanberg conducted the focus group and pilot study. All authors contributed to the analysis, synthesis and discussion of results. Vanessa Hanberg wrote the first draft of the manuscript and all authors reviewed, edited and approved the final version of the manuscript.

References

Alderman

Burgess

Knight

et al. (2003) Ecological validity of a simplified version of the multiple errands shopping test. Journal of the International Neuropsychological Society: JINS 9(1): 31–44.

Auger

Demers

Swaine

(2006) Making sense of pragmatic criteria for the selection of geriatric rehabilitation measurement tools. Archives of Gerontology and Geriatrics 43(1): 65–83.

Beauchamp

Catroppa

Godfrey

et al. (2011) Selective changes in executive functioning ten years after severe childhood traumatic brain injury. Developmental Neuropsychology 36(5): 578–595.

Blakemore

Robbins

(2012) Decision-making in the adolescent brain. Nature Neuroscience 15(9): 1184–1191.

Braun

Clarke

(2006) Using thematic analysis in psychology. Qualitative Research in Psychology 3(2): 77–101.

Cicerone

Levin

Malec

et al. (2006) Cognitive rehabilitation interventions for executive function: Moving from bench to bedside in patients with traumatic brain injury. Journal of Cognitive Neuroscience 18(7): 1212–1222.

Clark

Anderson

Nalder

et al. (2015) Reliability and construct validity of a revised Baycrest Multiple Errands Test. Neuropsychological Rehabilitation 27(5): 667–684.

Dawson D, Nalder E and Clark A (2016) The Multiple Errands Test: Developing, administering and scoring site specific versions. CAOT workshop materials. Available from the author.

Dawson

Anderson

Burgess

et al. (2009) Further development of the Multiple Errands Test: Standardized scoring, reliability, and ecological validity for the Baycrest version. Archives of Physical Medicine and Rehabilitation 90(11 Suppl): S41–S51.

10.

Depoy

Gitlin

(2011) Introduction to Research Understanding and Applying Multiple Strategies. Missouri: Elsevier.

11.

Hanberg

MacKenzie

Merritt

(2018) Scoping review of the Multiple Errands Test (MET): Is it relevant to youths with ABI? British Journal of Occupational Therapy 81(12): 673–686.

12.

Haynes

Richard

Kubany

(1995) Content validity in psychological assessment: A functional approach to concepts and methods. Psychological Assessment 7(3): 238–247.

13.

Hertzog

(2008) Considerations in determining sample size for pilot studies. Research in Nursing & Health 31(2): 180–191.

14.

Hollis

Openshaw

Goble

(2002) Conducting focus groups: Purpose and practicalities. British Journal of Occupational Therapy 65(1): 2–8.

15.

Hunt

Turner

Polatajko

et al. (2013) Executive function, self regulation and attribution in acquired brain injury: A scoping review. Neuropsychological Rehabilitation 23(6): 914–932.

16.

Ivanoff

Hultberg

(2006) Understanding the multiple realities of everyday life: Basic assumptions in focus-group methodology. Scandinavian Journal of Occupational Therapy 13(2): 125–132.

17.

Law

MacDermid

(2014) Evidence-Based Rehabilitation – A Guide to Practice, 3rd ed. Thorofare, NJ: SLACK Incorporated.

18.

Peach

Gaultney

(2013) Sleep, impulse control, and sensation-seeking predict delinquent behavior in adolescents, emerging adults, and adults. Journal of Adolescent Health 53(2): 293–299.

19.

Portney

Watkins

(2009) Foundations of Clinical Research Applications to Practice. New Jersey: Pearson Prentice Hall.

20.

Stuss

(2009) Rehabilitation of frontal lobe dysfunction: A working framework . In: Oddy

Worthington

(eds) The Rehabilitation of Executive Disorders: A Guide to Theory and Practice. Oxford: Oxford University Press, pp. 3–17.

21.

Taylor

Barker

Heavey

et al. (2013) The typical developmental trajectory of social and executive functions in late adolescence and early adulthood. Developmental Psychology 49(7): 1253–1265.

22.

Terwee

Prinsen

Chiarotto

et al. (2018) COSMIN standards and criteria for evaluating the content validity of health-related Patient Reported Outcome Measures: A Delphi study. Quality of Life Research 27(5): 1159–1170.