Measuring executive function in people with severe aphasia: Comparing neuropsychological tests and informant ratings

2.1 Participants

2.2 Material

2.3 Procedure

Both tests (CLQT and ST) were administered and scored by the first author and video recorded. If the significant others were present at the first meeting with the PWSA, they were given the BRIEF-A and CETI along with verbal instructions, which gave them the opportunity to ask for any clarifications. Significant others that could not be present were given instructions by telephone, and the questionnaires were sent by mail. They were encouraged to contact the first author if they had any questions while completing the questionnaire.

2.4 Data analysis

Non-parametric methods for statistical analyses were used because of non-normally distributed data.

Transformed CLQT scores were used for relating performance to criterion cut scores. Difference to criterion cut score was analysed using Mann-Whitney U test.

For BRIEF-A, raw scores and T-scores are presented.

For the comparison between CLQT and BRIEF-A, raw scores were used. These are the scores normally presented for CLQT. For BRIEF-A, on the other hand, the scores normally presented are the T scores. The BRIEF-A raw scores, by themselves, are hard to interpret, since all subscales include different numbers of items and thus have differing minimum and maximum scores. However, they were used in the correlation analysis because it was considered inappropriate to mix age adjusted T scores and non-age-adjusted CLQT raw scores in the analysis.

For investigating relations between functional communication and informant ratings of executive function, Scenario Test total scores and CETI mean scores were correlated against BRIEF-A raw scores.

All statistical analyses were conducted with IBM SPSS Statistics, version 25. P-values ≤.05 are presented in bold text, as an indication of the significance level commonly used in previous studies of relations between neuropsychological tests and ratings. Cohen (1988) guidelines for interpretation of correlation coefficients were used (.10–.29 = small,.30–.49 = medium,.50–1.00 = large). 95% BCa bootstrap (2000 samples) confidence intervals are presented for the correlation analyses.

The study was approved by the regional ethical review board of Uppsala, Dnr 2017/183.

3.1 Neuropsychological test of executive function –CLQT

Based on the CLQT total score, 79% of the participants performed below criterion cut score (Table 2). Symbol trails and Design generation were the most challenging subtests, with 65.8% and 73.7%, respectively, performing below cut score. The difference to criterion cut score was significant for CLQT total and all subtests except Mazes.

We found no systematic differences between the ratings by family members and formal care givers (Mann-Whitney U test, p-values ranging from .053 to .380), or between ratings by men and women (Mann-Whitney U test, p-values ranging from .125 to .971) on any of the clinical scales. The results of BRIEF-A are presented in Table 3. Higher scores indicate more severe problems, and a T score at or above 65 is considered to indicate clinically relevant executive dysfunction. Median T score was below 65 for all clinical scales. Only Working memory comes close with 16 participants (42.1%), having a T score above 65. In another three scales (Initiation, Planning/organising, and Task monitoring) about a fifth of the participants have median T scores above 65. A one-sample Wilcoxon signed rank test comparing T score median to hypothetical median of 65 shows that all subscale medians are significantly (all p < 0.001) below 65, except Working memory (p = 0.433). Based on the composite score GEC, 5.2% of the participants have clinically significant executive dysfunction.

Median score (inter-quartile range) for Scenario Test was 36 (19.5–44) out of maximum 54 (n = 37 for ST, as one participant did not complete the test), and for CETI 47.2 (37.0–59.1) out of maximum 100.

3.4 The relationship between performance-based tests (CLQT) and informant ratings (BRIEF-A)

Spearman’s rank correlations between subtests of CLQT and clinical scales of BRIEF-A are presented in Table 4. For both measures, raw scores are used. A higher CLQT score indicates better performance, while a higher BRIEF-A score indicates more frequent problems. Thus, negative correlations would be expected.

Six (10%) of the correlations were significant if adopting a significance level of p≤0.05. Five of the six significant correlations include CLQT Symbol trails, two BRIEF-A Working memory. Considering the risk of type 1 error introduced by adopting a somewhat generous significance level, the correlations between BRIEF-A Working memory and CLQT Symbol trails (r _s  = –0.61, p < 0.001), BRIEF-A Working memory and CLQT total (r _s  = –0.42, p = 0.009), and BRIEF-A MI and CLQT Symbol trails (r _s  = –.44, p = 0.006) appear to be the most robust. With Bonferroni correction (rendering an α-level of p≤0.0008) none of the correlations are significant.

Spearman’s rank correlations between clinical scales of BRIEF-A (raw scores) and the two measures of functional communication (Scenario Test and CETI) are presented in Table 5. No correlations with a p-value ≤.05 were found. The correlation between BRIEF-A Initiation and Scenario Test is the only one that approaches the significance level of p≤0.05 (r _s  = –.31, p = 0.063)

4.1 The relationship between neuropsychological test results and informant ratings of executive function in PWSA

Our results confirm that previously reported findings of few correlations between neuropsychological tests and ratings of executive function in other populations are also valid regarding PWSA. In fact, our findings indicate that in PWSA the correlation between the two types of instruments may be somewhat weaker than has been reported in other populations. In the review by Toplak et al. (2013), 19% of the correlations between BRIEF-A and various neuropsychological tests were considered statistically significant, whereas this was true for 10% of the possible correlations in the present study. A pattern emerged which indicates that results on CLQT Symbol trails are most closely related to the BRIEF-A results, since five of the six correlations with p≤0.05 include this task. However, when applying a significance level of p≤0.05, half of the significant correlations found (i.e. 5%) could be disregarded as possible products of chance. In fact, with Bonferroni correction, none of the correlations are significant. These results replicate findings in a study of several clinical samples by Lovstad et al. (2016), which identified no significant correlations between neuropsychological tests and informant ratings with BRIEF-A. Our results strongly question whether neuropsychological tests and informant ratings aiming to measure executive function succeed in tapping into the same construct when used with PWSA.

Additionally, the results indicate a consistent bias towards a more positive picture of PWSA functioning as measured on BRIEF-A compared to CLQT (only 1 participant had the reverse pattern). These results should be interpreted with some caution, since the cut-off scores of CLQT and BRIEF-A are based on American samples only. However, regarding the CLQT, our participants’ results are very similar to the ones of the left-hemisphere injured group reported by Helm-Estabrooks (2001), providing support that our participants perform as could be expected on this test. Regarding BRIEF-A, matters are a bit more complicated. The ratings are based upon what the informant perceives is a problem, which opens up for cultural differences regarding what is thought of as normal functioning and what might be considered a problem (Lovstad et al., 2016). In studies from Norway (which is culturally and socioeconomically more similar to Sweden than the USA) healthy control groups have been found to have lower mean scores than in the US normative data (Finnanger et al., 2015; Lovstad et al., 2012; Lovstad et al., 2016), which would lower the cut off for clinically relevant dysfunction (since higher scores indicate more problems). Lovstad et al. (2012) suggests that a cut off somewhere between 55–60 might be appropriate for Scandinavian populations. A rather large proportion of our participants fall within this “grey zone”; 17 participants (45%) have BRIEF-A GEC T scores between 65 (the US cut off) and 55, indicating that culture appropriate norms might have a large impact on this instrument’s potential for identifying possible clinically relevant problems.

However, the precarious interpretation of norms and cut offs does not impact the fact that the neuropsychological tests and informant ratings show very little correlation, and even if the bias is considerably less pronounced when comparing to Norwegian control group data, there is still a clear tendency that the informant ratings indicate less problems with executive function than do the neuropsychological test.

4.2 The relationship between the results of informant ratings on BRIEF-A and functional communication in people with severe aphasia

The second research question regarded the relationship between functional communication and BRIEF-A informant ratings. Executive function, as measured with neuropsychological tests, and functional communication have been shown to be related. We have previously reported a strong bivariate correlation between CLQT and Scenario Test (r _s  = 0.64, p < 0.001) and a moderate correlation between CLQT and CETI (r _s  = 0.42, p = 0.004) in a group from which the participants of this study are drawn (Olsson et al., 2019), possibly reflecting the impact of strategic skills on communication as described by Light and McNaughton (2014). If CLQT and BRIEF-A results did represent aspects of the same underlying construct, one would expect to find similar results when investigating the relationship between BRIEF-A and the measures of functional communication. Another possible result could have been that BRIEF-A could reveal previously undetected aspects of the relations between executive function and functional communication. However, all correlations were weak and none had p-values ≤.05 (Table 5). This suggests that BRIEF-A, in contrast to standardised neuropsychological tests, does not provide useful information about strategic or other skills that are directly relevant to functional communication.

4.3 Limitations

CLQT is an instrument intended for screening, and it is not sufficient for an in-depth assessment of level of cognitive functioning. Nonetheless, it was considered appropriate for this study since the main focus was not on establishing the absolute level of executive function in PWSA in relation to other populations, but investigating the relation between two different approaches to measuring executive function within this population. In addition, few standardised tests are appropriate for PWSA due to linguistic demands, and CLQT is one of few that is specifically intended for populations with communication disorders.

The ambition was originally to relate information about the PWSAs’ brain injuries to their scores on CLQT and BRIEF-A. This could have been interesting, since previous research has indicated different patterns of executive dysfunction with different sites of injury. For example, Lovstad et al. (2012) found that participants with dorsolateral prefrontal injuries demonstrated the pattern we found in the present study, with low performance on neuropsychological tests but relatively low levels of problems reported through self- or informant ratings, while participants with orbitofrontal injuries demonstrated the opposite pattern. Unfortunately, the information available from the medical records of our participants was frequently too imprecise to be used for such analyses.

4.4 Further research

BRIEF-A is a widely used instrument with good psychometric properties. However, the items leave the informant quite a lot of room for interpretation. The BRIEF-A questionnaire focuses on clinically relevant aspects of everyday functioning, however it may be more difficult to determine what the informant ratings represent. One way to investigate this issue further would be to conduct interviews with informants to gain insight into how they interpret and respond to the items. Such a study is ongoing.

Executive function status is an important factor in the rehabilitation of individuals following stroke, not least for those with severe aphasia. However, these patients are frequently not offered a proper assessment of their cognitive abilities, due to the challenges of using existing standardised instruments. As shown in the present study, informant ratings cannot be used as a substitute. Much work is needed to establish clinically feasible, valid ways of assessing executive (and other cognitive) functions in PWSA, a work that likely demands close cooperation between neuropsychologists and speech language pathologists.

Further research focusing on the analysis and understanding of the specific impact of the executive functions on functional communication is required. We know that some aspects of executive function (that can be captured by neuropsychological tests) are related to functional communication (Fridriksson et al., 2006; Olsson et al., 2019), a connection that also seems theoretically justified considering the importance of executive function to the use of compensatory strategies (Lewis et al., 2011). However, how can results of such tests be interpreted to help us design interventions, and which aspects of executive function are the most important? Considering the poor correlation between functional communication and BRIEF-A indicated by the present study, informant ratings of executive function does not seem to be the way forward in this specific issue. Perhaps, the impact of executive function on communication and communication intervention might be best understood if studied within the actual context of communication? An interesting attempt was made by Purdy and Koch (2006), in developing a new scoring system for Communicative Abilities of Daily Living (Holland, 1980) focusing on cognitive flexibility, and this seems to be an area worthy of continued research attention.

4.5 Conclusions

This study is, to our knowledge, the first investigating the use of informant ratings of executive function in PWSA. Our findings show that results of neuropsychological tests (CLQT) and informant rating (BRIEF-A) of executive function do not correlate with one another when used with PWSA. The executive impairments revealed by neuropsychological tests are not reflected in the informant ratings. Consequently, the two types of instrument cannot be assumed to measure the same underlying construct. Further, the lack of correlations between BRIEF-A and measures of functional communication indicates that aspects of executive function relevant to functional communication are not captured by BRIEF-A. These results need to be considered in clinical settings where assessment of executive function in PWSA is desired, since it is clear that informant ratings cannot be used as a substitute for neuropsychological tests. To provide a proper assessment of executive function in this population, instruments must apparently be chosen with great care.