Screening for Lifetime History of Traumatic Brain Injury Among Older American and Irish Adults at Risk for Dementia: Development and Validation of a Web-Based Survey

Abstract

Background:

Traumatic brain injury (TBI) is an established risk factor for dementia but mechanisms are uncertain. Accurate TBI exposure classification is critical for cognitive aging research studies seeking to discover mechanisms and treatments of post-TBI dementia. Brief TBI screens, commonly used in epidemiological studies of cognitive aging, are insensitive, leading to exposure mis-classification. Comprehensive TBI interviews, while more sensitive, may be impractical.

Objective:

We aimed to develop and validate a scalable, self-administered, comprehensive, web-based, TBI exposure survey for use in international cognitive aging research.

Methods:

We adapted a gold-standard comprehensive TBI interview (the Ohio State University TBI Identification Method; OSU TBI-ID) into a self-administered web-based survey for older adults (Older Adult modification of the OSU TBI-ID; OA OSU TBI-ID). We assessed reliability of our web-based survey versus the gold-standard interview among 97 older adults with normal cognition and mild cognitive impairment (MCI). In addition, we assessed sensitivity of the National Alzheimer’s Coordinating Center Uniform Data Set (NACC UDS) brief TBI screen versus the interview among 70 older adults with normal cognition.

Results:

Our OA OSU TBI-ID web-based survey had good to excellent reliability versus the interview (κ 0.66–0.73; ICCs 0.68–0.81) even among the sub-set with MCI (κ 0.74–0.88; ICCs 0.76–0.85), except for several age-at-injury variables. The NACC UDS brief TBI screen missed 50% of TBI exposures identified using the OSU TBI-ID interview.

Conclusion:

The OSU TBI-ID interview and web-based survey may facilitate more accurate TBI exposure classification in cognitive aging research thereby accelerating discovery of targetable mechanisms of post-TBI dementia.

Keywords

Clinical research cognitive aging reliability screening traumatic brain injury validation

INTRODUCTION

Traumatic brain injury (TBI) is common across the lifecourse and is an established risk factor for dementia [1, 2], increasing risk by two- to four-fold in some studies [3, 4]. While TBI in young adult athletes and veterans has received much recent attention, the highest incidence of TBI in civilian populations is in the elderly: One in 50 US adults age 75 years or older received medical care for a TBI in 2013, mostly due to ground-level falls [5]. The total lifetime prevalence of TBI among community-dwelling adults has been estimated at more than 40% for all-severity TBI [6] and more than 20% for TBI with loss of consciousness (LOC) [7]. Recently, several large epidemiological studies of administrative data have established that even mild TBI is an important risk factor for dementia in both civilians and military veterans [4 , 8–12]. Mild TBI comprises the vast majority of TBIs in the US and globally, affecting more than an estimated 42 million people worldwide every year [8]. Several studies have also found that increasing TBI severity or older age at injury is associated with increasing risk for dementia [4 , 8–12].

Mechanisms and prevention of post-TBI dementia, however, remain uncertain [13]. Thus, accurate measurement of total burden of lifetime TBI exposure, including number, severity, and timing of TBIs is of paramount importance in longitudinal cohort studies of cognitive aging that seek to unravel mechanisms and develop interventions for post-TBI cognitive decline and dementia. Relying solely on medical record documentation of TBI, while less prone to recall bias, under-estimates lifetime history of TBI exposure [14, 15]. Medical records are particularly inadequate for capturing lifetime history of TBI in older adults and those with milder TBIs as these sub-groups are the least likely to present to medical attention [14, 16]. Additionally, in aging cohorts, medical records may have never or may no longer exist for earlier-life exposures. For all of these reasons, it is usually necessary to rely on self-report in order to capture the lifetime history of TBI in aging cohorts. However, brief TBI exposure screens, comprised of one to three questions, may miss up to 80% of injuries that are identified using more comprehensive screens [16, 17]. In a large cohort study, a poorly sensitive screen will lead to substantial exposure mis-classification, thereby reducing power and limiting discovery of TBI-related sequelae.

Our goal in this study was to directly facilitate high-quality international cohort studies of post-TBI cognitive decline and dementia by developing and validating a low-cost, scalable, web-based, self-administered, comprehensive TBI exposure survey. Self-administration was considered important for scalability. Conducting a comprehensive TBI exposure interview requires staff training and may be prohibitively time-intensive or costly for many large longitudinal cohort studies of cognitive aging. First, we used qualitative methods to adapt an existing well-validated comprehensive semi-structured TBI exposure interview—the Ohio State University TBI Identification Method (OSU TBI-ID)—into an age-appropriate and culturally-appropriate interview and self-administered web-based survey appropriate for use in older adults residing in the US and Ireland. Next, we assessed test-retest reliability of the web-based survey against the ‘gold-standard’ interview in an international cohort of community-dwelling American and Irish older adults with normal cognition and mild cognitive impairment (MCI). Finally, to investigate the added value of using a comprehensive TBI screen versus a brief screen in a longitudinal cohort study of cognitive aging, we retrospectively evaluated the sensitivity and specificity of the brief TBI exposure screen currently used in the National Alzheimer’s Coordinating Center (NACC) Uniform Data Set (UDS) Version 3 [18] versus a comprehensive OSU TBI-ID interview.

METHODS

Study design and ethics

This was a 3-part study and took place at the University of California, San Francisco (UCSF), USA and Trinity College Dublin (Trinity), Ireland. Local ethics committees approved the study at both sites and all participants provided written informed consent. Part 1 (Survey Development) involved qualitative survey development at both sites and took place from March 2016 to March 2017. Part 2 (Survey Validation) was a two-site prospective cohort study with a within-participant repeated measures design and took place from April 2017 to November 2017. Part 3 (Screen Comparison) was a single-site retrospective cohort study leveraging data collected in an ongoing longitudinal cohort study of brain aging at UCSF between March 2016 and November 2017.

Sites and participants

At the US site, all participants were recruited from an ongoing longitudinal cohort study of brain aging. These US participants had all undergone neuropsychological testing and neurobehavioral evaluation by a neurologist within the past year and had a consensus diagnosis of normal cognition (e.g., not MCI or dementia) according to established research criteria [19].

At the Irish site, participants were recruited from several university hospital memory disorder clinics or directly from the community. Participants recruited from memory disorder clinics had undergone neuropsychological testing and a neurobehavioral evaluation by a neurologist within the past year and had a consensus clinical diagnosis of normal cognition or MCI according to established research criteria [19]. Participants recruited directly from the community were screened with the Montreal Cognitive Assessment tool (MoCA) and then categorized as dementia (MoCA < 18/30), MCI (MoCA 18–23/30), or normal cognition (MoCA 24–30/30) based on established cut-offs [20], normative data in Irish older adults [21], and expert opinion of a neurologist practicing in the university hospital memory disorder clinics (C.D.). Those categorized as dementia were excluded. Participants with MCI were encouraged to complete the interview and web-based survey with the assistance of an informant/caregiver who had known the participant for at least 5 years and who could read/speak English; however, this was not a requirement for participation.

Survey development methods

The Ohio State University TBI Identification Method (OSU TBI-ID)

The OSU TBI-ID is a validated, semi-structured, three-step interview that screens for and quantifies lifetime history of TBI including number of TBIs, severity of TBIs, and ages of TBIs [22, 23]. The OSU TBI-ID is a TBI Common Data Element and is recommended by the National Institutes of Neurological Disorders and Stroke (NINDS) for measurement of lifetime TBI exposure [24]. The instrument, including extensive administration and scoring information, is freely available on the internet [25]. The instrument has good to excellent inter-rater reliability, test/re-test reliability, and predictive validity for number and severity of TBIs, but was developed in predominantly younger to middle-aged adult populations [22 , 26]. While there are other similar lifetime TBI exposure screens available (such as the Brain Injury Screening Questionnaire [BISQ] [27]), we are not aware of any comprehensive lifetime TBI screens that have been specifically developed or validated in older adults. Thus, we chose to adapt the OSU TBI-ID because it is both an NINDS TBI Common Data Element and it is freely available online (unlike the BISQ) thereby facilitating our goal of developing a low-cost, scalable research instrument.

Qualitative development of the Preliminary Older Adult Modification of the OSU TBI-ID (Preliminary OA OSU TBI-ID) interview and web-based survey

Because the OSU TBI-ID was developed for use in US adults of all ages, we first developed a slightly modified version of the OSU TBI-ID tailored for older Irish and American adults. Beginning in March 2016, research assistants at each site were trained to administer the original un-modified OSU TBI-ID Interview and began administering the interview to older community dwelling adults at each site for either training purposes (Ireland) or as part of other ongoing longitudinal cohort studies of aging (US). During research team meetings, we elicited qualitative feedback from these interviewers about acceptability of the interview for older adults at each site and then made several targeted modifications to the instrument to improve acceptability (see the Supplementary Material for details of changes). We called this instrument the ‘Preliminary Older Adult Modification of the OSU TBI-ID (Preliminary OA OSU TBI-ID) Interview.’ Next, we developed a parallel web-based self-administered survey version of this instrument (Preliminary OA OSU TBI-ID Web-Based Survey). The web-based survey was created using REDCap, a HIPAA-compliant online data-management and data-collection platform [28]. Finally, we formally assessed readability of the Preliminary OA OSU TBI-ID Web-Based Survey by calculating the Flesch-Kincaid Grade Level (6.4) and Flesch-Kincaid Reading Ease score (68.4). These scores reflected that the instrument was written at approximately a 6^th grade reading level and had fairly good readability [29, 30].

Qualitative development of the final OA OSU TBI-ID interview and web-based survey

We administered the Preliminary OA OSU TBI-ID Interview (interviewer-administered, in-person interview) to 19 community-dwelling older adults (47% Ireland, 53% US, mean age 74±6.5 years [range 59–85 years], 63% female, 94% Caucasian, mean educational attainment 16.2±4.4 years of education, 16% MCI). Immediately following this interview, participants were given a structured ‘cognitive interview’ (see the Supplementary Material for a the full-text of the cognitive interview questions). Cognitive interviews are used widely in questionnaire development to detect problematic questions. They can identify the types of errors made by respondents, how they interpret and answer questions in a survey, and whether specific items are appropriate to a respondents’ cultural context [31]. These interviews were recorded and transcribed. Cultural and linguistic issues were flagged using the ‘behavior coding’ system described in detail by Napoles-Springer et al. [31]. Two to four weeks later, these same 19 participants were emailed a link to the Preliminary OA OSU TBI-ID Web-Based Survey which they completed independently. Respondents were then de-briefed by phone about the usability and acceptability of the web-based survey. Overall, participants at both sites found the interview easy to understand and reported that the interview aided their memory of remote injuries. Usability and acceptability of the web-based survey were also high with most respondents stating that the survey was easy to use and that the questions were straightforward, direct, and adequately captured their head injury history. Several minor web usability issues were identified that could be easily addressed (e.g., “I wasn’t positive that my response had gone through,” was addressed by adding a confirmation screen and an emailed confirmation after the respondent clicked on “submit.”). A few participants reported difficulty recalling the exact age of TBI. The perceived meaning of key phrases by participants at each site was highly accurate except that several participants erroneously stated that “injuries to your head or neck” meant only injuries that required medical attention in a hospital. Thus, a line was added to the instructions to clarify that “This survey asks about injuries to your head or neck ... even those that did not require medical attention.” Additional suggestions for improvement included clarifying that being hit by an object also counted as an injury and providing a few more examples in Step One that are relevant to older, rather than younger, adults. For example, 47% of injuries reported by these 19 participants were due to a fall. Thus, the Step One question, “Have you ever injured your head or neck in a fall or being hit by something?,” was separated into two separate questions and additional examples of falls were added such as, “ ... have you ever injured your head or neck in a fall (for example, falling in the house or outdoors ...)” Lastly, a question asking about interim injuries between TBI exposure instruments (interview and web-based survey) was added. The final optimized version of the OA OSU TBI-ID Interview and Web-Based Survey (full-text of interview, survey, and REDCap survey csv file for upload directly into REDCap are available in the Supplementary Material) has a Flesch-Kincaid Reading Ease Score of 69.9 and a Flesch-Kincaid Grade Level score of 6.2 reflecting approximately a 6th grade reading level and fairly good readability [29, 30].

Survey validation methods

Study design

This was a two-site prospective cohort study with a within-participant repeated measures design.

Participants

Inclusion criteria for participants at both sites were as follows: age 50 years or older, English-speaking, have access to a computer or tablet with internet connectivity, and ability to participate in both the in-person interview and web-based survey within two to four weeks of each other. Exclusion criteria at both sites included being a current prisoner or patient in custody, any pre-existing disabling medical, neurological, or psychiatric condition that would impair ability to complete the interview or survey (e.g., major stroke, multiple sclerosis, cancer, schizophrenia), diagnosis of dementia (specified further below), or non-English speaking.

As described above, all participants at the US site had a diagnosis of normal cognition based on a comprehensive neurobehavioral assessment; participants at the Irish site had a diagnosis of normal cognition or MCI based either on a comprehensive neurobehavioral assessment (if recruited from memory disorders clinics) or the MoCA (if recruited from the community). Participants with MCI were encouraged to complete the interview and web-based survey with the assistance of an informant/caregiver; however, this was not a requirement for participation and was not considered in the analysis. This was done to approximate how this instrument would likely be used in a research study of cognitive aging where a participant might elect to request help from a caregiver to complete the online survey either for technology support or for help recalling details of an injury.

Procedures

Upon providing written informed consent, baseline demographic and clinical variables were collected: age, sex, race/ethnicity, years of education, employment status, and cognitive diagnosis (normal versus MCI as described above). Each participant completed the OA OSU TBI-ID twice; once via the Web-Based Survey (self-administered on personal computer or tablet off-site) and once via the Interview (interviewer-administered, in-person at each site). The two instruments were spaced a minimum of seven days and a maximum of 28 days apart to minimize practice/recall effects and reduce likelihood of interim injury. Participants completing the second instrument outside of the 7–28 window were excluded from further analysis. The order of administration was randomized to balance practice/recall effects of one instrument on the other. The Web-Based Survey was accessible from any web-browser on any device with internet capability after navigating to the REDCap website and entering a unique pass code. This pass code was provided to participants via email.

Target enrollment was 50 participants at the US site (all with normal cognition) and 50 participants at the Irish site (half with normal cognition, half with MCI). Enrollment was capped upon enrolling N = 113 participants, of whom N = 97 met all inclusion criteria (See Fig. 1). Of those enrolled at the Irish site, N = 15 were recruited from memory disorders clinics (of whom, all were categorized as MCI) and N = 30 were recruited from the community (of whom N = 6 were categorized as MCI).

Fig.1

Study flow chart showing validation cohort participants included in the reliability analysis. Participants were included in the reliability analysis if they completed both the OA OSU TBI-ID interview and the OA OSU TBI-ID web-based survey spaced 7–28 days apart.

Screen comparison methods

Study design

This was a single-site retrospective cohort study using existing data collected in an ongoing longitudinal cohort study of brain aging at the US site from March 2016 through November 2017.

Participants

We retrospectively identified all cognitively normal participants age 50 years and older who completed both the OSU TBI-ID Interview (comprehensive TBI screen) and the National Alzheimer’s Coordinating Center (NACC) Uniform Data-Set (UDS) Version 3 TBI exposure screen (brief TBI screen) as part of their participation in an ongoing longitudinal cohort study of brain aging at the US site. All participants had undergone a comprehensive neurobehavioral assessment and were vetted as cognitively normal by expert consensus, as described above for cognitively normal participants at the US site. We excluded those who had also completed a Web-Based OSU TBI-ID Survey via participation in the Survey Development or Survey Validation studies described above.

We hypothesized that the NACC-UDS brief TBI screen would be poorly sensitive compared to a comprehensive “gold-standard” screen. Thus, we intentionally excluded individuals with cognitive impairment from the Screen Comparison study in order to investigate the accuracy of the NACC-UDS brief screen under the best possible circumstances in an aging cohort. If the NACC-UDS performed poorly even among cognitively normal older adults, as we hypothesized, then there would be no added value to investigating the performance of this screen among cognitively impaired older adults.

TBI screens

Either the unmodified OSU TBI-ID, preliminary OA OSU TBI-ID, or final OA OSU TBI-ID Interview (described in detail above) was administered by a trained research assistant as part of a larger battery of exposure and outcome surveys. The NACC UDS TBI screen was separately administered as part of a comprehensive neurobehavioral examination by a board-certified neurologist. The NACC UDS TBI screen consists of a single question about presence of “traumatic brain injury.” If present, there are four follow-up questions that record details of LOC and year of most recent TBI. The research assistants and neurologists were not specifically blinded to the results of the other TBI screen. However, it is considered unlikely that they each would have been aware of the findings of the other screen due to the flow of the study visits with multiple separate evaluations by different staff members without any intervening meetings between research assistants and neurologists. The order of administration of instruments was unknown and contingent only on scheduling considerations. Because our aim was to compare the NACC-UDS brief screen to a comprehensive TBI screen, we included participants who had completed any version of the OSU TBI-ID, all of which are considered comprehensive screens. Furthermore, the overall structure and content of the three versions of the OSU TBI-ID included in the screen comparison study is extremely similar (see the final OA OSU TBI-ID interview form provided in the Supplementary Materials versus the unmodified OSU TBI-ID interview form available on the web [32]).

Statistical analysis

All statistical analyses were performed using Stata 13 or Stata 15 (StataCorp LLC, TX, USA) [33, 34]. Participant characteristics were summarized using summary statistics. For Survey Validation, variables that were evaluated in previous reliability studies of the OSU TBI-ID were analyzed [23, 35]. Definitions for each of these variables are shown in Table 1. Reliability of the OA OSU TBI-ID Web-Based Survey versus the OA OSU-TBI-ID Interview was calculated using intraclass correlation coefficients (ICC) for continuous variables, unweighted Cohen κ for dichotomous variables, and both ICC and weighted Cohen κ for ordinal variables. To calculate ICCs, we used two-way random effects models evaluating for consistency, as described previously [26]. We additionally calculated reliability after stratification of the cohort by cognitive status, site, and order of instrument administration. ICC and Cohen κ were interpreted using the following commonly used benchmarks [36]: <0.4 poor reliability, 0.4–0.59 fair reliability, 0.6–0.74 good reliability, 0.75–1.0 excellent reliability. For Screen Comparison, fewer TBI variables were available for analysis given the brief nature of the NACC UDS TBI screen. TBI occurrence yes/no for any TBI and specifically for TBI with LOC was analyzed using unweighted Cohen κ. Additionally, the sensitivity, specificity, and overall accuracy of the brief screen compared to the comprehensive OSU TBI-ID interview were calculated.

Table 1

Traumatic brain injury variables examined in the survey validation study

Variable	Definition
Any TBI	Indicates whether the person reports any TBI occurred. TBI is defined as injury to the head or neck (“Yes” to any Step 1 question) with at least one reported injury resulting in being dazed, having a gap in memory, or LOC (determined via Step 2 questions).
TBI with LOC	Indicates whether the person reports any TBI with LOC occurred
TBI with LOC ≥ 30 min	Indicates whether the person reports any TBI with LOC≥30 min occurred
TBI before age 15 years	Indicates whether the first TBI occurred before age 15 years
TBI with LOC before age 15 years	Indicates whether the first TBI with LOC occurred before age 15 years
Number of TBIs	A count of the number of TBIs
Number of TBIs with LOC	A count of the number of TBIs with LOC
Number of TBIs with LOC ≥ 30 min	A count of the number of TBIs with LOC that equaled or exceeded 30 min
Number of mild TBIs	A count of the number of TBIs with no LOC or LOC < 30 min
Number of mild TBIs with LOC	A count of the number of TBIs with LOC > 0 and <30 min
Number of TBIs with LOC or requiring hospitalization	A count of the number of TBIs with LOC or requiring hospitalization
Age at first TBI	Age at which person’s first TBI occurred
Age at first TBI with LOC	Age at which person’s first TBI with LOC occurred
Injury group	Most severe lifetime injury based on the following severity gradient: No injury (“No” to all Step 1 Questions) = 1; injury but no TBI (“Yes” to at least one Step 1 Question, but injury does not meet above criteria for TBI) = 2; TBI with no LOC (at least one reported injury resulting in being dazed or having a gap in memory, but no LOC) = 3; TBI with LOC = 4
Worst TBI	Most severe lifetime TBI based on the following severity gradient: No TBI = 1; TBI with no LOC = 2; TBI with LOC < 30 min = 3; TBI with LOC≥30 min but <24 h = 4; TBI with LOC≥24 h = 5
Worst Injury	Most severe lifetime injury based on the following severity gradient: No injury = 1; injury but no TBI = 2; TBI with no LOC = 3; TBI with LOC < 30 min = 4; TBI with LOC≥30 min but < 24 h = 5; TBI with LOC≥24 h = 6

TBI, traumatic brain injury; LOC, loss of consciousness.

RESULTS

Survey validation results

Of the 113 enrolled participants, 104 (92%) completed the OA OSU TBI-ID Web-Based Survey. 97 (86%) completed both instruments within the required timeframe and were included in the reliability analysis (Fig. 1). All participants denied interim injuries between instruments. Participant characteristics are shown in Table 2.

Table 2

Characteristics of survey validation cohort and screen comparison cohort

Variable Values are n(%) or Mean(SD)	Survey Validation Cohort (n = 97; reliability of OA OSU TBI-ID web-based survey versus interview)	Screen Comparison Cohort (n = 70; sensitivity/specificity of NACC UDS brief) TBI screen versus OSU TBI-ID interview)
Age	70.9 (8.7)	73.1 (6.8)
Female	53 (54.6%)	39 (55.7%)
White race	88 (90.7%)	59 (84.3%)
Education (y)	15.9 (3.2)	18.0 (1.9)
Employment
Employed	27 (27.8%)	16 (22.9%)
Unemployed	6 (6.2%)	0 (0.0%)
Retired	63 (65.0%)	52 (74.3%)
Missing	1 (1.0%)	2 (2.9%)
Cognitive Diagnosis
Normal cognition	76 (78.4%)	70 (100%)
Mild cognitive impairment	21 (21.6%)	0 (0.0%)
Site
USA	52 (53.6%)	70 (100%)
Ireland	45 (46.4%)	0 (0.0%)

OA OSU TBI-ID, Older Adult Modification of the Ohio State University Traumatic Brain Injury Identification Method; OSU TBI-ID, Ohio State University Traumatic Brain Injury Identification Method; NACC UDS, National Alzheimer’s Coordinating Center Uniform Data Set; TBI, traumatic brain injury.

Based on the ‘gold-standard’ in-person OA OSU TBI-ID Interview, the lifetime prevalence of at least one TBI was 44%. 33/76 (43%) of cognitively normal participants and 10/21 (48%) of MCI participants endorsed lifetime history of TBI. Falls were the most commonly reported mechanism of injury (56% of all reported injuries), followed by motor vehicle accidents (47%) and sports related accidents (37%). TBIs were also reported to occur as a result of being hit by something (28%) and through assault (9%).

Overall, reliability of the Web-Based Survey versus the Interview ranged from good to excellent (κ 0.66–0.73; ICCs 0.68–0.81) for all variables except “TBI with LOC before age 15 years,” which was fair (κ 0.54). Reliability was similar to or exceeded that of prior reliability studies of the OSU TBI-ID (Table 3).

Table 3

Reliability of the OA OSU TBI-ID Web-Based survey vs. Interview compared to prior reliability studies of the OSU TBI-ID

	Current OA OSU TBI-ID survey validation study			Prior OSU TBI-ID Reliability Studies
	Test re-test reliability of web-based survey versus interview (N = 97 older community-dwelling adults without dementia)			Bogner et al., 2017 [26] (test-retest reliability of interview; N = 223 patients age ≥ 16 years, all with moderate to severe TBI		Cuthbert et al., 2016 [35] test-retest reliability of (computer-assisted telephone interview; N = 194 stratified random sample of Colorado residents)		Bogner and Corrigan, 2009 [23] (test-retest reliability of interview; N = 210 US prison inmates)
	N (%) or Mean (SD) based on interview	ICC	κ	ICC	κ	ICC	κ	ICC	κ
Any TBI	43 (44.3%)	–	0.72			–	0.50	–	–
TBI with LOC	24 (24.7%)	–	0.67			–	0.56	–	–
TBI with LOC ≥ 30 min	0 (0%)	–	N/A			–	0.62	–	–
TBI before age 15 years	18 (18.6%)	–	0.73			–	0.34	–	0.75
TBI with LOC before age 15 years	7 (7.2%)	–	0.54		0.96	–	0.32	–	0.71
Number of TBIs	1.58 (0.85)	0.75	–			0.09	–	0.88	–
Number of TBIs with LOC	0.79 (0.83)	0.72	–	0.81		0.21	–	0.86	–
Number of TBIs with LOC ≥ 30 min	0 (0)	none	–	0.87		0.70	–	0.70	–
Number of mild TBIs	1.58 (0.85)	0.75	–			0.06	–	–	–
Number of mild TBIs with LOC	0.79 (0.83)	0.72	–			0.18	–	–	–
Number of TBIs with LOC or requiring hospitalization	0.95 (0.87)	0.81	–			–	–	–	–
Age at first TBI	26.4 (20.4)	0.80	–			0.39	–	0.67	–
Age at first TBI with LOC	33.6 (23.2)	0.68	–			0.48	–	0.63	–
Injury group		0.73	0.66			0.62	–	–	–
No major injury	21 (21.7%)
Major injury but no TBI	33 (34.0%)
TBI with no LOC	19 (19.6%)
TBI with LOC	24 (24.7%)
Worst TBI		0.74	0.69	0.89	0.89	0.77	–	0.91	–
No TBI	54 (55.7%)
TBI with no LOC	19 (19.6%)
TBI with LOC < 30 min	24 (24.7%)
TBI with LOC 30 min to <24 h	0 (0%)
TBI with LOC≥24 h	0 (0%)
Worst Injury		0.73	0.66			0.78	–	–	–
No major injury	21 (21.7%)
Major injury but no TBI	33 (34.0%)
TBI with no LOC	19 (19.6%)
TBI with LOC < 30 min	24 (24.7%)
TBI with LOC 30 min to <24 h	0 (0%)
TBI with LOC≥24 h	0 (0%)

SD, standard deviation; ICC, intraclass correlation coefficient; κ, kappa coefficient; TBI, traumatic brain injury; LOC, loss of consciousness. Other abbreviations as in Table 2.

Reliability stratified by cognitive diagnosis, site, and order of administration is shown in Table 4. Reliability among participants with MCI was excellent (κ 0.74–0.88; ICCs 0.76–0.85) except for variables pertaining to age at injury which was fair to poor (κ 0.45; ICCs 0.30–0.50). Reliability of all variables, including those pertaining to age at injury among participants with normal cognition, however, was good to excellent (κ 0.61–0.78; ICCs 0.69–0.91). Reliability was slightly better in Ireland (where one research assistant administered all study interviews) versus the US (where a team of research assistants administered the interviews) except for variables pertaining to age at injury, which were slightly better at the US site (where all participants had normal cognition). Reliability was substantially better for participants who completed the web-based survey first compared to those who completed the interview first.

Table 4

Reliability of the OA OSU TBI-ID web-based survey stratified by cognitive diagnosis, site, and order of administration

	Cognitive diagnosis				Site				Order of administration
	Normal cognition N = 76		MCI N = 21		USA N = 52		Ireland N = 45		Interview first N = 44		Web-based survey first N = 53
ICC	κ	ICC	κ	ICC	κ	ICC	κ	ICC	κ	ICC	κ
Any TBI		0.70		0.80		0.69		0.73		0.56		0.85
TBI with LOC		0.61		0.88		0.66		0.68		0.53		0.76
TBI with LOC ≥ 30 min		none		none		none		none		none		none
TBI before age 15 years		0.78		0.45		0.85		0.60		none		0.95
TBI with LOC before age 15 years		0.64		none		0.78		none		none		0.70
Number of TBIs	0.74		0.76		0.74		0.74		0.82		0.72
Number of TBIs with LOC	0.69		0.85		0.70		0.77		0.58		0.84
Number of TBIs with LOC or requiring hospitalization	0.80		0.85		0.72		0.88		0.76		0.86
Age at first TBI	0.91		0.50		0.99		0.70		0.51		0.97
Age at first TBI with LOC	0.81		0.30		0.99		0.41		0.03		0.96
Injury group	0.71	0.63	0.81	0.74	0.72	0.63	0.74	0.68	0.61	0.50	0.85	079
Worst TBI	0.72	0.65	0.84	0.84	0.76	0.67	0.73	0.69	0.62	0.55	0.83	0.79
Worst Injury	0.71	0.63	0.81	0.74	0.72	0.63	0.74	0.68	0.61	0.50	0.85	0.79

Abbreviations as in Tables 2 and 3.

Screen comparison results

70 participants met inclusion criteria for the screen comparison study (Table 2). Overall, 24/70 (34.3%) of participants endorsed a lifetime history of at least one TBI of any severity on the OSU TBI-ID Interview versus 17/30 (24.3%) on the NACC UDS TBI screen. Compared to the OSU TBI-ID Interview, the NACC UDS TBI screen had a sensitivity of 50.0%, specificity of 89.1%, false positive rate of 10.9%, false negative rate of 50.0%, and correctly classified 75.7% of individuals with a lifetime history of TBI. Agreement between instruments was fair (κ 0.42). If only TBI with LOC was considered, findings were similar: sensitivity of 46.2%, specificity 93.0%, false positive rate 7.0%, false negative rate 53.9%, and correctly classified 84.3% of individuals with a lifetime history of TBI with LOC. Agreement between instruments was fair (κ 0.43).

DISCUSSION

We successfully adapted the OSU TBI-ID into an age- and culturally-appropriate TBI exposure interview and web-based survey for use in older community-dwelling adults in the US and Ireland who are at risk for dementia. We established reliability of our web-based survey versus the gold standard interview for comprehensively measuring lifetime history of TBI in this population. We demonstrated the added value of using a comprehensive OSU TBI-ID screen versus a brief screen in longitudinal cohort studies of cognitive aging. Specifically, we found that our web-based OSU TBI-ID survey is reliable as compared to the gold standard OSU TBI-ID interview for gathering all types of TBI history variables among older community dwelling adults in the US and Ireland with the exception of variables pertaining to age at injury among individuals with MCI.

Reliability of our web-based OA OSU TBI-ID survey was similar or superior to that of prior test-retest reliability studies of the OSU TBI-ID. This includes studies investigating test re-test reliability of the OSU TBI interview in US prison inmates [23], adults with moderate to severe TBI participating in the TBI Model Systems study [26], and the computer-assisted telephone interview (CATI) version of the OSU TBI-ID among adults residing in Colorado [35]. Recently, preliminary feasibility and predictive validity of a similar web-based self-administered OSU TBI-ID survey was demonstrated in a sample of US adults of all ages [37]. In this study, adults age 18 years and older (mean age 47 years, range 18–82 years) electively completed a web-based OSU TBI-ID via SurveyMonkey. Nearly all participants completed the survey in two to eight minutes. TBI severity was significantly associated with worse self-reported neurobehavioral symptoms, thereby establishing predictive validity of the survey [37]. Our study adds to this prior work by demonstrating reliability of a web-based OSU TBI-ID survey versus an in-person ‘gold standard’ OSU TBI-ID interview specifically in an older adult population with normal cognition and MCI. Our study thereby paves the way for use of this instrument in large cohort studies of cognitive aging.

In our survey reliability study, reliability was better among those who took the web-based survey before the interview as compared to those who were interviewed first. One possible explanation for this order effect is that self-administration of the web-based survey may have been more memorable to participants than the interview leading to greater recall effect (and higher reliability for survey-first participants). A related explanation is that respondents completing the web-based survey at home might have received assistance from another person which could further prime recall and carry over to the interview. An alternative explanation is that the semi-structured interview inadvertently informed the participants’ opinion about what should be considered a reportable injury thereby leading to modified/different reporting of injuries in the web-based survey (and lower reliability in interview-first participants). This finding therefore raises the testable hypothesis that test-retest reliability of the web-based survey may be superior than test-retest reliability of the interview.

Our screen comparison study found that while specificity of the NACC-UDS TBI screen was high, sensitivity was poor. This brief screen missed 50% of all-severity TBI exposures and 54% of TBI with LOC exposures reported via the OSU TBI-ID interview in a cohort of older cognitively normal adults. While it is surprising that sensitivity of the NACC-UDS was not higher among individuals with TBI with LOC (compared to those without LOC), one explanation for this finding is that most of the injuries reported in our study were extremely mild with only brief LOC. Thus, it remains possible that the NACC-UDS screen might perform better in a population with a higher prevalence of moderate to severe TBI with more prolonged periods of LOC. Overall, however, our findings are consistent with what has been reported in prior studies in predominantly younger adult populations of prisoners and veterans—who have a higher prevalence of moderate to severe TBI compared to our study population—which have reported that single-item TBI screens may miss up to 80% of TBI exposures identified via a comprehensive interview [16, 17].

Prevalence of TBI as measured by the OSU TBI-ID interview, differed slightly across our Survey Validation cohort (44%) versus our Screen Comparison cohort (34%). There are several likely contributors to this difference including chance and the relatively small sample size of these studies. Differential participant burden across the two studies (e.g., those with a history of TBI might have been more likely to complete the web-based OA OSU TBI-ID and therefore be included in the Survey Validation cohort), differential cognitive status across the two studies (normal cognition plus MCI versus normal cognition only), and differential nationalities across the two studies (US plus Irish versus US only) are also likely to have influenced results.

Exposure mis-classification may be non-differential (e.g., occurs at equal rates among those with and without the outcome of interest) or differential (e.g., occurs at a different rate among those with versus without the outcome of interest) [38]. If non-differential, the analysis may be biased toward the null thereby reducing power to detect an association between exposure and outcome if one is present. If differential, the analysis may be falsely biased away from the null. For example, if patients with dementia are less likely to remember and report their lifetime history of TBI in response to a brief screen as compared to cognitively normal controls, then a case-control study using a brief TBI screen may be biased toward identifying a protective effect of TBI on risk of dementia (e.g., a negative association). Importantly, our reliability study demonstrated that our comprehensive web-based OA OSU TBI-ID survey had excellent reliability even among individuals with MCI.

There have been several prior published studies using NACC-UDS TBI data to investigate associations of TBI exposure with various dementia-related outcomes [39 –44]. A number of these prior studies have failed to identify statistically significant associations between history of TBI and risk of the dementia outcomes of interest, although several had identified non-significant trends [40 –42]. Our findings suggest that more accurate exposure classification in NACC-UDS data, as could be achieved with the OA OSU TBI-ID interview or web-based survey, would likely have increased power and possibly reduced bias of these prior epidemiological analyses.

Our study has many strengths including combining qualitative methods for survey development and quantitative methods for survey validation and screen comparison. Additional strengths include being a two-site international study and inclusion of older adults with both normal cognition and MCI, which is important for generalizability to large longitudinal cohort studies of cognitive aging. Our survey validation study is limited by the relatively small size of our cohort and lack of measurement of time needed to complete the survey. Future studies could assess feasibility of the OA OSU TBI-ID web-based survey in a large population of older English speaking adults and determine whether completion time is indeed eight minutes or less as reported for the SurveyMonkey web-based OSU TBI-ID [37]. Future studies could additionally investigate whether the OA OSU TBI-ID is valid and reliable in younger populations. None of the participants in our survey validation study endorsed a history of moderate or severe TBI, potentially limiting generalizability to populations with more severe TBI. Prior studies of reliability of the OSU TBI-ID, however, have found somewhat higher reliabilities among populations with more severe TBI (Table 3), likely because milder injuries may be less memorable. These prior studies suggest that reliability of our OA OSU TBI-ID survey in older adults with a history of moderate to severe TBI would be expected to be as high if not higher than what we found in this study of older adults with and without mTBI.

In conclusion, we have demonstrated that older community-dwelling adults with MCI and normal cognition are capable of providing reliable self-reports of lifetime history of TBI via a web-based survey. We further demonstrate that the OSU TBI-ID interview is substantially more sensitive for both history of all severity TBI and history of TBI with LOC than the brief NACC UDS TBI screen. The web-based OA OSU TBI-ID survey and interview may enhance current and future longitudinal cohort studies of cognitive aging that seek to unravel mechanisms of TBI-related cognitive decline and dementia.

Footnotes

ACKNOWLEDGMENTS

This work was supported by the Global Brain Health Institute (to RCG and FW), The American Federation for Aging Research (to RCG), the National Institute for Neurological Disorders and Stroke (K23 NS095755 to RCG), and the Larry L. Hillblom Foundation (2014-A-004-NET and 2018-A-006-NET to JK). Dr. Corrigan’s and Bogner’s time was supported by a grant from the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR) to Ohio State University (90DPTB0001). NIDILRR is a Center within the Administration for Community Living (ACL), Department of Health and Human Services (HHS). The contents of this publication do not necessarily represent the policy of NIDILRR, ACL, or HHS, and you should not assume endorsement by the Federal Government.

Authors’ disclosures available online ().

The supplementary material is available in the electronic version of this article: .

References

Fleminger

, Oliver

, Lovestone

, Rabe-Hesketh

, Giora

(2003) Head injury as a risk factor for Alzheimer’s disease: The evidence 10 years on; a partial replication. J Neurol Neurosurg Psychiatry 74, 857–862.

Perry

, Sturm

, Peterson

, Pieper

, Bullock

, Boeve

, Miller

, Guskiewicz

, Berger

, Kramer

, Welsh-Bohmer

(2016) Association of traumatic brain injury with subsequent neurological and psychiatric disease: A meta-analysis. J Neurosurg 124, 511–526.

Plassman

, Havlik

, Steffens

, Helms

, Newman

, Drosdick

, Phillips

, Gau

, Welsh-Bohmer

, Burke

, Guralnik

, Breitner

(2000) Documented head injury in early adulthood and risk of Alzheimer’s disease and other dementias. Neurology 55, 1158–1166.

Barnes

, Byers

, Gardner

, Seal

, Boscardin

, Yaffe

(2018) Association of mild traumatic brain injury with and without loss of consciousness with dementia in US Military Veterans. JAMA Neurol 75, 1055–1061.

Taylor

, Bell

, Breiding

, Xu

(2017) Traumatic brain injury-related emergency department visits, hospitalizations, and deaths – United States, 2007 and 2013. MMWR Surveill Summ 66, 1–16.

Whiteneck

, Cuthbert

, Corrigan

, Bogner

(2016) Prevalence of self-reported lifetime history of traumatic brain injury and associated disability: A statewide population-based survey. J Head Trauma Rehabil 31, E55–62.

Corrigan

, Yang

, Singichetti

, Manchester

, Bogner

(2018) Lifetime prevalence of traumatic brain injury with loss of consciousness. Inj Prev 24, 396–404.

Gardner

, Yaffe

(2015) Epidemiology of mild traumatic brain injury and neurodegenerative disease. Mol Cell Neurosci 66, 75–80.

Gardner

, Burke

, Nettiksimmons

, Kaup

, Barnes

, Yaffe

(2014) Dementia risk after traumatic brain injury vs nonbrain trauma: The role of age and severity. JAMA Neurol 71, 1490–1497.

10.

Nordstrom

, Nordstrom

(2018) Traumatic brain injury and the risk of dementia diagnosis: A nationwide cohort study. PLoS Med 15, e1002496.

11.

Lee

, Hou

, Lee

, Hsu

, Huang

, Su

(2013) Increased risk of dementia in patients with mild traumatic brain injury: A nationwide cohort study. PLoS One 8, e62422.

12.

Fann

, Ribe

, Pedersen

, Fenger-Gron

, Christensen

, Benros

, Vestergaard

(2018) Long-term risk of dementia among people with traumatic brain injury in Denmark: A population-based observational cohort study. Lancet Psychiatry 5, 424–431.

13.

Mendez

(2017) What is the relationship of traumatic brain injury to dementia? J Alzheimers Dis 57, 667–681.

14.

Setnik

, Bazarian

(2007) The characteristics of patients who do not seek medical treatment for traumatic brain injury. Brain Inj 21, 1–9.

15.

Powell

, Ferraro

, Dikmen

, Temkin

, Bell

(2008) Accuracy of mild traumatic brain injury diagnosis. Arch Phys Med Rehabil 89, 1550–1555.

16.

Diamond

, Harzke

, Magaletta

, Cummins

, Frankowski

(2007) Screening for traumatic brain injury in an offender sample: A first look at the reliability and validity of the Traumatic Brain Injury Questionnaire. J Head Trauma Rehabil 22, 330–338.

17.

Russell

, Devore

, Barnes

, Forster

, Hostetter

, Montgomery

, Casey

, Kane

, Brenner

(2013) Challenges associated with screening for traumatic brain injury among US veterans seeking homeless services. Am J Public Health 103 Suppl 2, S211–212.

18.

National Alzheimer’s Coordinating Center: UDS forms and documentation, NACC, https://www.alz.washington.edu/WEB/forms_uds.html, Accessed August 2, 2019.

19.

Albert

, DeKosky

, Dickson

, Dubois

, Feldman

, Fox

, Gamst

, Holtzman

, Jagust

, Petersen

, Snyder

, Carrillo

, Thies

, Phelps

(2011) The diagnosis of mild cognitive impairment due to Alzheimer’s disease: Recommendations from the National Institute on Aging-Alzheimer’s Association workgroups on diagnostic guidelines for Alzheimer’s disease. Alzheimers Dement 7, 270–279.

20.

Nasreddine

, Phillips

, Bedirian

, Charbonneau

, Whitehead

, Collin

, Cummings

, Chertkow

(2005) The Montreal Cognitive Assessment, MoCA: A brief screening tool for mild cognitive impairment. J Am Geriatr Soc 53, 695–699.

21.

Kenny

, Coen

, Frewen

, Donoghue

, Cronin

, Savva

(2013) Normative values of cognitive and physical function in older adults: Findings from the Irish Longitudinal Study on Ageing. J Am Geriatr Soc 61 Suppl 2, S279–290.

22.

Corrigan

, Bogner

(2007) Initial reliability and validity of the Ohio State University TBI Identification Method. J Head Trauma Rehabil 22, 318–329.

23.

Bogner

, Corrigan

(2009) Reliability and predictive validity of the Ohio State University TBI identification method with prisoners. J Head Trauma Rehabil 24, 279–291.

24.

National Institutes of Neurological Diseases and Stroke Common Data Elements: Traumatic Brain Injury [online], National Institutes of Neurological Diseases and Stroke, https://www.commondataelements.ninds.nih.gov/Traumatic%20Brain%20Injury.

25.

Ohio State University TBI Identification Method Website,The Ohio State University Wexner Medical Center, https://wexnermedical.osu.edu/neurological-institute/departments-and-centers/research-centers/ohio-valley-center-for-brain-injury-prevention-and-rehabilitation/osu-tbi-id, Accessed July 29, 2019.

26.

Bogner

, Whiteneck

, MacDonald

, Juengst

, Brown

, Philippus

, Marwitz

, Lengenfelder

, Mellick

, Arenth

, Corrigan

(2017) Test-retest reliability of traumatic brain injury outcome measures: A traumatic brain injury model systems study. J Head Trauma Rehabil 32, E1–E16.

27.

Dams-O’Connor

, Cantor

, Brown

, Dijkers

, Spielman

, Gordon

(2014) Screening for traumatic brain injury: Findings and public health implications. J Head Trauma Rehabil 29, 479–489.

28.

Harris

, Taylor

, Thielke

, Payne

, Gonzalez

, Conde

(2009) Research electronic data capture (REDCap)–a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 42, 377–381.

29.

Flesch

(1948) A new readability yardstick. J Appl Psychol 32, 221–233.

30.

Readable.io, https://readable.io/, Accessed February 14, 2019.

31.

Napoles-Springer

, Santoyo-Olsson

, O’Brien

, Stewart

(2006) Using cognitive interviews to develop surveys in diverse populations. Med Care 44, S21–30.

32.

Ohio State University TBI Identification Method – Interview Form, Wexner Medical Ohio State University, https://wexnermedical.osu.edu/-/media/files/wexnermedical/patient-care/healthcare-services/neurological-institute/departments-and-centers/research-centers/ohio-valley/osu_tbiform_july2013.pdf, Accessed December 24, 2019.

33.

StataCorp (2013) StataCorp LP, College Station, TX.

34.

StataCorp (2017) StataCorp LLC, College Station, TX.

35.

Cuthbert

, Whiteneck

, Corrigan

, Bogner

(2016) The reliability of a computer-assisted telephone interview version of the Ohio State University Traumatic Brain Injury Identification Method. J Head Trauma Rehabil 31, E36–42.

36.

Cicchetti

(2001) The precision of reliability and validity estimates re-visited: Distinguishing between clinical and statistical significance of sample size requirements. J Clin Exp Neuropsychol 23, 695–700.

37.

Lequerica

, Lucca

, Chiaravalloti

, Ward

, Corrigan

(2018) Feasibility and preliminary validation of an online version of the Ohio State University Traumatic Brain Injury Identification Method. Arch Phys Med Rehabil 99, 1811–1817.

38.

Kesmodel

(2018) Information bias in epidemiological studies with a special focus on obstetrics and gynecology. Acta Obstet Gynecol Scand 97, 417–423.

39.

Mendez

, Paholpak

, Lin

, Zhang

, Teng

(2015) Prevalence of traumatic brain injury in early versus late-onset Alzheimer’s disease. J Alzheimers Dis 47, 985–993.

40.

LoBue

, Denney

, Hynan

, Rossetti

, Lacritz

, Hart

, Womack

, Woon

, Cullum

(2016) Self-reported traumatic brain injury and mild cognitive impairment: Increased risk and earlier age of diagnosis. J Alzheimers Dis 51, 727–736.

41.

Tripodis

, Alosco

, Zirogiannis

, Gavett

, Chaisson

, Martin

, McClean

, Mez

, Kowall

, Stern

(2017) The effect of traumatic brain injury history with loss of consciousness on rate of cognitive decline among older adults with normal cognition and Alzheimer’s disease dementia. J Alzheimers Dis 59, 251–263.

42.

Nguyen

, Schaffert

, LoBue

, Womack

, Hart

, Cullum

(2018) Traumatic brain injury and age of onset of dementia with Lewy bodies. J Alzheimers Dis 66, 717–723.

43.

Sayed

, Culver

, Dams-O’Connor

, Hammond

, Diaz-Arrastia

(2013) Clinical phenotype of dementia after traumatic brain injury. J Neurotrauma 30, 1117–1122.

44.

Dams-O’Connor

, Spielman

, Hammond

, Sayed

, Culver

, Diaz-Arrastia

(2013) An exploration of clinical dementia phenotypes among individuals with and without traumatic brain injury. Neurorehabilitation 32, 199–209.