How Generalizable Are Findings from a Community-Based Prospective Cohort Study? Extending Estimates from the Adult Changes in Thought Study to Its Source Population

Abstract

Background:

The Adult Changes in Thought (ACT) study is a cohort of Kaiser Permanente Washington members ages 65+ that began in 1994.

Objective:

We wanted to know how well ACT participants represented all older adults in the region, and how well ACT findings on eye disease and its relationship with Alzheimer’s disease generalized to all older adults in the Seattle Metropolitan Region.

Methods:

We used participation weights derived from pooling ACT and Behavioral Risk Factor Surveillance System (BRFSS) data to estimate prevalences of common eye diseases and their associations with Alzheimer’s disease incidence. Cox proportional hazards models accounted for age, education, smoking, sex, and APOE genotype. Confidence intervals for weighted analyses were bootstrapped to account for error in estimating the weights.

Results:

ACT participants were fairly similar to older adults in the region. The largest differences were more self-reported current cholesterol medication use in BRFSS and higher proportions with low education in ACT. Incorporating the weights had little impact on prevalence estimates for age-related macular degeneration or glaucoma. Weighted estimates were slightly higher for diabetic retinopathy (weighted 5.7% (95% Confidence Interval 4.3, 7.1); unweighted 4.1% (3.6, 4.6)) and cataract history (weighted 51.8% (49.6, 54.3); unweighted 48.6% (47.3, 49.9)). The weighted hazard ratio for recent diabetic retinopathy diagnosis and Alzheimer’s disease was 1.84 (0.34, 4.29), versus 1.32 (0.87, 2.00) in unweighted ACT.

Conclusions:

Most, but not all, associations were similar after participation weighting. Even in community-based cohorts, extending inferences to broader populations may benefit from evaluation with participation weights.

Keywords

Alzheimer’s disease bias dementia epidemiological research design eye diseases generalizability integrated health-care delivery systems prevalence transportability

INTRODUCTION

Dementia research study participants are a select group of older adults [1 –6], leading to concerns about generalizability of findings [1 , 8]. There are large population-representative surveys such as the Health and Retirement Study [8] and the Behavioral Risk Factor Surveillance System (BRFSS) [9] that use sampling weights to be representative of the United States population. Those surveys rely on interview data rather than clinical diagnoses of Alzheimer’s disease and other conditions. Recent literature focuses on how such studies can be used to extend results from randomized trials to larger, more representative populations [2 , 10–19]. To date there has been only one publication in dementia research that extends estimates from an observational cohort study to its source population (BRFSS California) [1].

Being able to generalize study results to a broader population is necessary to better understand disease prevalence and to ensure conclusions based on risk factors identified in selected samples would be applicable to the broader population [20, 21]. A statistical method of weighting study participants in proportion to their representation in the population can aid results’ generalization. For example, in a Swedish study where true disease prevalence was known, some disease rates were underestimated when weights were not applied to the selected cohort; applying participation weights [1] provided more accurate estimates [22].

Here, we examine the effect of applying participation weights to data from the Adult Changes in Thought (ACT) study to reflect the population of older adults in the Seattle Metropolitan Region (Seattle Metropolitan Region). ACT is an ongoing, population-based cohort in the Seattle, Washington area that is focused on improving understanding of cognitive and brain aging and prevention of Alzheimer’s disease and related dementias. The ACT sample comprises randomly-selected members of Kaiser Permanente Washington (previously Group Health) over the age of 65 residing in the greater Seattle area who consent to the study [23] and are cognitively normal at baseline. We used data from BRFSS to estimate sociodemographic characteristics of the Seattle Metropolitan Region population over the age of 65 and compared those to ACT. We then constructed participation weights [1, 22], which incorporate the probability someone in the Seattle Metropolitan Region would be included in ACT. As an example of how we could use these weights to transport results from our community-based cohort study to the surrounding metropolitan area, we examined the impact of the participation weights on estimates of the prevalence of common eye conditions and on the associations between eye conditions and the risk of Alzheimer’s disease.

MATERIALS AND METHODS

Participants

Detailed ACT study methods have been published [23, 24]. Briefly, the ACT study is an ongoing, population-based, prospective cohort study of consenting older adults randomly selected from Kaiser Permanente Washington membership rolls and followed until dementia development, death or study drop out (88% retention) [23]. The Original Cohort consisted of 2,581 participants recruited from 1994 to 1996, the Expansion Cohort included 811 participants recruited from 2000 to 2003, and the Replacement Cohort comprised 2,371 participants recruited from 2004 to the 2020 data freeze, for a total of 5,762 participants. In the Original Cohort, 48% of those invited to be screened enrolled in the study [23]. To reduce bias associated with ability to attend research clinic visits, ACT participants are able to complete assessments at home [25]. At enrollment and during biennial visits (conducted at the participant’s home or in the research clinic), participants received cognitive screening tests, brief physical evaluations, medical history and risk factor assessments, and provided a blood sample for APOE genotyping [24, 25]. Participants with Cognitive Abilities Screening Instrument [26] scores ≤85 underwent a standardized dementia diagnostic evaluation, including additional physical and neurologic examinations and a neuropsychological test battery [27]. Dementia diagnoses were determined at consensus conferences using Diagnostic and Statistical Manual of Mental Disorders IV criteria [28], and probable and possible AD-type dementia diagnoses were determined using the National Institute of Neurological and Communicative Disorders and Stroke – Alzheimer’s Disease and Related Disorders Association (NINCDS-ADRDA) criteria [29]. This study was approved by the Institutional Review Boards of Kaiser Permanente Washington and the University of Washington and conducted in accordance with the Declaration of Helsinki of 1975.

The BRFSS is a U.S. population-based, nationally representative telephone survey of non-institutionalized adults which collects sociodemographic and health data. We used survey-weighted [30] 2019 data on everyone 65 and older from the Seattle-Bellevue-Everett, WA, Metropolitan Division (Seattle Metropolitan Region). The BRFSS was approved by the CDC Human Research Protection Office and the analyst for this paper was approved for the use of deidentified BRFSS data.

Harmonized measures

Our goal was to align observable characteristics of ACT participants at ACT study enrollment with the 2019 BRFSS. We were able to harmonize 20 self-reported sociodemographic and health characteristic variables assessed in 2019 BRFSS and ACT baseline interviews: age, sex, education, race, Hispanic ethnicity, marital status, employment status, smoking, current hypertension medication use, current cholesterol medication use, history of heart disease, stroke, asthma, chronic obstructive pulmonary disease or emphysema, diabetes, non-melanoma cancer, osteoarthritis, self-perceived general health, and any difficulty dressing or bathing and any difficulty walking or climbing stairs. Age in ACT was converted into the 5-year BRFSS groups. For some variables, we only needed to recode values (e.g., 2 = no for Hispanic in BRFSS, while 0 = no in ACT). For others, we needed to recategorize and sometimes combine questions so that the responses matched (e.g. two BRFSS variables were needed to align with ACT’s question on hypertension medicine use). Specifics for each variable are in Supplementary Table 1.

Eye diseases

Diagnoses of four common eye diseases, diabetic retinopathy, age-related macular degeneration, glaucoma, and cataracts, in ACT participants at baseline were based on ICD-9 and ICD-10 codes from the participants’ electronic medical records, as described in Supplementary Table 2. We calculated the prevalence of each of these diseases at baseline. We also examined the associations of diabetic retinopathy and age-related macular degeneration with Alzheimer’s disease. We considered time-varying windows of the length of time since the eye disease diagnosis, divided into 1) no diagnosis to date, 2) diagnosis within the previous 5 years (“recent”), and 3) diagnosis more than 5 years ago (established disease), based on previous research indicating that associations vary based on the time since diagnosis [31].

Statistical methods

Analyses were conducted in Stata 18.0 [32]. Stata code is available in https://github.com/gibbonsl/ACT_transport.

All harmonized variables had less than 5% missingness in BRFSS. In ACT, self-reported asthma, chronic obstructive pulmonary disease or emphysema, and osteoarthritis each had 11% missingness, largely because the related questions were dropped in 2016. Nine percent were missing self-reported diabetes, and the rest had less than 5% missing (Supplementary Table 3). Even though the data were relatively complete, cumulatively 12% of BRFSS and 15% of ACT participants were missing at least one variable, so we used multiple imputation with chained equations, by cohort, before computing weights [33] (40 imputations).

To estimate and evaluate participation weights, we pooled the 2019 BRFSS and ACT baseline datasets. We calculated stabilized inverse odds of participation weights for all ACT participants. These participation weights were derived in an iterative process [1] in which we 1) estimated the probability of being in ACT, 2) calculated participation weights, and 3) assessed the resulting covariate balance.

We estimated the probability of being in ACT for each individual in the Seattle Metropolitan Region using a logistic regression model. We used all the variables in Table 1 except for cancer history, which was not needed. Next, we calculated participation weights (i.e., inverse probability weights) for ACT participants as the inverse odds of being in ACT, given the covariates in the logistic regression [11]. We stabilized the weights by multiplying the inverse odds of being in ACT by the unconditional odds of being in ACT. We Winsorized the weights by recoding the top 1% to the 99th percentile, and the bottom 1% to the first percentile. Further details are in Supplementary Methods 1.

Table 1

Characteristics of BRFSS and ACT samples by aligned variables

	BRFSS (with survey weights)^a		ACT
Characteristic^b	n = 431701.1	%	n = 5762	%	p ^c
Age					0.4259
65–69	137,618.2	32%	1,665	29%
70–74	125,939.0	29%	1,743	30%
75–79	83,525.4	19%	1,162	20%
80–99	84,618.6	20%	1,192	21%
Sex					0.1005
Male	194,716.1	45%	2,420	42%
Female	236,985.0	55%	3,342	58%
Education					<0.0001
Less than HS	8,064.8	2%	481	8%
HS degree	235,533.0	55%	2,443	42%
College or higher	185,809.9	43%	2,837	49%
Unknown	2,293.4	1%	1	0%
Race					0.0003
White	361,152.8	84%	5,146	89%
Black	10,588.5	2%	207	4%
Other^d	55,059.8	13%	401	7%
Unknown	4,900.0	1%	8	0%
Hispanic					0.0010
No	417,084.4	97%	5,683	99%
Yes	11,931.4	3%	70	1%
Unknown	2,685.3	1%	9	0%
Marital status					0.6173
Never Married	23,585.0	5%	255	4%
Married or Living together	242,857.7	56%	3,234	56%
Separated, Divorced, Other	74,518.4	17%	996	17%
Widowed	90,740.1	21%	1,277	22%
Employment status					0.0003
Retired	311,351.7	72%	4,389	76%
Employed	86,542.5	20%	878	15%
Other	25,926.5	6%	253	4%
Unknown	7,880.5	2%	242	4%
Smoker					0.0077
Never	218,960.9	51%	2,816	49%
Past	167,780.8	39%	2,658	46%
Current	25,696.9	6%	274	5%
Unknown	19,262.5	4%	14	0%
Current hypertension medication					<0.0001
No	219,195.4	51%	3,391	59%
Yes	210,664.1	49%	2,342	41%
Unknown	1,841.6	0%	29	1%
Current cholesterol medication					<0.0001
No	265,412.0	61%	4,511	78%
Yes	155,713.9	36%	1,225	21%
Unknown	10,575.2	2%	26	0%
Any heart disease history					<0.0001
No	390,452.3	90%	4,766	83%
Yes	38,565.3	9%	952	17%
Unknown	2,683.6	1%	44	1%
Stroke history					<0.0001
No	406,011.6	94%	5,582	97%
Yes	24,621.1	6%	164	3%
Unknown	1,068.4	0%	16	0%
Asthma history					0.0032
No	368,103.9	85%	4,585	80%
Yes	62,443.8	14%	562	10%
Unknown	1,153.5	0%	615	11%
Chronic obstructive pulmonary disease or emphysema history					0.3911
No	391,080.7	91%	4,634	80%
Yes	37,890.5	9%	504	9%
Unknown	2,729.9	1%	624	11%
Diabetes history					<0.0001
No	359,826.4	83%	4,654	81%
Yes	71,586.6	17%	582	10%
Unknown	288.0	0%	526	9%
Non-melanoma cancer history					0.4668
No	346,427.5	80%	4,687	81%
Yes	84,043.5	19%	1,057	18%
Unknown	1,230.2	0%	18	0%
Osteoarthritis history					0.0004
No	222,767.1	52%	2,991	52%
Yes	208,665.5	48%	2,126	37%
Unknown	268.5	0%	645	11%
Self-perceived general health					0.0174
Fair – Poor	74,657.8	17%	807	14%
Good – Excellent	356,397.4	83%	4,943	86%
Unknown	645.9	0%	12	0%
Any difficulty dressing or bathing					0.0082
No	397,142.6	92%	5,289	92%
Yes	20,831.7	5%	447	8%
Unknown	13,726.9	3%	26	0%
Any difficulty walking or climbing stairs					< 0.0001
No	334,564.2	77%	4,123	72%
Yes	82,389.0	19%	1,604	28%
Unknown	14,747.9	3%	35	1%

^aSurvey weights from the CDC [30] were applied to a sample of 1,158 BRFSS respondents to generalize to the Seattle metropolitan area. ^bAll variables are self-report. ^cChi-squared p-value for non-missing values comparing ACT and BRFSS, using BRFSS survey weights. ^dHispanic, Asian, American Indian, Alaskan Native, Native Hawaiian, Pacific Islander, Mixed Race, or “Other”.

We assessed covariate balance between ACT and survey-weighted BRFSS participants by calculating standardized mean differences as the (mean_ACT – mean_BRFSS)/SD_BRFSS [34]. For each variable in Table 1, we compared the standardized mean differences with and without ACT participation weights. Values within ±0.25 have been said to represent adequate covariate balance [34], but we used a more stringent threshold of ±0.10 [35].

Next, we used the weights to obtain eye disease estimates for the Seattle Metropolitan Region, the target population represented by the BRFSS. We applied the participation weights to ACT data to extend estimates of the prevalence of the four common eye diseases from ACT study baseline to all adults over age 65 in the Seattle Metropolitan Region in 2019. We also examined the expected associations of diabetic retinopathy and age-related macular degeneration with probable or possible Alzheimer’s disease incidence for adults over age 65 in the Seattle Metropolitan Region in 2019, assuming we could follow them into the future. We used separate Cox models with a time-dependent variable for eye disease status (ever/never), adjusting for baseline age, sex, years of education, and smoking history (past/current/never). Age was the time axis. To meet proportional hazards assumptions, we allowed the baseline hazards to differ by sex and by the presence of any APOE ɛ4 alleles. We calculated hazard ratios (HR) and 95% confidence intervals (CI) for each model in both unweighted ACT data and in ACT data with participation weights applied.

To account for uncertainty due to use of weights and Rubin’s rules for combining variance across imputations, we calculated confidence intervals using bootstrapping [36, 37]. We resampled from a combined ACT and BRFSS sample 500 times from each of the 40 imputed ACT datasets and re-estimated the participation weights. We then applied these weights to estimates of the prevalence of eye diseases and the association of eye disease with Alzheimer’s disease in the ACT sample; we defined confidence intervals as the 2.5th and 97.5th percentiles of the 20,000 total bootstrapped estimates [36]. Sensitivity analyses were not bootstrapped.

As sensitivity analyses, we calculated eye disease prevalence and Alzheimer’s disease risks excluding ACT participants born before 1920 and thus not eligible to be in BRFSS 2019, which only reports ages under 100. We used weights derived in that subsample. We also conducted the Alzheimer’s disease analyses using weights which omitted “Any difficulty dressing or bathing” and “Any difficulty walking or climbing stairs” because these activities of daily living (ADLs) could be on the causal pathway from eye disease to Alzheimer’s disease.

RESULTS

Differences in demographics and health without participation weights

The 2019 BRFSS included 1158 people in the Seattle Metropolitan Region. Each participant has a survey weight to make the 1158 represent over 400,000 adults over age 65 in the Seattle Metropolitan Region [30]. We compared the frequency of aligned variables for the 5762 ACT participants at baseline with the weighted BRFSS sample (Table 1). Our main interest was in the magnitude of the standardized mean differences. Only two standardized mean differences exceeded ±0.25: self-reported current cholesterol medication (reported by a higher proportion of BRFSS participants) and education level (higher proportions of ACT participants reported less than high school education) (Fig. 1). In addition, standardized mean differences were less than ±0.25 but greater than ±0.10 for current hypertension medication, diabetes, and osteoarthritis, which were all higher in BRFSS, and for histories of heart disease and difficulty walking or climbing stairs, which were higher in ACT.

Fig. 1

Covariate balance between ACT and the survey-weighted BRFSS. Standardized mean differences were calculated as the mean in ACT minus the mean in BRFSS, then divided by the SD in BRFSS, so that a positive value means greater frequency in ACT. For variables with more than 2 levels, each level is shown compared to all others. Ideally all values are between the dotted lines at –0.10 and 0.10. Accessibility description: Using survey weights only, two of the 20 demographic and health characteristics have standardized mean differences with magnitude greater than 0.25. Self-reported current cholesterol medication was reported by a higher proportion of BRFSS participants, while more ACT participants reported less than high school education. Using the participation weights, all 20 characteristics are between –0.10 and 0.10.

Differences in demographics and health with participation weights

After developing participation weights, we compared covariate balance between participation-weighted ACT and survey-weighted BRFSS data (Fig. 1). Using the participation weights in ACT yielded good balance on demographic and health characteristics relative to survey-weighted BRFSS. The magnitudes of the differences were all within or on our desired 0.10 threshold.

Transporting eye disease findings to the Seattle Metropolitan Region

The main reason to compute the participation weights is to transport results that can only be obtained in ACT to all older adults in the Seattle Metropolitan Region. To illustrate this, we looked at the prevalence of four common eye conditions at ACT study baseline that were not measured in BRFSS. When we used the participation weights to transport ACT data to the Seattle Metropolitan Region, the prevalences of age-related macular degeneration and glaucoma were quite similar (Table 2). The estimated proportions of people with a history of cataracts and with diabetic retinopathy were more common in results weighted to represent Seattle Metropolitan Region.

Table 2

Eye condition prevalence estimates (95% confidence intervals) within the ACT study and transported to the Seattle Metropolitan Region

Eye Disease	Prevalence (95% confidence interval)
Diabetic retinopathy
ACT study (unweighted)	4.1% (3.6, 4.6)
Transported to the Seattle Metropolitan Region (weighted^b)	5.6% (4.3, 7.1)
Age-related macular degeneration
ACT study (unweighted)	9.5% (8.8, 10.3)
Transported to the Seattle Metropolitan Region (weighted^b)	9.7% (8.6, 11.0)
Glaucoma
ACT study (unweighted)	9.2% (8.5, 10.0)
Transported to the Seattle Metropolitan Region (weighted^a)	8.9% (7.9, 10.0)
Cataract history
ACT study (unweighted)	48.6% (47.3, 49.9)
Transported to the Seattle Metropolitan Region (weighted^a)	51.8% (49.6, 54.3)

^aConfidence intervals for the participation-weighted estimates were obtained by bootstrapping in the 40 imputed datasets. Incorporating Rubin’s rules and the error in estimating the weights results in wider confidence intervals.

In the sensitivity analysis omitting ACT participants born before 1920 (Supplementary Table 4), participation-weighted prevalence estimates were similar to the unweighted prevalence estimates for diabetic retinopathy and glaucoma. The estimated proportions were higher using the participation weights for age-related macular degeneration and for a history of cataracts.

We used the participation weights to extend estimates of association of eye disease with Alzheimer’s disease incidence from ACT to the Seattle Metropolitan Region. We evaluated time-dependent eye disease status as a risk factor for Alzheimer’s disease using both unweighted and participation-weighted ACT data. Compared to people with no diagnosis of diabetic retinopathy, people with a recent diagnosis of diabetic retinopathy had a HR of 1.32 without weights and 1.84 with weights (Table 3).

Table 3

Hazard ratios (HR) and 95% confidence intervals (CI) for Alzheimer’s disease in four separate Cox models for time-dependent eye disease status within the ACT study and transported to the Seattle Metropolitan Region

Model^a	Years with diagnosis	ACT Study (unweighted)	Transported to the Seattle Metropolitan Region (weighted^b)
Diabetic retinopathy	>0–5	1.32 (0.87, 2.00)	1.84 (0.34, 4.29)
	>5	1.39 (1.02, 1.88)	1.29 (0.47, 2.46)
Age-related macular degeneration	>0–5	0.74 (0.59, 0.94)	0.63 (0.30, 1.03)
	>5	1.44 (1.22, 1.70)	1.35 (0.81, 1.81)

^aAll models were adjusted for baseline age, years of education, and smoking history. Baseline hazards were allowed to differ by sex and APOE genotype (≥1 ɛ4 alleles). Age was the time axis. ^bConfidence intervals for the participation-weighted estimates were obtained by bootstrapping in the 40 imputed datasets. Incorporating Rubin’s rules and the error in estimating the weights results in wider confidence intervals.

Note that bootstrapping to incorporate the error estimating the weights and Rubin’s rules for combining variance across imputations results in much larger confidence intervals. See Supplementary Table 5 to compare the intervals in Table 3 to those that do not incorporate the error in estimating the weights. For example, in Table 3, the bootstrapped confidence interval for recent diagnosis of diabetic retinopathy is (0.34, 4.29), compared to (1.08, 3.14) in the weighted analysis without bootstrapping in Supplementary Table 5. For diagnosis more than 5 years ago, the interval is (0.47, 2.46) with bootstrapping and (0.87, 1.91) without.

Leaving out ACT participants born before 1920 yielded fairly similar results with and without weighting (Supplementary Table 6). Compared to models using the weights that included the ADLs, omitting the ADLs from the weights gave similar results for both Alzheimer’s disease models (Supplementary Table 5).

DISCUSSION

The ACT study is community-based, drawing participants from an integrated health-care delivery system. On variables measured in both groups, differences between those in ACT and the general population of older adults in the region (BRFSS) were generally small in magnitude. Eighteen of the 20 aligned characteristics had a standardized mean difference within ±0.25, which has been cited as adequate covariate balance [34], and more than half were within the most stringent threshold of ±0.10 [35] before any participation weights were applied. Thus the ACT sample is fairly representative of the broader Seattle population of older adults. This is consistent with papers that reported that Kaiser Permanente members of all ages are fairly similar to the general population in their general region [38, 39].

Still, differences between ACT study volunteers and older adults in the Seattle Metropolitan Region might affect the generalizability of some research findings. Indeed, when we used the participation weights in ACT to generalize our eye-disease findings to the Seattle Metropolitan Region, some estimates differed noticeably, and others did not, compared to the un-weighted ACT results. Like the Swedish study [22], our results varied by condition. Baseline estimates for the prevalence of age-related macular degeneration and glaucoma were similar without and with weights, while the estimates for diabetic retinopathy and for a history of cataracts was higher, suggesting that the ACT study underestimated the true prevalence of this condition for the Seattle Metropolitan Region.

We evaluated time-dependent eye disease status as a risk factor for Alzheimer’s disease, divided into recent and longer duration of diagnosis, based on previous research [31]. When we applied participation weights, we observed a higher estimated effect of recent diabetic retinopathy diagnosis. This may reflect potential differences in the severity of diabetic retinopathy when participants are initially diagnosed. It is possible that the ACT study participants are diagnosed with diabetic retinopathy sooner due to better access to care compared to the Seattle Metropolitan Region population. When we apply the weights, we may be upweighting people with the more severe instances of recent diabetes retinopathy diagnosis, resulting in a larger effect on Alzheimer’s disease. With the participation weights, recent age-related macular degeneration was more protective.

The association between recently diagnosed age-related macular degeneration and a lower risk of Alzheimer’s disease may seem counterintuitive, but the relationship between age-related macular degeneration and Alzheimer’s disease is rather complex and not well understood. For instance, APOE ɛ2 is a risk factor for age-related macular degeneration [40] but protective for AD, while relationship is the reverse for APOE ɛ4, suggesting that the relationship between APOE, age-related macular degeneration, and Alzheimer’s disease needs further research. Both the diabetic retinopathy and the age-related macular degeneration examples suggest that estimates within the ACT study cannot be directly extended to the broader population, though the bootstrapped confidence intervals were wide.

We were very successful in balancing variables measured in both ACT and BRFSS, meeting the ±0.10 threshold for all characteristics. This provides some confidence in the validity of using participation-weighted ACT data to obtain estimates for the Seattle Metropolitan Region population 65+. However, we could not account for any characteristics not measured in both ACT and BRFSS, and there are potential differences between the two samples that could not be reconciled with available data. For example, ACT participants do not have dementia at baseline, and it is possible that some BRFSS participants had dementia; however, this concern is mitigated by the fact that they were all cognitively intact enough to complete the survey. Perhaps the biggest issue in generalizing health findings using participation-weighted ACT data is that everyone in ACT has health insurance, and despite being over 65, BRFSS participants may not have Medicare or their health care may be of lesser quality. Additionally, at the time of ACT enrollment, participants had been enrolled Kaiser Permanente (or Group Health) members for a median of 24.8 years. We do not have information on insurance history for BRFSS participants, thus we cannot account for differences related to duration of quality healthcare. Finally, since the 2019 BRFSS data is limited to people 99 years old or younger, individuals born before 1920 are not represented in this cohort but are included in the ACT cohort. We did a sensitivity analysis to address this difference.

Another key challenge is the potential role of secular trends affecting our outcomes of interest. ACT baseline data go back to 1994, while the BRFSS sample used in this study was obtained in 2019. There are many potentially relevant changes in medical care. Many treatments for eye diseases available now did not exist in the 1990 s, so a 70-year-old in 1994 had different treatment options than a 70-year-old in 2020. Our approach had to assume that birth cohort or secular trends (e.g., changes in medical care, education, or health behaviors such as smoking) do not modify true prevalence, likelihood of disease detection, or the effect of eye disease on Alzheimer’s disease. This assumption may not hold here, or for other diseases or their associations with Alzheimer’s disease. It is somewhat reassuring that omitting those born before 1920 (the oldest birth cohort in ACT) yielded fairly similar results.

Health research should aim to be inclusive of and applicable to diverse populations. ACT was designed to include a random selection of older Kaiser Permanente Washington members. This paper focused on how to transport findings to the population of the Seattle Metropolitan Region, generalizing beyond Kaiser Permanente Washington members. The ACT cohort has historically included predominantly white adults, reflecting the Seattle area, limiting our ability to generalize to more diverse populations. Current ACT recruitment is emphasizing increasing the racial, ethnic and socioeconomic diversity of study participants, which will help us to generalize beyond the Seattle region to the broader USA population in the future.

Should all research studies transport their findings to a target population? When is it necessary? Certainly if one is trying to describe or draw inferences about a target population for whom the research study is non-representative, weighting would be needed. Our results on eye disease prevalence suggest that even small differences in representativeness between the target population and study sample may affect estimates and thus would benefit from weighting. What about causal questions like the relationship between eye disease and Alzheimer’s disease? If we suspect meaningful differences between the study sample on variables that affect the outcome, such as confounders and moderators, we believe participation weights should be employed. Future studies should gather data in a way that aligns with variables collected in population-representative surveys so that better participation weights can be constructed. As more research is transported, we will have a better sense of how often and for what kind of questions this matters.

The goals of this paper were to compare ACT participants’ demographic and health characteristics to the Seattle Metropolitan Region and to evaluate the impact of participation weights on estimates of the prevalence of common eye conditions and on the associations between eye conditions and the risk of Alzheimer’s disease. Given ACT’s extensive publication history, it is comforting to find that even without participation weights, ACT characteristics were fairly similar to the Seattle Metropolitan Region. Still, even in a study as community-based as ACT, generalizing research on older adults to the broader population may benefit from evaluation with the use of statistical approaches to ensure validity of estimates.

AUTHOR CONTRIBUTIONS

Laura E. Gibbons (Conceptualization; Formal analysis; Methodology; Writing – original draft); Taylor Mobley (Methodology; Writing – review & editing); Elizabeth Rose Mayeda (Methodology; Writing – review & editing); Cecilia S. Lee (Writing – review & editing); Nicole M. Gatto (Writing – review & editing); Andrea Z. LaCroix (Funding acquisition; Writing – review & editing); Linda K. McEvoy (Funding acquisition; Writing – review & editing); Paul K. Crane (Funding acquisition; Writing – review & editing); Eleanor Hayes-Larson (Methodology; Writing – review & editing).

Footnotes

ACKNOWLEDGMENTS

We thank the participants of the Adult Changes in Thought (ACT) study for the data they have provided and the many ACT investigators and staff who steward that data. You can learn more about ACT at: . Thanks also to Connie Nakano for help with manuscript preparation and to R. Elizabeth Sanders for figure editing.

FUNDING

This work was supported by the National Institutes of Health [grant numbers U19AG066567, U01AG006781, R56AG069126, P50AG005136, R01AG060942, NIA K99AG075317, 5R13AG030995]. The NIH had no involvement in the study design, in the collection, analysis and interpretation of data, in the writing of the report, and in the decision to submit the article for publication.

CONFLICT OF INTEREST

Dr. Lee reports consulting fees from Boehringer Ingelheim International, outside of the submitted work. Other authors have no declarations of interest.

DATA AVAILABILITY

Data cannot be made publicly available for ethical and legal reasons. In order to replicate our findings, a researcher must have access to personal health identifiers (PHI) including dates of birth and death, dates of diagnoses, and ages over 89. These are required variables for the analyses, and we cannot publicly release this information without IRB approval and a Data Use Agreement with interested researchers. However, external researchers can request these data with proper IRB and HIPAA approvals. If a researcher is interested in requesting data, the process is available on the Adult Changes in Thought (ACT) website: actagingresearch.org.

The supplementary material is available in the electronic version of this article: .

References

Hayes-Larson

, Mobley

, Mungas

, Seamans

, Glymour

, Gilsanz

, DeCarli

, Whitmer

, Mayeda

(2022) Accounting for lack of representation in dementia research: Generalizing KHANDLE study findings on the prevalence of cognitive impairment to the California older population. Alzheimers Dement 18, 2209–2217.

Gianattasio

, Bennett

, Wei

, Mehrotra

, Mosley

, Gottesman

, Wong

, Stuart

, Griswold

, Couper

, Glymour

, Power

(2021) Generalizability of findings from a clinical sample to a community-based sample: A comparison of ADNI and ARIC. Alzheimers Dement 17, 1265–1276.

Ganguli

, Lee

, Hughes

, Snitz

, Jakubcak

, Duara

, Chang

(2015) Who wants a free brain scan? Assessing and correcting for recruitment biases in a population-based sMRI pilot study. Brain Imaging Behav 9, 204–212.

Brewster

, Barnes

, Haan

, Johnson

, Manly

, Napoles

, Whitmer

, Carvajal-Carmona

, Early

, Farias

, Mayeda

, Melrose

, Meyer

, Al Hazzouri

, Hinton

, Mungas

(2019) Progress and future challenges in aging and diversity research in the United States. Alzheimers Dement 15, 995–1003.

National Institute of Aging, Aging Well in the 21st Century: Strategic Directions for Research on Aging https://www.nia.nih.gov/sites/default/files/2017-07/nia-strategic-directions-2016.pdf, Accessed February 20, 2024.

National Institute of Neurological Disorders and Stroke, ADRD Summit 2019 Report to the National Advisory Neurological Disorders and Stroke Council, https://www.ninds.nih.gov/sites/default/files/migrate-documents/2019_adrd_summit_recommendations_508c.pdf, Accessed February 20, 2024.

Batty

, Gale

, Kivimäki

, Deary

, Bell

(2020) Comparison of risk factor associations in UK Biobank against representative, general population based studies with conventional response rates: Prospective cohort study and individual participant meta-analysis. BMJ 368, m131.

National Institute of Aging, Growing Older in America: The Health and Retirement Study,National Institute on Aging, National Institutes of Health, https://www.nia.nih.gov/sites/default/files/2017-06/health_and_retirement_study_0.pdf, Accessed February 20, 2024.

Centers for Disease Control and Prevention (CDC), Behavioral Risk Factor Surveillance System Survey Data (BRFSS), https://www.cdc.gov/brfss/, Accessed February 20, 2024.

10.

Lesko

, Buchanan

, Westreich

, Edwards

, Hudgens

, Cole

(2017) Generalizing study results: A potential outcomes perspective. Epidemiology 28, 553–561.

11.

Westreich

, Edwards

, Lesko

, Stuart

, Cole

(2017) Transportability of trial results using inverse odds of sampling weights. Am J Epidemiol 186, 1010–1014.

12.

Dahabreh

, Haneuse

SJA

, Robins

, Robertson

, Buchanan

, Stuart

, Hernán

(2021) Study designs for extending causal inferences from a randomized trial to a target population. Am J Epidemiol 190, 1632–1642.

13.

Ackerman

, Lesko

, Siddique

, Susukida

, Stuart

(2021) Generalizing randomized trial findings to a target population using complex survey population data. Stat Med 40, 1101–1120.

14.

Lesko

, Ackerman

, Webster-Clark

, Edwards

(2020) Target validity: Bringing treatment of external validity in line with internal validity. Curr Epidemiol Rep 7, 117–124.

15.

Robertson

, Steingrimsson

, Dahabreh

(2023) Regression-based estimation of heterogeneous treatment effects when extending inferences from a randomized trial to a target population. Eur J Epidemiol 38, 123–133.

16.

Zivich

, Edwards

, Lofgren

, Cole

, Shook-Sa

, Lessler

(2024) Transportability without positivity: A synthesis of statistical and simulation modeling. Epidemiology 35, 23–31.

17.

Dahabreh

, Robins

, Haneuse

SJA

, Saeed

, Robertson

, Stuart

, Hernán

(2023) Sensitivity analysis using bias functions for studies extending inferences from a randomized trial to a target population. Stat Med 42, 2029–2043.

18.

Aslanyan

, Pa

, Hodis

, St John

, Kono

, Henderson

, Mack

(2023) Generalizability of cognitive results from clinical trial participants to an older adult population: Addressing external validity. Alzheimers Dement (Amst) 15, e12417.

19.

Dahabreh

, Matthews

, Steingrimsson

, Scharfstein

, Stuart

(2023) Using trial and observational data to assess effectiveness: Trial emulation, transportability, benchmarking, and joint analysis. Epidemiol Rev, doi: 10.1093/epirev/mxac011.

20.

Mayeda

, Hayes-Larson

(2023) Who is represented in neurology data sources? A call for rigorous evaluation of generalizability of neurology research. Neurology 101, 463–464.

21.

Ling

, Montez-Rath

, Carita

, Chandross

, Lucats

, Meng

, Sebastien

, Kapphahn

, Desai

(2023) An overview of current methods for real-world applications to generalize or transport clinical trial findings to target populations of interest. Epidemiology 34, 627–636.

22.

Bonander

, Nilsson

, Björk

, Bergström

GML

, Strömberg

(2019) Participation weighting based on sociodemographic register data improved external validity in a population-based cohort study. J Clin Epidemiol 108, 54–63.

23.

Kukull

, Higdon

, Bowen

, McCormick

, Teri

, Schellenberg

, van Belle

, Jolley

, Larson

(2002) Dementia and Alzheimer disease incidence: A prospective cohort study. Arch Neurol 59, 1737–1746.

24.

Larson

, Wang

, Bowen

, McCormick

, Teri

, Crane

, Kukull

(2006) Exercise is associated with reduced risk for incident dementia among persons 65 years of age and older. Ann Intern Med 144, 73–81.

25.

Crane

, Gibbons

, McCurry

, McCormick

, Bowen

, Sonnen

, Keene

, Grabowski

, Montine

, Larson

(2016) Importance of home study visit capacity in dementia studies. Alzheimers Dement 12, 419–426.

26.

Teng

, Hasegawa

, Homma

, Imai

, Larson

, Graves

, Sugimoto

, Yamaguchi

, Sasaki

, Chiu

, et al. (1994) The Cognitive Abilities Screening Instrument (CASI): A practical test for cross-cultural epidemiological studies of dementia. Int Psychogeriatr 6, 45–58; discussion 62.

27.

Crane

, Trittschuh

, Mukherjee

, Saykin

, Sanders

, Larson

, McCurry

, McCormick

, Bowen

, Grabowski

, Moore

, Bauman

, Gross

, Keene

, Bird

, Gibbons

, Mez

(2017) Incidence of cognitively defined late-onset Alzheimer’s dementia subgroups from a prospective cohort study. Alzheimers Dement 13, 1307–1316.

28.

(1994) Diagnostic and statistical manual of mental disorders. 4th ed, American Psychiatric Association (APA), Washington, DC.

29.

McKhann

, Knopman

, Chertkow

, Hyman

, Jack

Jr. , Kawas

, Klunk

, Koroshetz

, Manly

, Mayeux

, Mohs

, Morris

, Rossor

, Scheltens

, Carrillo

, Thies

, Weintraub

, Phelps

(2011) The diagnosis of dementia due to Alzheimer’s disease: Recommendations from the National Institute on Aging-Alzheimer’s Association workgroups on diagnostic guidelines for Alzheimer’s disease. Alzheimers Dement 7, 263–269.

30.

U.S. Centers for Disease Control and Prevention (CDC), Weighting the BRFSS Data, CDC, https://www.cdc.gov/brfss/annual_data/2019/pdf/weighting-2019-508.pdf, Accessed February 20, 2024.

31.

Lee

, Larson

, Gibbons

, Lee

, McCurry

, Bowen

, McCormick

, Crane

(2019) Associations between recent and established ophthalmic conditions and risk of Alzheimer’s disease. Alzheimers Dement 15, 34–41.

32.

StataCorp (2023) Stata Statistical Software: Release 18, StataCorp LLC, College Station, TX.

33.

Ross

, Breskin

, Breger

, Westreich

(2022) Reflection on modern methods: Combining weights for confounding and missing data. Int J Epidemiol 51, 679–684.

34.

Stuart

(2010) Matching methods for causal inference: A review and a look forward. Stat Sci 25, 1–21.

35.

Austin

, Stuart

(2015) Moving towards best practice when using inverse probability of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects in observational studies. Stat Med 34, 3661–3679.

36.

Schomaker

, Heumann

(2018) Bootstrap inference when using multiple imputation. Stat Med 37, 2252–2266.

37.

Lesko

, Cole

, Hall

, Westreich

, Miller

, Eron

, Li

, Mugavero

(2016) The effect of antiretroviral therapy on all-cause mortality, generalized to persons diagnosed with HIV in the USA, 2009-11. Int J Epidemiol 45, 140–150.

38.

Davis

, Voelkel

, Remmers

, Adams

, McGlynn

(2023) Comparing Kaiser Permanente members to the general population: Implications for generalizability of research. Perm J 27, 87–98.

39.

Gordon

, Similarity of the Adult Kaiser Permanente Membership in Northern California to the Insured and General Population in Northern California: Statistics from the 2011 California Health Interview Survey, https://memberhealthsurvey.kaiser.org/Documents/chis_non_kp_2011.pdf, Accessed February 20, 2024.

40.

Rasmussen

, Tybjærg-Hansen

, Nordestgaard

, Frikke-Schmidt

(2023) Associations of Alzheimer disease-protective APOE variants with age-related macular degeneration. JAMA Ophthalmol 141, 13–21.