Improving Regression Analysis with Imputation in a Longitudinal Study of Alzheimer’s Disease

Study data

The data used for the statistical analyses in this article were from ADNIMERGE.csv, FAMXHPAR.csv, and FHQ.csv, all downloaded from the ADNI database on October 28, 2021. The ADNI was launched in 2003 as a public-private partnership, led by Principal Investigator Michael W. Weiner, MD. The primary goal of ADNI has been to test whether serial magnetic resonance imaging, positron emission tomography, other biological markers, and clinical and neuropsychological assessment can be combined to measure the progression of MCI and early AD. For up-to-date information, see https://adni.loni.usc.edu/.

We studied 1,063 subjects diagnosed with MCI. Study protocols describe follow-ups every 6 months from the initial visit until month 24. After month 24, subjects are followed up once every 12 months. Cognitive, imaging, genetic, and biochemical data are collected during each visit.

Before performing any analyses, levels of CSF biomarkers were clipped at their technical limits as in [22–24]. For example, Aβ₄₂ levels could only be detected within the range 200 pg/ml to 1700 pg/ml. For subjects with levels outside of this range, those levels were reported as <200 or >1700. We set such values to their respective limits of 200 and 1700. Analogously, we clipped pTau and tTau levels at their respective technical limits.

FAMXHPAR.csv and FHQ.csv were used to obtain information about patient-reported family history of AD. Family history was said to be present if the patient reported either of their parents having AD. If the patient reported that neither parent had AD, family history was said to be absent. Any other response (unsure or no entry) was considered a missing value to be imputed.

Statistical analysis

We are interested in the ability of baseline covariates of MCI patients to predict their average ADAS-Cog 13 score across visits. To explore this, we developed five linear mixed-effects models— each involving one of the CSF biomarkers— based on scientific and statistical considerations. We refer to these models as the analysis models. Biomarker ratios were log-transformed throughout the analyses. We check all five of the models’ underlying normality assumptions by graphical assessment of the distribution of estimated random intercepts. Then, the analysis models are fitted under two cases: with and without performing imputation beforehand. The results from these two different cases are then compared.

In order to fit the analysis models without imputation, we remove all subjects with any missing baseline measurements. We refer to this procedure as an ACA. We argue later in this section that the ACA is expected to produce valid inference— that is, that the fixed-effects and standard error estimates are all expected to be appropriate estimates of the true population quantities. However, since we are discarding potentially useful information, we may still suffer loss of statistical efficiency. As we expect both the ACA and imputation analysis to produce valid fixed-effects estimates, we can compare their statistical efficiencies based on estimated confidence intervals. A narrower confidence interval is indicative of a more efficient procedure.

Next, we fit the analysis models after imputing the missing data. We refer to this procedure as the imputation analysis, which we would like to compare with the ACA. The first step in designing an imputation procedure is to choose an imputation approach. We considered multilevel imputation with PAN to account for within-subject correlations. However, simulation studies suggest that imputation with PAN can be problematic under MAR data when missingness occurs in both the outcome and the predictors (Zhang P & Xie SX, unpublished manuscript). As a result, we decided to perform single-level imputation of our data with MICE, an iterative regression-based approach commonly used for MAR data.

For each of the variables that we will impute (we will refer to these as target variables), we must specify a model which MICE will use to generate imputations. We design these so-called imputation models for ADAS-Cog 13, each of the five CSF biomarkers, and patient-reported family history of AD, since these are the variables with the most substantial missingness, and consequently for which proper imputation is most crucial for producing valid results (see Tables 1 and 2 for information about missingness for each variable). The design of the imputation models is driven by three primary factors [3]: first, every predictor in the analysis model should be included in the imputation models. Second, any variable that predicts missingness of a target variable should be included in its imputation model. These variables can be identified by univariate tests of association. Third, variables that can explain considerable variance in the target variable should be included in its imputation model.

We performed MICE for each of the five analysis models separately. We considered 50 imputations based on recommendations from [25] with 40 maximum iterations of the MICE algorithm based on the convergence plots in Fig. 1. Convergence is achieved when the lines, each tracking one of 50 imputations, stabilize around some mean and standard deviation region, which should be comparable to the means and standard deviations of the observed data set. Furthermore, the lines within a single graph should be intertwined, not showing any obvious trend. The results of the 50 imputations are pooled using Rubin’s Rules [3].

OPEN IN VIEWER

Fig. 1

Mean (left) and standard deviation (right) of imputed values for different variables across iterations for the Aβ₄₂ model. Each line (represented by a different color) is one of 50 imputations. Each of the three rows is for a different variable being imputed. Convergence is achieved when the lines stabilize at some mean and standard deviation region, and so long as there is no observable trend between the 50 imputation lines.

Numerical data were imputed by Bayesian linear regression, and categorical data by logistic regression. We also considered imputation by predictive mean matching (pmm) for numerical data to assess robustness to different imputation procedures. Because the original CSF measurements were clipped, we clipped imputed CSF values at their technical limits after imputation. Results are shown for Bayesian linear regression imputation after clipping, although we verified that the results did not change significantly whether Bayesian linear regression or pmm was used, and whether or not we performed clipping.

We imputed the time-varying outcome, ADAS-Cog 13 scores, only at times for which the proportion of data missing was below a specified tolerance, since imputations become increasingly dependent on the correctness of the chosen imputation model when there is a larger percentage of missingness [4]. We chose tolerances of 50% (wherein we tolerate times for which up to 50% of ADAS-Cog 13 scores are missing), 65%, and 75%, and compared results across the three tolerances.

In the following subsections, we develop the analysis and imputation models we will use for this study.

Analysis model

The analysis model is motivated by the scientific objective. We are interested in the ability of baseline characteristics of MCI patients to predict their average ADAS-Cog 13 across visits. This motivates a linear mixed-effects model as our analysis model, with longitudinal ADAS-Cog 13 as our outcome and time (in months), sex, age, years of education, self-reported family history of AD, APOE ɛ4 allele presence, baseline CDR-SB score, and baseline CSF level as predictors. Since there are five CSF biomarkers, we really have five separate analysis models, each using a different biomarker as a predictor.

The mixed-effects model assumes a normal distribution on the random intercept. To assess adherence to this assumption, we plotted the histograms of the random intercepts for each of the five models when fitted with the available cases. The plots are shown in Fig. 2 and they appear to fit a normal distribution well. It is important to note, however, that normality of the random intercept is a tricky assumption to verify, particularly due to the intangible nature of unobservable quantities like random effects [26]. Therefore, while it is still important to check these assumptions to detect egregious violations [27], the reader should keep in mind the limitations of such tests. It is encouraging, however, that fixed-effects and variance estimates for linear mixed-effects models seem reasonably robust to deviations from the normality assumption (though the same cannot be said of nonlinear models like the generalized linear mixed models) [26–29]. Altogether, these findings suggest that our model is still reasonable.

OPEN IN VIEWER

Fig. 2

Distributions of random intercepts from the ACA for each of the five models: Aβ₄₂ (top left), pTau (top right), tTau (middle left), log-transformed pTau/Aβ₄₂ (middle right), and log-transformed tTau/Aβ₄₂ (bottom).

To be able to compare efficiencies of the ACA and the imputation analysis, we must establish validity of the ACA for these analysis models. In regression settings, including longitudinal regression, ACA on MAR data can still produce valid inference under certain conditions on the analysis model. Namely, if missingness of the covariates is independent of observed outcome values conditional on observed covariates, and if the ADAS-Cog 13 outcome is missing at random, the fixed-effects estimates of the ACA will be valid [30, 31]. Therefore, to assess validity of the ACA, we first perform a multivariate logistic regression where the outcome is the missingness incidence of our target covariate and the baseline ADAS-Cog 13 score is a predictor, along with all the baseline covariates. We need not test association with ADAS-Cog 13 scores later than at baseline since our covariate missingness only occurs in baseline covariates.

Since we have five analysis models, we need to check that all of them will be valid by performing multivariate tests for each. Conditional on all other baseline covariates in the analysis models, ADAS-Cog 13 score at baseline was not associated with missingness in CSF measurement (odds ratio of 1.00 with 95% CI 0.98–1.03). For all five analysis models, we find no association between missingness of patient-reported family history and baseline ADAS-Cog 13 (odds ratio of 1.00 with 95% CI 0.98–1.03 for the three models with absolute CSF biomarkers, and odds ratio of 1.01 with 95% CI 0.98–1.04 for the two models with ratio biomarkers). These results suggest that the missingness of covariates is independent of observed outcome values conditional on observed covariates.

The second necessary condition to establish validity of the ACA is that the ADAS-Cog 13 outcome is MAR. Without additional information beyond observed data, it is inherently impossible to statistically determine whether the data is MAR or MNAR. This assessment needs to be made based on clinical expertise and knowledge of the data collection [2, 3]. We have searched the literature related to ADNI study design and have not found scientific reasons against the MAR assumption of the outcome. Given the absence of scientific evidence against the MAR assumption, we use MAR as a starting point, which is often sensible to begin with [3]. Significant deviations from the MAR assumption may manifest in diagnostic plots for the imputed data [3]. These diagnostic plots are used to explore possible evidence of deviations from the MAR assumption. They are discussed in the Results section, wherein we conclude that we have not found evidence against the MAR assumption for the longitudinal ADAS-Cog 13 score. Thus, for this study, we consider the outcome data to be MAR. Taken together, the assessment of the two necessary conditions suggests that the fixed-effects estimates of the ACA will produce valid inference.

Imputation model

To perform imputation with MICE, we must first decide which variables to include in the imputation models. As per guidelines in [3], the imputation model for a target variable should include every variable in the analysis model and variables with explanatory power for that target variable. Furthermore, variables that predict missingness of the target variable should be included in its imputation model. In our case, [1] provides reason to believe that there are variables beyond those in our analysis model that should be included in the imputation model for CSF biomarkers because of association with their missingness. In particular, they found that family history of AD was associated with missingness of baseline CSF biomarker measurements. We tested this again with the larger data set that has become available since the mentioned article and found consistent results (odds of missingness of CSF at baseline decreases by a factor of 0.46 (95% CI: 0.33–0.66) with reported family history of AD). Our final imputation models for CSF biomarkers, patient-reported family history of AD, and ADAS-Cog 13 scores are as follows. The variable to the left of the tilde is the variable that needs to be imputed, while variables to the right are all covariates. Parentheses indicate time; for example, ADAS13(0) is the ADAS-Cog 13 score at month 0, i.e. baseline. CSF bl ∼ AGE + GENDER + EDUCAT + APOE 4 + CDRSB bl + FamHist + ADAS 13 ( 0 ) FamHist ∼ AGE + GENDER + EDUCAT + APOE 4 + CDR SB bl + CSF bl ADAS 13 ( t ) ∼ AGE + GENDER + EDUCAT + APOE 4 + CDRSB bl + FamHist + CSF bl + ADAS 13 ( t 1 ) + … + ADAS 13 ( t k )

In the imputation model for ADAS-Cog 13 scores, we only impute scores collected at times t for which the missingness incidence of ADAS-Cog 13 scores was below the chosen tolerance (50%, 65%, or 75%). As predictors for this imputation model, we include ADAS-Cog 13 at times t₁ through t _k , which are all the times for which the missingness was below the tolerance, excluding time t itself (since a target variable should not be included as a predictor in its own imputation model).

It is worth noting that the times t₁ through t _k may include times after t –that is, the imputation of the scores at time t may include scores in future times as predictors. However, this should not be taken as a statement about the influence of future measurements on past measurements. Rather, it is simply a way to improve predictive accuracy of the imputation model on ADAS-Cog 13 scores. Indeed, as discussed earlier, variables were included in the above imputation models based on guidelines in [3], one of which is that variables with explanatory power for target variables should be included in their imputation. This improves predictive accuracy of the target variable, improving the quality of the imputed data sets. To that end, in our model ADAS-Cog 13 scores at time t are imputed with those at other times, past or future. Nevertheless, the imputation model could still be designed to include only those times prior to t; we verified that this valid alternative approach does not reduce the quality of the imputed data sets or change the qualitative results of this particular study.

In imputing CSF values with Bayesian linear regression, we assume a linear relationship between the CSF values and the predictors of our imputation model. We assess the validity of this model visually with the residual plots of the imputation models. As an example, consider the imputation model for tTau:

tTAU bl ∼ AGE + GENDER + EDUCAT + APOE 4 + CDR SB bl + FamHist + ADAS 13 ( 0 )

The top-left plot of Fig. 3 shows the residuals from the above model fitted to the observed (i.e. not imputed) values in the tTau model. The residuals are mean-zero with similar variance across fitted values, indicating adherence to a linear relationship. We also checked the distribution of residuals from the imputation models using a histogram and a quantile-quantile plot (top-center and -right of Fig. 3). The plots show a reasonable adherence to a normal distribution. We arrive at the same conclusion for the imputation models for the other four CSF biomarkers.

OPEN IN VIEWER

Fig. 3

(top left) Residual plot of linear model when observed tTau (not imputed) is the dependent variable and the independent variables are baseline features (Age, Sex, Education, APOE4, Family History of AD, baseline CDR-SB, and baseline ADAS-Cog 13). The x-axis (the fitted values) is estimated tTau values from the linear model. (top center) Histogram of standardized residual values from the same model. (top right) Quantile-quantile plot of standardized residual values from the same model. Theoretical quantiles are for the standard normal distribution. (bottom row) The same set of plots as in the top row, but for the linear model when observed ADAS-Cog 13 score at month 6 (not imputed) is the dependent variable and the independent variables are baseline features (Age, Sex, Education, APOE4, Family History of AD, baseline CDR-SB, and tTau) and ADAS-Cog 13 scores at all other times for which missingness was below 50% tolerance, with the exception of month 6 (since this is the month considered in the dependent variable).

The same diagnostics can be applied to the imputation of the longitudinal outcome. We assessed the validity of the linear model of ADAS-Cog 13 scores with the same procedure we used above for the tTau imputation model. Since there are multiple imputation models for ADAS-Cog 13 (one for each time point), we performed the diagnostics for each one. The bottom row of Fig. 3 shows the same plots as with the tTau imputation model for the ADAS-Cog 13 model at month 6. The plots also show a reasonable adherence to a normal distribution, as do those for the scores at all other time points, though these are not shown.

3.1 Demographic data

Table 1 provides baseline demographic information for the 1063 subjects in this study. The proportions of values missing are also provided. Family history of AD and baseline CSF measurements have the most significant missingness— nearly 50%.

3.2 Available case analysis

The results of the ACA are shown in Fig. 4 for the Aβ₄₂, pTau, and tTau analysis models, and in Fig. 5 for the pTau/Aβ₄₂ and tTau/Aβ₄₂ models, in the columns labeled ACA. Point estimates for predictors are shown along with their confidence intervals. The figures selectively show results for salient features (Age at baseline, APOE ɛ4 allele presence, CSF biomarker level at baseline, and years of education); the complete results can be found in Supplementary Tables 1 and 2. The ACA reduces our available data set to 301 subjects, less than one-third of the number of subjects in our complete data set. However, since we expect the ACA to produce valid inference, our fixed-effects estimates should still be reliable, and we can check that they align with clinical expectations. The effects of visit time and age are positive, which is in line with our expectation that cognitive performance will progressively decline on average over time and with age in the MCI group (higher ADAS-Cog 13 scores reflect greater cognitive impairment). Decreasing Aβ₄₂ and increasing values of the other four biomarkers are also associated with poorer cognitive function [6, 7], in line with our estimates. APOE ɛ4 allele presence is associated with lower cognitive performance [21, 32], although in the Aβ₄₂ and ratio biomarker models the ACA estimate is not significant.

OPEN IN VIEWER

Fig. 4

Comparison of ACA and MICE estimates with 95% confidence intervals for the effect of Age at baseline, APOE ɛ4 allele presence, CSF level at baseline, and level of education. (top row) Aβ₄₂ analysis model, (middle row) pTau analysis model, and (bottom row) tTau analysis model. Black dots represent the estimates for the quantities in the subplot titles. Confidence intervals which cross the dashed line at 0 suggest that the estimate is not significant.

OPEN IN VIEWER

Fig. 5

Comparison of ACA and MICE estimates with 95% confidence intervals for the effect of Age at baseline, APOE ɛ4 allele presence, CSF level at baseline, and level of education. (top row) pTau/Aβ₄₂ analysis model and (bottom row) tTau/Aβ₄₂ analysis model. Black dots represent the estimates for the quantities in the subplot titles. Confidence intervals which cross the dashed line at 0 suggest that the estimate is not significant.

Imputation analysis and imputation model diagnostics

Results presented in the MICE columns of Figs. 4 and 5 are for the case of 50% missingness tolerance and Bayesian linear regression with clipping of CSF values at their technical limits. From Table 2, we see that a tolerance of 50% would allow us to impute ADAS-Cog 13 scores at visit times 0, 6, 12, 24, and 36 months. Before we compare the results of the mixed-effects models between the available and imputed cases, we must first ensure that our imputed data sets look reasonable. Since CSF measurements, family history, and ADAS-Cog 13 scores are our major sources of missingness, we focus on assessing their imputed values.

To ensure that imputed values are reasonable, we look at boxplots of a subset of the 50 imputed data sets for the CSF values, which had significant missingness, alongside boxplots of the observed data. These plots are shown in Fig. 6. Since CSF is clipped, we see the same upper and lower bounds for the distribution of the imputed data sets as for the observed data. As mentioned previously, whether or not we clipped did not have an impact on the qualitative results discussed in this section. Importantly, the median, range, and interquartile range of the imputed data sets are comparable to those of the observed data.

OPEN IN VIEWER

Fig. 6

Boxplots of observed and imputed CSF values for each of the five models for the first 15 imputations: Aβ₄₂ (top left), pTau (top right), tTau (middle left), log-transformed pTau/Aβ₄₂ (middle right), and log-transformed tTau/Aβ₄₂ (bottom). Observed data are shown in blue and imputed data sets are in red. Only the first 15 of the 50 imputed data sets are plotted.

We also look at the proportions of subjects with and without patient-reported family history of AD. Figure 7 displays these proportions for the observed data (in black) and for the first 15 imputed data sets (the different shades of blue). In the observed data set, we see that about 60% of subjects reported family history of AD while the other 40% reported no family history of AD. The imputed data sets have proportions comparable to these, giving some evidence that they are reasonable imputations.

OPEN IN VIEWER

Fig. 7

Marginal proportion of presence (TRUE) or absence (FALSE) of patient-reported family history of AD. Black bars are observed data while blue bars are each different imputed data sets from the tTau model. Only the first 15 of the 50 imputed data sets are plotted. Plots were generated by the propplot function obtained from GitHub.

Since we have also imputed the time-varying outcome, we analyze the plots shown in Fig. 8 of the behavior of imputed ADAS-Cog 13 scores for the tTau model. We use months 12 and 36 as illustrative examples. The top two boxplots in Fig. 8 show that the median and interquartile range of the imputed data sets are comparable with those of the observed data for both months 12 and 36. However, the range of the imputed data sets exceeds the range of the observed data sets. Indeed, ADAS-Cog 13 scores are values between 0 and 85, but the imputed values are not restricted to this possible range. It is not always necessary that individual values be realistic, so long as sample estimates, like mean and standard deviation, are preserved in the imputed data sets [33]. Still, to assess robustness we performed our analyses again using predictive mean matching for ADAS-Cog 13 scores. The results regarding estimates and confidence intervals are nearly identical to those obtained by using Bayesian linear regression, so our interpretations remain the same. The bottom two plots in Fig. 8 show the behavior with predictive mean matching of the imputed ADAS-Cog 13 scores at 12 and 36 months, which now take on only values between 0 and 85. Though not shown in the figure, these are also nearly identical to results obtained when using yet another imputation method: Bayesian linear regression followed by clipping imputed values between 0 and 85.

OPEN IN VIEWER

Fig. 8

Boxplots of observed and imputed ADAS-Cog 13 values for the tTau models for the first 15 imputations at 12 months (left) and 36 months (right). (top) data imputed by Bayesian linear regression. (bottom) data imputed by predictive mean matching. Observed data are shown in blue and imputed data sets in red. Only the first 15 of the 50 imputed data sets are plotted.

Figure 8 also gives us some insight into the MAR assumption for the longitudinal outcome of ADAS-Cog 13 score. As mentioned previously, without additional information beyond observed data, it is inherently impossible to statistically determine whether the data is MAR or MNAR [2, 3]. However, significant deviations from the MAR assumption may manifest in diagnostic plots for the imputed data, such as Fig. 8 of our study, in which the differences between the distributions of the observed data (blue) and imputed data (red) could end up being very different [3]. It is encouraging, in our case, that those distributions appear very similar, as displayed in Fig. 8. Thus we have not found evidence against the MAR assumption for the longitudinal ADAS-Cog 13 score.

To be able to compare ACA with MICE results, we must ensure that the estimates do not deviate far from those of the ACA, as this would indicate bias in our imputation model. We can get a rough sense of whether the estimates are reasonably close to each other by observing whether their confidence intervals overlap. All of our estimates are reasonably close to the ACA estimates by this standard, as can be seen in Figs. 4 and 5, and further verified with Supplementary Tables 1 and 2.

We see that, for the Aβ₄₂ and pTau/Aβ₄₂ models, the confidence intervals surrounding the APOE ɛ4 estimate has become tighter, and now indicates a significant effect. This is not surprising given that with imputation we are able to recover the remaining two-thirds of the subjects, drastically increasing our available data set. As mentioned earlier, this is in line with clinical expectations [21, 32]. For the tTau/Aβ₄₂ model, the estimate has become tighter after imputation, but not enough to indicate a significant effect. For the pTau and tTau models, APOE ɛ4 is significant already in the ACA, with confidence intervals becoming even tighter after imputation. Also worth noting are the tighter confidence intervals around the Age estimates, which become significant after imputation for the analysis models with pTau/Aβ₄₂ and tTau/Aβ₄₂, as well as the education estimates, which become significant across all five analysis models after imputation. There are some studies which are in line with this finding regarding education [34, 35], while others do not find any significant association between ADAS-Cog scores and education level [14].

When performing these same analyses for tolerances 65%, and 75%, we draw the same conclusions described above for all of the results. Overall, these results demonstrate the increased statistical efficiency of the imputed data analysis over the ACA.