Abstract
Recently, some researchers have employed nonbinary measures of successful aging. Little has been done to determine whether these newer measures yield similar findings compared to traditional binary measures. To test for differences, three measures of successful aging were constructed within five waves of the Americans’ Changing Lives data set. A number of demographic, behavioral, and psychosocial predictors were used to predict each outcome, examining whether estimates of effect sizes and statistical significance were similar across measures. Although many effect sizes were similar, conclusions regarding statistical significance were inconsistent. For instance, the binary measure downplayed income gradients, the ordinal measure found more racial disparities, and the continuous measure was most likely to detect effects for stressful life events. These differences may be due to the statistical techniques used to handle each outcome. Results imply that uneven application of operationalization approaches may complicate replication efforts, suggesting a need for consistent measurement standards.
One of the most common approaches to measuring successful aging has been to construct a simple binary that classifies individuals as successfully aged or not (Cosco, Prina, Perales, Stephan, & Brayne, 2014; Depp & Jeste, 2006). Recently, however, some have raised concerns regarding this dichotomous approach, leading to the pursuit of alternatives, such as the ordinal scale proposed by Young and colleagues (2009) and a continuous coding scheme by Cosco and colleagues (2015). Although fairly new, the adoption of various nonbinary measures is a noticeable trend in recent successful aging studies (Hsu & Jones, 2012; Kok, Aartsen, Deeg, & Huisman, 2017a, b; Manierre, 2018; Pruchno, Wilson-Genderson, & Cartwright, 2010; Whitley, Popham, & Benzeval, 2016). Although seemingly benign, it is important to understand whether the choice of measure can shape the substantive conclusions of a study.
While inconsistency is nothing new to the successful aging literature, discussions have focused on how disagreements about the distinct traits that constitute success might limit comparability by undermining measurement consistency (Cosco et al., 2014). There is a reason to expect, however, that the choice of nonbinary and binary measures is another threat to comparability. On one hand, the two options reflect a conceptual distinction, defining successful aging as a matter of degree along a broad continuum versus a discrete state that reflects membership in an elite group (Bowling, 2007). At the same time, the statistical techniques used to analyze these outcomes can also generate differences in point estimates and hypothesis tests due to differences in statistical power and data handling (Silberzahn et al., 2018). These two factors suggest that binary and nonbinary permutations of the same data might generate divergent conclusions.
This discrepancy in measurement preferences is also relevant to discussions regarding the replicability of research in the sciences. Analysts and commentators have suggested the emergence of a “replication crisis,” wherein results in the social sciences and medicine are not adequately reproduced in subsequent studies (Freedman, Cockburn, & Simcoe, 2015; Open Science Collaboration, 2015). This undermines the capacity of science to inform policy and build theory effectively. In response to this concern, The Gerontologist has recently called for the assessment of the scope of replication issues (Pruchno et al., 2015). The current study adds to this dialogue by exploring one such facet of the replicability of gerontology research, exploring whether different measures of successful aging have their own unique quirks that limit the comparability of results across studies.
Literature Review
Successful Aging
This study explores three strategies for operationalizing the same underlying conceptualization of successful aging. In this case, the three-dimensional model of Rowe and Kahn (1997, 2015) is utilized. Rowe and Kahn’s model specifies three components: First, an individual must avoid disease and disability. Second, individuals must also maintain high physical and cognitive functioning into old age. Lastly, the potential for active living granted by the two earlier points must then be capitalized on in the form of active social engagement. Active engagement involves maintaining interpersonal relationships and engaging in some form of activity that is useful to others. For instance, a 75-year-old with no mobility issues or chronic illnesses, as well as no signs of memory loss and frequent interactions, would be considered successfully aged if they were also still involved with their friends and family.
Rowe and Kahn’s conceptual model is used here because of its amenability to quantification and its popularity in the successful aging literature, which increases the reach of the current analysis. It should be emphasized that there have been many valid critiques of Rowe and Kahn’s model, though, including its overstatement of biomedical criteria and its omission of mental health (see Bülow & Söderqvist, 2014; Martinson & Berridge, 2015, for reviews). Other conceptual models attempt to rectify these limitations in various ways, but the general point of this analysis extends beyond conceptual debates. Instead, this study reflects a hypothetical “what-if” scenario in which conceptualization was held constant. The distinction between binary and nonbinary measures tested here would likely apply to most quantifiable models of successful aging, not just Rowe and Kahn’s.
Transitions in the Measurement of Successful Aging
Success as a dichotomy
As noted, a widely used measurement strategy has been the use of “successful/not successful” dichotomies to identify success (Cosco et al., 2014). The general process has been to first gather a set of measures that reflect some or all of the elements of their concept. Then, the researcher sets criteria for success on each measure or dimension. If all criteria are met, the respondent is classified as successfully aged. Many recent studies have employed this general process with only small variations, though conceptualizations vary across studies (e.g., Brandt, Deindl, & Hank, 2012; Feng, Son, & Zeng, 2015; Nosraty, Pulkki, Raitanen, Enroth, & Jylhä, 2017; Schafer & Ferraro, 2011). The advantages of this approach are twofold. First, binaries are fairly simple to create from a set of indicators. Second, they seem to have a definite interpretation, reflecting the proportion of respondents with several desirable traits.
Critiques to the binary approach
In more recent years, some have argued that binary measures are inadequate, providing motivation for the adoption of new measurement approaches. The first criticism raised against binary measures of successful aging is that they do not adequately capture variation in aging outcomes (Cosco, Stephan, & Brayne, 2015; Hsu & Jones, 2012; Wickrama, Mancini, Kwag, & Kwon, 2013; Young, Frick, & Phelan, 2009). This critique is especially relevant following efforts to define successful aging as one part of a broad distribution that reflects all possible aging outcomes (Bowling, 2007; Cosco et al., 2014; Kahn, 2002; Young et al., 2009). The heterogeneity hidden by binary measures was highlighted by Hsu and Jones (2012), who classified respondents into one of the three levels of success. Taken as a binary, they found that 29.1% of respondents had aged successfully and 70.9% “failed” to do so. When the “unsuccessful” group was broken into middling and poorly aged groups, it revealed that only 11.4% of respondents were “failing” to age decently, while the remaining 59.5% fell into one of two middling categories. This latter result is both less dismal and more precise in its policy prescriptions, suggesting that a binary measure might paint a misleading picture.
Others have suggested that the binary measure’s interpretation is not as straightforward as assumed, as they tend to capture variation in physical health and age (Cosco, Stephan, & Brayne, 2014). This critique is exemplified by studies that have found that older individuals who are happy or socially engaged are often classified as unsuccessfully aged due to declines in physical health that disqualify them from the “successful” category (Cho, Martin, & Poon, 2012; Cosco et al., 2014). This is an issue for two reasons. First, it means that measures that purport to capture multiple dimensions of successful aging are mostly measuring variation in physical health. Second, loss of physical functioning is strongly correlated with age, meaning that successful aging is unintentionally conflated with youthfulness (Cosco et al., 2014).
Emerging alternative measures of successful aging
In light of these critiques and a general desire for nuance, several nonbinary methods have been proposed, although none have been widely adopted yet. The first of these are ordinal scales with limited variability (Kok et al., 2017b; Manierre, 2018; Whitley et al., 2016; Young et al., 2009). The main modification to the binary approach is that after a list of criteria for each dimension is set, the researcher simply tallies the number met instead of treating it as an all-or-nothing assessment. The result is a scale that expands on the binary measures by extending their range while retaining the all-or-nothing incidence measure in the form of a perfect score. Heterogeneity is captured to a degree, and variation from all dimensions is directly assessed.
A second alternative to the binary measure was recently proposed by Cosco and colleagues (2015). While the prior measures had a small range, this method used a more complex calculation process to create a more granular measure. First, all items are reweighted so that response categories range from 0 to 100 points. Once rescaled, questions are separated into their respective dimensions and averaged. Then, all of the dimension-specific averages are averaged to create an overall successful aging score. Like the ordinal approach, this method could be applied consistently to any data set where a binary measure was previously employed since it is merely a difference in indicator manipulation.
These two simpler alternatives are also joined by a number of more sophisticated methods. A few studies have sought to estimate successful aging as a latent variable using structural equation modeling (Pruchno et al., 2010). Others have employed latent group classifications, such as group-based trajectory modeling or latent class analysis, to inductively generate multiple levels of successful aging (e.g., Hsu & Jones, 2012; Kok et al., 2017a; Pruchno, Wilson-Genderson, Rose, & Cartwright, 2010). The previously described study by Hsu and Jones highlights how this method, which captures successful aging as a trajectory that is followed over time, can be used to inductively create nonbinary measures of successful aging.
These sophisticated approaches have the advantage of providing more assurances of model fit with few assumptions about the weighting and cutoffs for different indicators or groups. However, the cost is complexity. Unlike the other two methods, these are challenging to implement, requiring large samples, advanced statistical modeling, and/or longitudinal data. This can limit the accessibility of successful aging research based on funding availability. Furthermore, latent trajectory models hinge on a modified definition of successful aging, defining it not as a discrete state but instead as something that is persistent. Although consistent with a life-course model of successful aging, Rowe and Kahn’s conceptual model has been criticized specifically for not functioning well in this context (Stowe & Cooney, 2015). This life-course emphasis does not apply to this study, which is focused on cross-sectional applications. Structural equation modeling is also problematic because it depends on factor loadings to adjust for measurement error, which amounts to certain indicators or dimensions being weighted heavier than others, implicitly downplaying their importance. As a result, though useful in specialized circumstances, these techniques may not be the best fit for identifying a widely applicable and appropriate alternative to binary measures.
The challenge of comparability
It is already understood that conceptual differences in the study of successful aging might limit replication efforts (Cosco et al., 2014; Pruchno, 2015). However, it seems prudent to expect that the uncritical adoption of nonbinary measures will lead to further inconsistencies. Pruchno (2015) observed these measurement challenges, urging gerontologists to “develop consensus about what successful aging is and how it should be measured. We should approach this goal knowing that our measures will not be perfect, but at least out findings will be comparable” (p. 3). The increased uptake of nonbinary measures points to a need to establish if these different indicators are actually comparable with past research that utilized other measurement strategies. To date, most studies employing one type of measure freely cite and attempt to build upon studies that utilize different measurement strategies. It’s not clear, however, if such comparisons are valid or if we are drawing false parallels.
There are two potential reasons why these different measurement approaches might yield different results. First, binary measures capture membership in a very specifically defined group, ultimately comparing successful with unsuccessful using a logistic regression model. A continuous measure instead illustrates a distribution, typically predicted using a linear regression model, that estimates change anywhere along that distribution, perhaps from low to medium or maybe from high to very high. This would manifest itself in the form of differences in both point estimates and hypothesis tests across models. Another explanation is that logit models are also less efficient than ordered logit models (OLS), resulting in inflated standard errors. This would yield inconsistent hypothesis tests but similar point estimates.
This study probes for inconsistencies by exploring if a binary, ordinal, and continuous measure of successful aging agrees about the effect size and significance of various correlates. These comparisons are conducted in five separate waves, providing multiple points of reference over time. By putting these different indicators of successful aging side by side, we will be able to see if they yield consistent conclusions and policy implications or if the data’s story is shaped by seemingly mundane coding decisions.
Research Design
Data
This analysis uses five waves of the Americans’ Changing Lives (ACL) data set, collected in 1986, 1989, 1994, 2002, and 2010, respectively. The ACL uses a stratified random sample of the White and Black U.S. population. African Americans were oversampled at a ratio of 2:1 to ensure that there would be sufficient statistical power for multivariate analyses. When weighted, these data are representative of the American population in 1986 (House, 2014). Each of the cross-sectional analyses incorporates only individuals who were 50 or older at the time of the wave.
This was a useful data set for this study’s research question, as it included many indicators for successful aging and a robust set of psychosocial, demographic, and behavioral risk factors. These data have been widely used in social gerontology research, including studies of successful aging, meaning this comparison will retrospectively shed light on past studies as well. Compared to other studies, such as the newer Health and Retirement study, it has a more consistent set of measures of psychosocial health and life events and a very long timescale. Recent research by Manierre (2018), which used the ACL data, also provided a useful foundation for some of the measures used this study.
Note that respondents who turned 50 in a later wave were included in the analyses for that wave and all later waves. Although there is overlap between waves, each wave also adds in entirely new cases. This, in essence, refreshes the study sample over time, helping to retain statistical power and variation. It also affords a degree of replication by performing the analyses using new cases. For instance, in Wave 3, 19% of cases analyzed were not used in Wave 1. By Wave 5, that number had increased to 78.7%. It also has the benefit of ensuring that each wave is not just tracking a static group over time. This means that inconsistencies detected probably reflect biases inherent to the measure rather an anomaly that is limited to a particular cohort.
Measures
Successful aging
Three measures of Rowe and Kahn’s model (avoiding disease and disability, maintaining high functioning, and continued engagement with life) were constructed from the same set of variables. The summary statistics for the variables used to construct each of these measures can be found in Table 1. Details on how the binary, ordinal, and continuous measures were constructed from the following indicators are presented in Table 2.
Weighted Descriptives for Successful Aging Measure’s Components.
Summary of Method for Constructing Each Successful Aging Measure.
The first dimension, avoiding disease and disability, was measured by a self-report of the following health problems for the past 12 months: arthritis/rheumatism, lung disease, hypertension, heart attack or other heart trouble, diabetes, cancer, stroke, broken bones, and incontinence. Avoiding disability was assessed by first asking if the respondent had difficulty bathing or getting out of bed. 1 If no, the respondent was then asked about less severe limitations. The second dimension, continued high cognitive functioning, was assessed with a tally of errors on the Short Portable Mental Status Questionnaire (Pfeiffer, 1975). Participants were asked the date, the day of the week, their mother’s maiden name, the past two presidents of the United States, and to subtract 3 from 20 repeatedly. The third dimension, social engagement, was measured following prior operationalizations of Rowe and Kahn’s model (Brandt et al., 2012; Hank, 2010; McLaughlin, Jette, & Connell, 2011). Two subcategories were made to reflect productive and informal social engagement. Productive activity was measured with questions asking if the respondent was employed and how many hours they had volunteered in the past year. Also included was the respondent’s estimate of how many hours they spent a year helping others with housework, childcare, transportation, and other unpaid labor. To capture informal social engagement, the respondent was asked how frequently they attended religious services and how often they attended meetings of groups, clubs, or organizations. A respondent’s marital status was also included in this measure, as per prior research (Brandt et al., 2012).
Predictors of Successful Aging
This study investigates several factors that have been found to predict successful aging in the past or have been the focus of other studies (see Depp & Jeste, 2006; Rowe & Kahn, 1997, for detailed discussions). These encompass numerous demographic, behavioral, and psychosocial risk factors, which are elaborated below and summarized in Table 3.
Weighted Descriptive Statistics for Predictors.
Note. All indicators were standardized for analyses.
Sociodemographics
All models include measures distinguishing between sex, White versus non-White status, and age. Measures of family income and education were also assessed. In terms of demographics, women tend to be more likely to successfully age, likely owing to greater longevity overall (Depp & Jeste, 2006; Kok et al., 2017b). It has also been found that minority status, low income, and low education are consistently associated with greater health risks in the future, including reduced odds of successful aging (Brandt et al., 2012; Kok et al., 2017b; Marmot, 2015; Williams & Sternthal, 2010).
Health behaviors
Various lifestyle and environmental factors also effect one’s odds of successfully aging. Proactive health behaviors such as not smoking, a healthy diet, moderate alcohol consumption, and exercise are associated with successful aging (Franklin & Tate, 2009; Rowe & Kahn, 1997). To account for these, physical activity was measured with questions asking how often the respondent worked in the garden or yard, took walks, and engaged in active sports or exercise. These items were averaged and standardized. Respondents were also classified based on their smoking and typical monthly drinking habits. Note that drinking was excluded from Waves 4 and 5 analyses, as too few respondents drank more than 60 drinks in a month. The ACL did not have any specific measures of diet, but three body mass index (BMI) categories were included as a proxy.
Psychosocial risks
Researchers have also documented the effect of stress, social support, and mastery/self-efficacy on successful aging (Kok et al., 2017b; Rowe & Kahn, 1997). In general, the existing literature on these topics suggests that high chronic stress and/or numerous stressful events are deleterious to one’s health prospects (Thoits, 2010). However, high levels of mastery (i.e., a sense of control over one’s life) and social support from family and friends tend to buffer the deleterious effects of those stressors, thereby facilitating successful aging (Rowe & Kahn, 1997; Thoits, 2010).
Stress exposure was measured in two ways: the occurrence of stressful life events and the presence of self-reported chronic stressors. In 1986, respondents were asked if they had experienced the death of a parent, child, or spouse; a divorce; or an assault. In addition, each subsequent wave asked about seven other stressors that occurred in the past 3 years, including job loss, criminal victimization, and financial difficulties. Each wave’s count of life events includes all documented events from prior waves.
A measure of chronic stress exposure was crafted following a prior analysis of the ACL (Umberson, Williams, Powers, Liu, & Needham, 2005). This measure captures stress related to finances, parenting, marriage, caregiving, and work. Financial and parenting stress were each measured with a three-question scale. Marital strain was assessed with a 7-question scale and a count of marital problems, including cheating and abuse. Job and caregiving strain were derived from a single Likert-type scale directly asking how stressful this work was to them. The items from each scale were first standardized and averaged separately. Then, the 5 scales were averaged and standardized a final time to create a summary of overall stress exposure.
Sense of mastery was measured with a 6-item scale. Positive social support from the respondent’s friends, mother, father, children, and spouse was assessed with 2 items for each source of support in both Waves 1 and 2. Like the stress scale, both of these indicators were standardized.
Analytic Approach
Five sets of three regression models were estimated to determine the correlates of successful aging in each wave. The continuous measure of successful aging was modeled using an ordinary least squares regression model, the ordinal measure was predicted using an ordered logistic regression model, and the binary measure was estimated using a logistic regression model. Missing responses were deleted list wise, leading to anywhere from 6.8% (Wave 2) to 16.3% (Wave 5) of responses being discarded.
Models were compared by examining both point estimates and the statistical significance of those estimates. Both forms of logit model yielded odds ratios, so the OLS model’s estimates were transformed to reflect an odds ratio scale. This is done by first transforming a standardized OLS coefficient and confidence intervals into Cohen’s D and then further transforming this value into an odds ratio (Borenstein, Hedges, Higgins, & Rothstein, 2011; Silberzahn et al., 2018). Both are accomplished via the following equation, where β represents a standardized OLS coefficient:
These estimates were graphed in forest plots to visualize differences.
Significance tests were also monitored for discrepancies. Although some have raised concerns about the usefulness of estimates of significance, this is important to consider because of their prevalence in prior research (Wasserstein & Lazar, 2016). An inconsistency was logged if a covariate was significant at the p < .05 level for only one or two of the three measurement variations.
Results
Summary statistics for each successful aging measure are in Table 4. Depending on the wave, the binary measures of successful aging found that anywhere from 9.9% to 13.9% of the sample met all the criteria for successful aging. For the ordinal measure, the highest score (3) had the exact same proportions as the binary, but it also showed that most respondents had middling scores between 1 and 2. Respondents with scores of 0 comprised between 13.5% and 22.9% of each wave. The continuous measure had a normal distribution after squaring, with most respondents in each wave clustering around average scores of 4,466–5,122 out of 10,000. The spread for this distribution was fairly wide in each wave, with standard deviations ranging between 1,948 and 2,051.
Weighted Summary Statistics for Each Successful Aging Measure.
Comparing Effect Sizes
The results of the five sets of regression models can be found in the Online Appendix tables as well as forest plots for all waves. Figure 1 presents a plot of odds ratios and confidence intervals from each of the three models. In one sense, the estimates were quite consistent in that they were almost always within one another’s confidence intervals. Based on this, the estimates can be considered statistically equivalent except for a few anomalies—physical activity, in particular. However, there is still some evidence of systematic variation in the estimation of effect sizes. Comparing ordinal and binary models, 20.8% of the estimates for diverged by a factor of .2. Conversely, the ordinal versus continuous comparison diverged 41.6% of the time, and the continuous versus binary model differed by .2 or more 43.8% of the time. This illustrates that despite statistical similar, the continuous model tended to yield larger effect sizes, especially for income, education, and physical activity. These three variables, plus mastery and life events, were the source of almost all the largest disagreements about effect sizes. Based on this, it also appears that the ordinal and binary models tend to agree more closely regarding point estimates.
Summary of effects and 95% confidence intervals across three models within three different waves of data.
Comparing Statistical Significance Decisions
All models agreed on the significance of certain variables. Heavy drinking and chronic stress were found to always have null associations, and being overweight was almost always found to have no significant association save one exception. Social support was typically nonsignificant. Old age was always associated with lower odds of successful aging. All measures were also consistent in finding that higher levels of educational attainment correspond with outcomes that were more favorable.
More often, however, models yielded different conclusions about statistical significance. These disagreements are tallied in Table 5. All but one binary model determined that high physical activity was not associated with successful aging, but the other measures always found a strong positive association. The binary model was much less sensitive to income gradients, especially on the middling income categories, reflecting the effect of smaller point estimates and/or larger standard errors. The dichotomous measure also found that mastery was significantly associated with successful aging only once, while the continuous and ordinal measures found associations in five and three waves, respectively.
Number of Waves Where Successful Aging Measure Yielded Differences in Conclusions.
The ordinal and continuous measures also had unique findings of their own. The ordinal measure was more prone to detecting racial disparities, with Black respondents having lower odds of high scores in four of the five waves versus zero of five for the binary and only two of five for the continuous measure. The continuous scale was also noticeably more sensitive to the psychosocial risk factors, always finding that an excess of stressful life events and low mastery were negatively correlated with success. It also identified current smoking as a significant correlate in four of five waves versus the ordinal’s one and binary’s zero.
One way to summarize how frequently measures disagreed is to calculate the percentage of covariates’ estimates, 96 in total, for which there is an inconsistency. This information is presented in Table 6. It was found that about half of covariates had at least one disagreement in each wave. The continuous measure disagreed with the binary on an average of 40.6% of covariates. Similarly, the ordinal measure disagreed an average of 33.3% of the time. The continuous and ordinal measures were also more likely to agree with each other, although they did still disagree an average of 20.83% of the time, with the highest rate of disagreement being 25% of covariates estimated in wave 1.
Summary of Conflicting Significance Tests in Different Successful Aging Measures.
Discussion
This study was conducted in response to a number of studies either calling for or implementing nonbinary measures of successful aging (Cosco et al., 2015; Kok et al., 2017a; Whitley et al., 2016; Young et al., 2009). Adopting these nonbinary measures seems prudent in light of issues that have been raised, but it is important to determine whether these measures yield the same substantive conclusions. This analysis suggests that operationalizing the successful aging measure in different ways can alter conclusions, especially about statistical significance. The binary measure tended to downplay the effect of income inequality, perhaps casting doubt on assertions that policies must address income inequality to increase successful aging (Katz & Calasanti, 2015; Whitley et al., 2016). This measure also suggested that physical activity is not such a strong correlate of successful aging after all, despite findings to the contrary (Depp & Jeste, 2006). Conversely, the ordinal measure implied the existence of racial disparities in successful aging, which is consistent with the broader health disparities literature (Williams & Sternthal, 2010). At the same time, the continuous measure was the most consistent at reproducing the finding that stressful life events as associated with negative health outcomes, as Kok and colleagues (2017b) recently found with a similar measure. Assessed independently, each of these different conclusions might have been found and reported as evidence for policy or research agendas, but placed by side, it can be seen that these conclusions were shaped by seemingly mundane measurement choices.
These findings echo prior research which has demonstrated that analysts’ use of different modeling strategies can alter the conclusions drawn from the same data set (Silberzahn et al., 2018). In the current study’s case, it appears that these different conclusions are partially attributable to differences in statistical power between modeling strategies, which lead to a variable standard errors and thusly inconsistent significance decisions. Although some rare exceptions existed, the effect sizes for many covariates were within one another’s confidence intervals, suggesting that though there was an underlying conceptual variation (e.g., success as continuum vs. binary), the correlates of either were actually very similar. However, there was some evidence to counter this, as the continuous measure had inflated effects sizes relative to the other two measures, which may be attributable to underlying conceptual differences. Regardless of mechanism, this exercise illustrates statisticians’ caution against leaning solely on p values as a mechanism to determine the “realness” of results instead of carefully considering effects sizes and the replication of results (Wasserstein & Lazar, 2016).
Until a consistent approach to measuring successful aging is adopted, however, the field of successful aging may be vulnerable from two angles. First is the aforementioned “replication crisis,” wherein many analyses are not adequately reproduced (Freedman et al., 2015; Open Science Collaboration, 2015). Here, we can see that the mix of conceptual variations, and now operationalization variations, reinforces the illusion of replication when it may not be occurring. A second concern is that the differences observed in this study reflect an avenue through which results can be consciously manipulated via p hacking (the manipulation of statistical significance to garner favorable results). In other words, these different measures could be selected to create an analysis with more sensational or desirable results. Most fields show a publication bias toward statistically significant results, which serves to motivate this behavior (Head, Holman, Lanfear, Kahn, & Jennions, 2015). To guard against this, researchers and reviewers should determine whether substantive results are being influenced by coding decisions by assessing several different coding schemes to confirm the robustness of results across multiple measures.
The ideal way to avoid the issues presented in this study is for researchers to exercise consistency in how we measure successful aging, to establish measurement standards, and to exercise caution in suggesting that a comparison of studies using binary and nonbinary measures is meaningful if only statistical significance is used as grounds for comparison. However, measures may never be fully harmonized due to valid conceptual disagreements. Still, this study does provide some empirical justification for favoring continuous or ordinal measures over binaries. The ordinal approach presents a useful middle ground, agreeing on significance decisions most often with the continuous measure, while having fairly comparable effect sizes with the binary variant. Avoidance of the binary is advisable since the binary used in this study varied mostly along lines of physical health (Cho et al., 2012; Cosco et al., 2014; McLaughlin, Connell, Heeringa, Li, & Roberts, 2009). For instance, in the first wave, 59.6% of participants met the cutoff for social engagement, while only 26.3% met the cutoffs for having no functional limitations and conditions. As such, the binary measure is deficient in the sense that it largely conceals variation in the other dimensions of successful aging.
Despite the inconsistencies, there are also certain results that were reliable across outcomes. These included education’s consistently strong gradient, the presence of at least some income-related effects, the negative association between age and success, and the null effects for chronic stress and social support. The finding that successful aging is associated strongly with socioeconomic indicators conforms with a wide body of research linking social standing with health (Katz & Calasanti, 2015; Kok et al., 2017b; Marmot, 2015). At the same time, the limited effects of chronic stress are somewhat surprising. This is likely due to an unexplored mediating pathway from stress to health via mastery and support, as typically specified by stress process models (Wheaton, 2010). In this case, the effect of chronic stress is significant until mastery and support are added to the model, consistent with basic criteria for mediation (Baron & Kenny, 1986).
It should be emphasized, however, that none of the measures examined here were flawless. The binary measure omits heterogeneity, but it yielded a useful estimate of incidence. The ordinal measure was, ultimately, just a series of binaries added together. It still (over)simplifies thinking about successful aging, only now this occurs on the level of a specific dimension instead of the overall measure. The continuous measure has a much greater capacity for nuance than the ordinal variant, but it was designed in such a way that it ignores benchmarks of internal consistency (Cosco et al., 2015). Unfortunately, this risks scale incoherence, muddying correlations and limiting interpretability. It also seems to overestimate effects of some covariates, perhaps reflecting its different conceptual underpinnings. Furthermore, all three of these measures are also strongly correlated with age, meaning that they do not overcome the implication that successful aging equals youth (Cosco et al., 2014). It will be up to future research to explore avenues to alleviate this issue.
Limitations
There are several limitations that should be considered when interpreting this study’s results. First, the current study depends on older data, ranging from 1986 to 2010. Many pivotal policy changes have occurred since 2010 that may alter the substantive correlations detected here, though it is not clear if this would alter the primary conclusion of this study. For instance, the Affordable Care Act, passed in 2010, ensured better drug coverage for Medicare recipients and an elimination of out-of-pocket costs on preventative care. Both of these might attenuate the socioeconomic gradients observed in this study. The results of this study should be viewed with some caution until they are replicated in other major studies of aging with larger and more current samples, such as the Health and Retirement Study.
The study’s sample is also skewed toward the “young–old” since the cutoff for entry into the study was 50 years of age. The low age limit was necessary due to very few respondents over 65 meeting all of the cutoffs for the binary and ordinal measures of successful aging. This suggests that the criteria implied by Rowe and Kahn’s model were too restrictive, setting expectations that could not possibly be met. One advantage of the continuous measure is that it was not susceptible to this issue, as it did not require the definition of arbitrary “all or nothing” cutoffs in the same way. Subsequent studies may yield different results to the current analysis when looking specifically at the 65 or older bracket.
Conclusion
Although moving away from binary measures is an appealing prospect, this investigation suggests that binary and nonbinary measures do not always agree about the significance of many correlates, even when point estimates are similar. Researchers will need to be mindful of how these measurement choices can shape conclusions. More importantly, however, theorists should be clear about if their model should be operationalized in binary/nonbinary terms to preempt a proliferation of disparate approaches. Conceptualization provides one part of this puzzle, but follow-up research should also establish an empirical foundation for measurement choices. For example, validation studies can pit each operationalization side by side to determine if one is closer to reality. Other analyses can also replicate this study’s investigation if there is consistency in the areas where binary and nonbinary measures disagree, so that more informed design choices can be made. There is also ample room for the development of new nonbinary measures of successful aging, especially ones that are easily applied across many data sets. Creating a strong methodological foundation for future research will enrich the study of successful aging and improve its utility to policy makers.
Supplemental Material
Supplemental Material, APPENDICES - Binary and Nonbinary Measures of Successful Aging: Do They Yield Comparable Conclusions?
Supplemental Material, APPENDICES for Binary and Nonbinary Measures of Successful Aging: Do They Yield Comparable Conclusions? by Matthew Manierre in Research on Aging
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Supplemental Material
Supplemental material for this article is available online.
Note
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
