Categorization of Health Insurance Coverage Type from Survey Questions after Health Reform: The Case of the Current Population Survey

Abstract

Measuring health insurance in surveys has always been challenging, and the Affordable Care Act (ACA) introduced considerable ambiguities. For example, the public/private line was blurred with the introduction of marketplace coverage, which is considered private coverage even though in some cases, it is partially or fully subsidized by the government. This study uses a rigorous design where administrative records are linked to survey data. We compare alternative algorithms that employ survey data points found in several major national surveys to categorize coverage type, focusing on the very difficult challenge of separating private marketplace coverage from public coverage. This is important, given researchers’ and policymakers’ need to produce estimates of public versus private coverage from survey data. Results indicate that integrating a data point on plan name reduces a more simplistic algorithm’s overestimation of marketplace coverage and results in significant improvements in accurate categorization across public and private coverage types.

Keywords

marketplace public coverage administrative records validation

Introduction

Measurement error in health insurance surveys is well documented (Davern et al. 2008; Nelson et al. 2000; Pascale 2016). Studies have identified numerous reporting errors such as failing to report Medicaid (Call et al. 2013; Noon et al. 2019), confusing one coverage type for another (e.g., Medicaid and Medicare), double reporting the same coverage in more than one category, and reporting out of scope plans (e.g., dental/vision plans) (Cantor et al. 2007; Loomis 2000; Pascale 2008, 2016). Implementation of the Affordable Care Act (ACA) in 2014 created additional measurement challenges, namely introducing a new coverage type (“marketplace”), and subsidies that increasingly blur the distinction between public and private coverage. Marketplace coverage is considered private, but the premium is often subsidized, sometimes in full. Meanwhile Medicaid and other types of public coverage generally do not require the enrollee to pay a premium, but both pre- and post-ACA, several of these public programs do require enrollees to contribute toward the premium. Another issue is terminology. The term “marketplace” (aka healthcare.gov) has a dual meaning: as both the coverage itself and as the “portal”—the place to peruse coverage options, apply for, and enroll in a plan. Compounding the problem is that a spectrum of coverage types is available on the portal, from unsubsidized private marketplace coverage to fully subsidized Medicaid. All these factors combined to create ambiguity that is baked into the post-ACA health care landscape.

The implications for measurement error in surveys are that simply adding a question on marketplace coverage to an existing battery of questions on coverage types risks that enrollees who obtained their Medicaid on the marketplace portal will select “marketplace” rather than Medicaid. This is especially problematic, given the already chronic and high levels of Medicaid underreporting. Thus, while the relatively low prevalence of marketplace coverage may suggest that measurement error of marketplace coverage per se is a trivial concern, to the extent that misreporting and miscategorization of marketplace coverage reduces correct categorization of public coverage, marketplace measurement error compromises the aggregate public versus private estimates, which is a major concern.

In anticipation of the implementation of the ACA in 2014, research was conducted in 2011 in Massachusetts (given its state-level law on which the ACA was based) exploring how to adapt major federal surveys for the new marketplace. Results suggested that due to the ambiguities noted above, for some types of coverage (particularly public and marketplace), it would be insufficient to rely on any one survey question on coverage type. However, it was found that surveys could maintain their existing health insurance modules asking about conventional (i.e., pre-ACA) coverage types, and add three new questions post-ACA: whether the coverage was obtained on the portal (e.g., healthcare.gov), whether there was a premium, and, if so, whether it was subsidized (Pascale et al. 2013). Answers to questions on conventional coverage type could then be used in tandem with these three new items to categorize coverage type post-ACA (Pascale et al. 2013).

After the ACA was implemented, several surveys, including the Current Population Survey Annual Social and Economic Supplement (CPS-ASEC), the National Health Interview Survey (NHIS), and the Medical Expenditure Panel Survey (MEPS), added the three new items recommended by the Massachusetts study (portal, premium, and subsidy), and the American Community Survey (ACS) added the premium and subsidy items. The specifics on how, exactly, to combine the data points to maximize the accuracy of coverage type categorization hinges on the details of how the pre-ACA question series is asked. In this study, we explore the question of how to best stitch the data points together—that is, what algorithm to use—for the CPS health insurance module. We first developed an algorithm for classifying coverage type that relies solely on a conceptual understanding of the general tendencies of the health insurance landscape (we label this conceptual algorithm “CON”). Then, we developed an alternative algorithm guided by data from an experimental study called Comparing Health Insurance Measurement Error (CHIME) that included responses to the post-ACA redesigned CPS survey matched to administrative records (AR) from a private health insurer (we label this AR algorithm “AR”). We compared the two algorithms by examining three reporting accuracy metrics of coverage type classification—underreporting, overreporting, and prevalence—to assess whether the AR algorithm could improve the accurate categorization of private versus public coverage and to better understand any trade-offs. This is particularly important, given researchers’ and policymakers’ need to produce estimates of public versus private coverage from survey data.

We proceed as follows: first we describe the basic structure of the redesigned CPS health insurance module, how it was adapted after the ACA, and how the CON algorithm was developed. Next, we provide highlights of the CHIME data collection methodology, focusing on the key questions in the CPS health insurance module that capture specific features of the coverage. We then describe the logic and methods used to construct the AR algorithm. In the “Results” section, we present reporting accuracy metrics on both algorithms. We conclude with implications of the findings for surveys beyond the CPS.

The CPS Redesign

The CPS-ASEC health insurance module was redesigned in 2013, following years of criticism stemming from the fact that its measure of the uninsured tracked higher than most other major surveys (Bhandari 2004; Congressional Budget Office 2003). In response to the criticism, a research program got underway in 1999 to identify questionnaire design features driving measurement error, which eventually led to a major redesign of the survey (Pascale 2016; Pascale, Boudreaux, and King 2015). Starting with data for calendar year 2013—prior to implementation of the ACA and the marketplace—the questionnaire redesign was implemented.

A key change in the redesign was the basic flow of the questions. Most surveys, including the traditional CPS, use a series where each question corresponds to a specific coverage type: “Are you covered by Coverage Type X? Are you covered by Coverage Type Y?” and so on. Answers to these individual questions are used to categorize coverage type directly. That is, there is a one-to-one correspondence between the question and the coverage type or “bucket” into which the respondent’s coverage is categorized. The redesigned CPS departs from this and instead uses a funneling approach beginning with general questions, which are followed by detailed questions tailored to the answers to the general questions. Figure 1a displays a simplified version of the basic content and flow of the CPS redesign. Below we describe the questions and flow of the questionnaire in more detail, noting where coverage type categorization is direct and then discussing where an algorithm is needed.

Figure 1.

CPS redesign basic question flow (pre- and post-ACA), conceptual and administrative records algorithms.

The series begins with a question on any coverage at all, and if yes, a question on general source is asked, and each general source has its own set of follow-up questions. First, those who report “job” as the general source are asked if the coverage is related to military service in any way. If no, the coverage is classified as employer-sponsored insurance (ESI); if yes, it goes into the military “bucket.” Second, if general source was direct purchase, the coverage is categorized as nongroup. Third, those who report “government” as general source are asked what type of government coverage. Reports of “military” and “Medicare” are categorized directly into their respective buckets. All other responses (Medicaid, Other, do not know, and refused) are routed to another follow-up question (GovPlan) which asks, verbatim, “What do you call the program?” Response categories include generic names of public programs (e.g., Medicaid, Medical Assistance) as well as all known state-specific names for federal- and state-level public programs. The original intent of the GovPlan question was not to capture coverage type, but rather to collect a respondent-friendly label to help both the respondent and the interviewer keep track of what plan they were referring to in the many follow-up questions on months and other household members covered by the same plan or program (Pascale et al., 2019a).

A year after the CPS redesign was implemented, the ACA marketplace was initiated, and the questionnaire was adapted accordingly. First, the three new survey items noted above—on portal, premiums, and subsidies—were added. Second, the question asking about general source of coverage was modified; the response category for “government” was revised to “government or state” to accommodate respondents who obtained their coverage via the “state” (as in the state-based marketplace) (see Figure 1b). In terms of coverage type categorization, for some types (ESI, military, and Medicare), no change was needed since those were unaffected by the ACA. However, given the ambiguities introduced by the ACA noted above, for public, marketplace, and nongroup coverage, an algorithm is needed, hence the gray oval at the bottom right of the flow chart.

Conceptual “CON” Algorithm

The first algorithm we developed relies on broad brush assumptions and sets aside the ambiguities discussed earlier. The algorithm is similar to the pre-ACA scheme, but it uses two of the post-ACA items to separate marketplace from public and nongroup coverage types. First, for those who report government/state as the general source, we use the premium question. A “yes” goes in the marketplace bucket and a “no” goes in the public bucket. Second, among those who report the general source as direct purchase, we use the item on whether it was obtained on the portal. A “yes” answer goes in the marketplace bucket, while a “no” goes in the nongroup bucket. See Figure 1c. We now provide details of the CHIME data and methodology and then move on to describe the development of the AR algorithm.

Methods

Data and Sample

To develop the AR algorithm and to compare the accuracy of the two algorithms, we use data from the CHIME study. The sample source was AR provided by a large private health insurer in Minnesota. Specifically, phone numbers of households with enrollees were randomly selected from five different coverage types or strata: two types of public coverage (Medicaid and MinnesotaCare, a state-based public plan with a premium contribution) and three types of private coverage (ESI, nongroup coverage obtained outside the marketplace, and marketplace coverage). The sample size in each stratum was based on a power analysis which took into account estimates of measurement error from the existing literature (Call et al. 2013; Davern et al. 2008). As such, ESI was undersampled and public was oversampled. At the time of the sample draw (December 2014), the total population of the health insurer across these strata was about 700,000. Based on estimates of response rates, unusable phone numbers, and opt-outs, a total of 16,000 phone numbers were selected for delivery to the Census Bureau.

All interviews were conducted by Census Bureau telephone interviewers at the Hagerstown, MD, facility from late May until late June 2015. The questionnaire began with a subset of items from the CPS on demographics, labor force, and unearned income, followed by the CPS health insurance module in its entirety. To maximize the utility of the overall CHIME study, there were two experimental arms of the health insurance module—one based on the CPS and the other based on the ACS (not used in this analysis)—and the sample was randomly assigned to the two arms. A household respondent (18 years or older) answered questions about all household members. The average administration time was 17 minutes, and the response rate was 22 percent. Interviewing yielded about 2,700 completed household interviews representing about 6,700 people. Several weeks after completion of data collection, the insurer delivered a second file of AR that contained data on coverage type enrollment on the day of the interview, as well as demographics needed for a person-level match back to the survey data. The survey person-records were matched to their counterpart AR using a combination of variables found on both datasets, such as the unique household ID, sex, date of birth, age, and address. In 87 percent of households, at least one health plan enrollee person-record was matched to the survey data. Because Medicare is out of scope for the study, those over the age of 65 years were dropped, leaving a final matched person-level dataset of about 4,000 cases, about half of which were administered the CPS questionnaire. The data were weighted to the health insurer’s population at the time of the sample draw. More details on the CHIME methodology can be found in Fertig et al. (2018). The final sample for this analysis is about 2,000 individuals with known health insurance coverage.

Administrative Records “AR” Algorithm

Construction of the AR Algorithm

To create the AR algorithm, we first identified common patterns of responses to questions on coverage type and attributes, and then examined the distribution of coverage type according to the records within each pattern to assign coverage type. More specifically, we began with the five relevant questions on features of health insurance coverage that had some potential for guiding categorization. These were two items from the pre-ACA questionnaire—on type of government coverage and program name—along with the three items post-ACA (on portal, premium, and subsidies). See Figure 2. Our first task was data reduction. We began by grouping response categories where the substance of the response was similar (e.g., for the question “What do you call the program?” four distinct responses—Medicaid, Medical Assistance, MinnesotaCare, and Prepaid Medical Assistance Program—all represent public coverage). Before simply collapsing responses, however, we examined the distribution of coverage type for each response category independently to ensure no loss of information. For example, if the records indicated those who chose Medicaid happened to be comprising 90 percent public enrollees and 10 percent marketplace enrollees, while those who chose MinnesotaCare were comprising, say, 30 percent public enrollees and 70 percent marketplace enrollees, we would not collapse them. In this particular case, regardless of which of the four public program names respondents chose, the distribution of enrollment according to the AR was highly skewed toward public and consistent across all four, so they were collapsed. Figure 3 displays the verbatim wording of both the question and all response categories, along with the collapsed response categories.

Figure 2.

Subset of items in the CPS health insurance module.

Figure 3.

Question text and response categories (original and collapsed).

We then created a list of all mathematically possible permutations of responses from the five items with collapsed response categories, separately for the “GovPath” (respondents who reported “Government/State” as the general source) and for the “DirPath” (those who reported “Direct Purchase” as the general source). In total, the result was 30 unique permutations in the GovPath and 6 in the DirPath. We then examined enrollment distributions within each of the 36 permutations and conducted another round of data reduction. In general, we maintained each permutation for each combination of the GovType and GovPlan items. But within these permutations, in many cases, regardless of the response to the portal and subsidy items, the enrollment distribution was very similar. The result of the second round of data reduction was a total of 18 unique permutations—13 in GovPath and 5 in DirPath.

Table 1 displays these permutations, where the upper panel displays GovPath and the lower panel displays DirPath details. Column (1) provides the arbitrary label assigned to each (e.g., Gov1 in the GovPath). The next set of Column (2) displays survey responses to the five items on characteristics of coverage, based on the collapsed response categories shown in Figure 3. Column (3) provides the weighted prevalence of survey respondents in each permutation. In the middle panel, the first set of column (4) shows the weighted coverage type distribution according to the AR within a permutation. It was these distributions that aided in collapsing the permutations. For example, Gov7 represents respondents who reported “public” at both GovType and GovPlan and that they paid a premium. AR indicated that when this combination of responses was reported for these three items, the vast majority of enrollees were public, regardless of their answers to the portal and subsidy items. Hence for the portal and subsidy survey items, the response of “any” is indicated for those cells for Gov7. Finally, Column (5) displays the coverage type assigned to each permutation, for both the AR and CON algorithms.

Table 1.

Permutations of Survey Reporting Patterns, Their Enrollment Distributions, and Coverage Type Assignments.

SURVEY DATA							ADMINISTRATIVE RECORD DATA					(5) Coverage type assignment
(1) Path	(2) Responses to survey questions					(3) Prevalence	(4) Coverage type according to records					(5) Coverage type assignment
(1) Path	GovType	GovPlan	Portal	Premium	Subsidy	Wtd %	ESI	NonG	Mkt	Pub	Total	CON	AR
GovPath						100
Gov1	Public	Public	Any/n/a	No	n/a	36.4	3.3	0.2	0.1	96.6	100	Pub	Pub
Gov2	Other	Public				26.3	0.1	1.1	0.1	98.8	100
Gov3	Public	Other				6.7	0.5	0	0	99.5	100
Gov4	Public	Market				1.6	5.3	3.5	0	96.5	105
Gov5	Other	Other				12.2	0.5	0.4	0	99.4	100
Gov6	Other	Market				9.2	0	0.3	0.3	99.5	100
Gov7	Public	Public		Yes	Any	0.8	0	4.3	2.7	93.0	100	Mkt
Gov8	Other	Public				3.9	0	1.1	1.4	97.5	100
Gov9	Other	Other				1.2	0	16.8	6.7	76.5	100
Gov10	Other	Market			Yes	1.6	48.2	5.0	14.5	32.4	100		Mkt
Gov11	Public				Yes	0.01	0	0	100	0	100
Gov12	Other				No	0.05	0	50.7	49.3	0	100
Gov13	Public	Other			No	0.02	0	100	0	0	100
DirPath					100
Dir1	n/a		No	Yes	Any	66.1	18.0	79.5	0.5	2.0	100	Non-Group	Non-Group
Dir2			No	No	n/a	20.1	85.9	6.1	0	8.1	100	Non-Group	Non-Group
Dir3			Yes	Yes	Yes	5.5	0	31.6	22.1	46.3	100	Mkt	Mkt
Dir4				Yes	No	7.0	56.4	15.0	9.1	19.5	100
Dir5				No	n/a	1.3	0	0	0	100	100

Note. The distribution of coverage type according to the records (Column 4) is shown separately for GovPath and DirPath because some respondents could have reported plans in both paths. For all but one permutation (Gov4), the distribution of coverage types adds up to 100. This was by design, because respondents were expected to be covered by only one type of coverage at a time. However, in some rare instances, an enrollee was found to be covered by more than one plan in the records. This was the case for Gov4, and the relatively low number of unweighted cases, combined with the high ESI weight, resulted in the distribution for this permutation adding up to more than 100. CON = Conceptual algorithm; AR = Administrative Records algorithm; n/a = not applicable. ESI = employer-sponsored insurance; NonG = Non-Group; Mkt = Marketplace; Pub = Public; Wtd = weighted.

Table 1 demonstrates that, for Gov1 through Gov8, records indicate that 93.0 to 99.5 percent of respondents with those particular reporting patterns were public enrollees, according to the records (see Column [4] Pub). These eight permutations combined represent 97.1 percent of the total weighted sample in the GovPath (see Column [3]). AR for the Gov9 permutation (representing 1.2 percent of cases in the GovPath) was also fairly robust, with records indicating that 76.5 percent were public enrollees. Thus, for 98.3 percent of the GovPath sample, assignment to public coverage is well supported by the AR. In the DirPath, for one of the five permutations—Dir1, representing 66.1 percent of the weighted DirPath sample—the AR indicated that almost 80 percent were nongroup enrollees (see Column [4] NonG).

For the remaining permutations, the AR were inconclusive due to the sparse but heavily weighted ESI cases, small cell sizes, and/or the fact that enrollees did not cluster into any one type of coverage according to the records. For all these permutations where the AR could not be used to guide categorization, we reverted to coverage type assignments according to the conceptual algorithm. On this first issue, we expected few, if any, ESI enrollees to mistakenly report their general source of coverage as Government/State or Direct Purchase, instead of Job. Indeed, there were few, but there were some. ESI is by far the most dominant source of coverage overall, and we deliberately undersampled ESI in our sampling scheme. The combination of these two factors resulted in ESI cases receiving an exceptionally large weight compared to the other strata, causing some enrollment coverage type distributions to be somewhat artificially dominated by a very small number of unweighted ESI cases. In the DirPath, 85.9 percent in Dir2 and 56.4 percent in Dir4 were actually ESI enrollees, and in the GovPath, in Gov10, nearly half of the respondents were actually enrolled in ESI according to the records. Regarding small cell sizes, Gov11, Gov12, and Gov13 all represented vanishingly small proportions of the weighted sample (less than 0.06 each), meaning that a single case could skew results considerably. Finally, for some permutations, the AR enrollment distribution was too mixed to be informative; for Gov12, records were about evenly split between nongroup and marketplace, and for Dir3, there was no majority across nongroup, marketplace, and public AR.

Description of the AR Algorithm

The final AR algorithm is shown in Figure 1d. Ultimately, the difference between the two algorithms is that among those in GovPath, the AR algorithm puts more respondents (Gov7, Gov8, and Gov9) into the public than the marketplace bucket, relative to the CON algorithm. The AR algorithm does this by leveraging the GovPlan item. The algorithms are identical up until the GovType item, and the distinction between algorithms comes into play among those who say “public” or “other” to GovType. Among that universe of cases, the CON algorithm uses only one data point: Premium. If “yes,” the case goes into the marketplace bucket, and if “no,” it goes into the public bucket. Among that same universe of cases, the AR algorithm uses two data points—Premium and GovPlan—to basically intercept some cases and place them in the public bucket before the premium item is considered. The cases that are intercepted are those whose combined answers to GovType and GovPlan do not mention marketplace. In other words, respondents who answer a mix and match of “public” and “other” to the GovType and GovPlan items all go into the public bucket, but if GovPlan is “market,” the cases are routed to the premium item, where a “yes” goes into the marketplace bucket and a “no” goes into the public bucket. Note there is one exception to this general rule, and it is based on a single permutation (Gov13) that had such a small prevalence (0.02 percent of weighted cases) that without guidance from the AR we reverted to the more simplistic CON assignment.

Measures to Evaluate Reporting Accuracy

We use three different reporting accuracy measures to compare the algorithms. First, we calculate the percent of respondents known to have coverage type X according to the records for whom coverage type X is reported in the survey, producing a measure of underreporting. Second, we calculate the percent of respondents for whom coverage type X is reported who are validated to actually have coverage type X in the AR, rendering a measure of overreporting. The study design limits this metric somewhat; if a survey report of coverage type X cannot be validated in our AR, we cannot rule out the possibility that the survey report is correct via coverage from a different insurer. For our third measure, we compare the prevalence of coverage type X in the AR to the prevalence based on each algorithm’s categorization.

In general, if an algorithm defines coverage type too broadly, then many with that coverage type will be captured, leading to low underreporting, but many of those captured will not actually have the coverage, leading to high overreporting. If coverage is too narrowly defined, then many with that coverage will be missed, leading to high underreporting, but many of those captured will be validated in the records, leading to low overreporting. Thus, we assessed both underreporting and overreporting and, given the prospects that those two errors can cancel each other out, we also examined the prevalence in the population as indicated by the records, compared to the survey estimates produced by the two algorithms.

Results

Table 2 displays results for all three reporting accuracy metrics for both algorithms. The first column shows coverage type—both disaggregated and aggregated categories. Note that the CON and AR algorithms are identical for ESI and nongroup coverage, as well as overall insurance coverage. We show the aggregated categories to allow the reader to see the combined effects of the accuracy metric and prevalence of an individual coverage type relative to the other types within the aggregated category. For example, ESI by far dominates the coverage landscape and its accuracy metrics are high, while marketplace coverage is rare and its accuracy metrics are low. Thus, the aggregated private category, which is a combination of ESI, nongroup, and marketplace coverage, generally demonstrates how the low accuracy metrics of marketplace coverage are muted by the high prevalence and high accuracy metrics of ESI.

Table 2.

Contrasting Reporting Accuracy Metrics for Conceptual (CON) and Administrative Records (AR) Algorithms.

	Underreporting			Overreporting			Prevalence of coverage type
Coverage type	CON	AR	CON-AR	CON	AR	CON-AR	RECS	CON	AR	CON-RECS	AR-RECS	CON-AR
Public	21.7	17.0	4.7**	1.9	2.2	-0.3	28.9	23.1	24.6	-5.8**	-4.4**	-1.5**
Private	1.0	1.1	-0.2	4.4	2.6	1.8*	71.2	73.7	72.3	2.6**	1.1**	1.5**
ESI	1.7	1.7	n/a	2.9	2.9	n/a	67.3	68.2	68.2	0.9*	0.9*	n/a
NongMkt	19.7	22.5	-2.8**	54.6	44.1	10.6**	3.8	6.8	5.3	2.9**	1.5**	1.5**
Non-group	26.0	26.0	n/a	37.6	37.6	n/a	3.6	4.2	4.2	0.7	0.7	n/a
Market	29.4	43.3	-13.9**	92.3	85.6	6.7*	0.3	2.6	1.1	2.3**	0.8**	1.5**
Insured	2.1	2.1	n/a	n/a	n/a	n/a	100	97.9	97.9	2.1**	2.1**	n/a

Note. Figures shown are rounded to the first decimal. However, differences were calculated based on figures rounded to the second decimal, which accounts for slight variation in the difference columns of the table. For underreporting and overreporting, lower values in the CON and AR columns indicate more concordance with the administrative records (i.e.: higher accuracy). For prevalence, lower values in the CON-RECS and AR-RECS columns indicate more concordance with the administrative records. RECS = Enrollment Records; n/a = not applicable. ESI = employer-sponsored insurance; Nong = Non-group; kt/Market = Marketplace.

p < .05. **p < .01.

The left panel of Table 2 shows the level of underreporting for each algorithm, the difference between the two, and the results of statistical t tests of the difference. Underreporting of public coverage was higher in CON than in AR by almost five percentage points (17.0 versus 21.7 percent).

For the aggregate private category, underreporting was low and the difference between algorithms was trivial (1.0 versus 1.1 percent) and not statistically significant. Among disaggregated private coverage types, for ESI and nongroup, again we note that the logic across algorithms was identical so there was no difference across algorithms, but we highlight that levels of underreporting of ESI were much lower (1.7 percent) than for nongroup (26 percent). Marketplace underreporting was higher in AR than in CON, by almost 14 percentage points, and this difference persisted in the aggregated nongroup/marketplace category, but the effects were muted due to the higher prevalence of nongroup relative to marketplace, making for a difference of about 3 percentage points (22.5 versus 19.7 percent).

Moving on to the middle panel of Table 2, overreporting of public coverage was low, and the difference between algorithms was trivial and nonsignificant (1.9 versus 2.2 percent). For private coverage, overreporting was higher in CON than in AR, by almost 2 percentage points (4.4 versus 2.6 percent). As with underreporting, the levels of overreporting for nongroup were much higher than those for ESI (37.6 versus 2.9 percent). For marketplace, overreporting was excessively high in both algorithms, but it was almost seven percentage points higher in CON than in AR (85.6 versus 92.3 percent). This difference drove differences in the aggregated nongroup/marketplace category, where overreporting was almost 11 percentage points higher in CON than in AR (54.6 versus 44.1 percent).

The right panel of Table 2 displays the prevalence according to the records (which we label “RECS”) and the estimates of prevalence from both algorithms. For public coverage, the RECS prevalence was 28.9 percent and both CON and AR estimates fell short of that prevalence, by 5.8 and 4.4 percentage points, respectively. However, the AR estimate was closer to the RECS prevalence than the CON estimate, by 1.5 percentage points, which was a significant difference. For private coverage, the RECS prevalence was 71.2 percent and both CON and AR estimates were higher, by 2.6 and 1.1 percentage points, respectively. However, again, the AR estimate was closer to the RECS prevalence than the CON estimate, by 1.5 percentage points, which was a significant difference. For ESI and nongroup coverage, the survey estimates were both slightly higher than the RECS prevalence, but both were within a percentage point (0.9 for ESI and 0.7 for nongroup). For marketplace, the RECS prevalence was very low (0.3 percent), and both algorithms’ estimates were higher, but the CON was much higher than the AR estimate (2.6 versus 1.1 percent), for a 1.5 percentage point difference that was statistically significant. That difference drove results for the aggregated nongroup/marketplace category, where RECS prevalence was 3.8 percent, and both algorithms produced higher estimates, at 6.8 and 5.3 percent (CON and AR, respectively). Again, the difference between algorithms’ estimates (1.5 percentage points) was statistically significant.

Limitations

There are several limitations to this study. First, findings are based on one state, and it is an atypical state in some ways. The Minnesota population is predominately White (83 percent) and well educated (over 93 percent have a high school diploma and about one third have a college degree), and has high income, low unemployment, and low poverty relative to other states (U.S. Census Bureau, 2024). Second, the analysis is based on one health insurer, and there is reason to be cautious about generalizing to all health insurers, particularly for marketplace coverage. The health insurer in this study had a relatively low market share in marketplace coverage, and their marketplace plans had higher premiums than most plans offered in the state during the study year. Thus, it is possible that CHIME participants in the marketplace strata are more educated and financially secure than those in the marketplace population overall, and that these characteristics impact reporting accuracy. However, a separate CHIME analysis found that reporting accuracy was not associated with sociodemographic characteristics for those enrolled in private nongroup and marketplace plans (Call et al. 2022), suggesting low likelihood of bias resulting from this limitation. Third, the study addressed only the effects of the questionnaire on the estimates, but the final published CPS estimates are the product of imputation, weighting, mode, edits, and other factors.

Discussion and Summary

Given the ambiguities of the post-ACA health insurance landscape, for some patterns of self-reported data points, it is literally impossible to classify respondents into one and only one coverage type bucket. We examined the records to see if enrollees themselves happened to sort themselves into one bucket or the other with their patterns of response to key survey items, thereby suggesting an algorithm that minimized coverage type classification error.

The key difference between the two algorithms is that the CON algorithm defined marketplace coverage more broadly than the AR algorithm. In the AR algorithm, we zero in on particular permutations where the CON scheme classified respondents as marketplace when the AR indicated that the vast majority of cases were actually public enrollees. More specifically, both algorithms rely on the premium item, and all who say “no” go into the public bucket. Among those who say “yes” to the premium item, the CON algorithm classifies all seven permutations as marketplace, while the AR does not. It separates this pool of respondents and skims off three permutations (Gov7–Gov9) to classify them as public because the AR indicate that 76.5 to 97.5 percent of these cases are actually public enrollees. These three permutations represent about 6 percent of the GovPath sample, which was enough to drive important results.

We found that the AR algorithm produced measurable improvements over the CON algorithm, with only one minor trade-off. Underreporting of public coverage was reduced by about 5 percentage points, and overreporting of private, marketplace, and nongroup/marketplace coverage was reduced (by almost 2 percentage points; almost 7 percentage points; and almost 11 percentage points, respectively). In terms of prevalence, for public coverage, the AR algorithm produced estimates that were closer to the AR point estimate by a statistically significant 1.5 percentage point for all coverage types—public, private, marketplace, and nongroup/marketplace coverage. The cost of these improved metrics was higher underreporting of marketplace and nongroup/marketplace coverage (by almost 14 percentage points and 3 percentage points, respectively). One primary motivator for this research project was to address the chronic underreporting of public coverage (Call et al. 2013) and slight overreporting of private coverage (Abraham, Karaca-Mandic, and Boudreaux 2013; Boudreaux et al. 2014; Mach and O’Hara 2011). Given that context, and the relatively low prevalence of marketplace coverage, we suggest that underreporting of marketplace coverage is a small price to pay for the other improved metrics offered by the AR algorithm.

Regarding applications to production CPS, in 2023, the Census Bureau announced an effort to modernize both the core CPS and the ASEC (which includes the health insurance module). Plans include an initiative to add an internet self-response mode to the data collection operation and to reprogram existing modes into the new CPS enterprise data collection solution (U.S. Census Bureau 2023). This initiative creates new opportunities to embed the AR algorithm into the processing system for production data, and to make the questionnaire more efficient. Specifically, we found the subsidy question did not aid in coverage type classification and could be dropped; the portal question was useful for the DirPath but not the GovPath; and the premium item was useful for the GovPath but not the DirPath. Thus, dropping the portal and premium items within these respective paths could reduce respondent burden. We emphasize, however, that the utility of the premium, portal, and subsidy items is intertwined with the nuances of questions in the core module on coverage types, so these particular suggestions are specific to the CPS-ASEC.

While the CPS health module is unique in its structure, results hold promise of generalizing to other surveys. Our analysis indicates that for purposes of coverage type classification, using the premium item in tandem with survey responses to questions on program or plan type can avoid misclassifying some public enrollees as marketplace. Specifically, in the CPS question routine, the GovPlan item includes a spectrum of response options—from clearly public (e.g., Medicaid) to private-leaning (e.g., marketplace) and several options in-between (e.g., other, do not know, and refused). Skimming off only the private-leaning responses and using those in tandem with a “yes” response to the premium item resulted in improved coverage type categorization of public, marketplace, and private coverage.

Health insurance modules of several other major federal government surveys now include key data points that enable the application of a version of the AR algorithm produced by this analysis. For example, in the NHIS, among those who report coverage, there is a simple laundry-list question on coverage type with typical response options (e.g., private health insurance, Medicaid). Respondents are then asked for the “complete name of the plan” with an open-text response. Similarly, in the MEPS, respondents who report “any other type of health insurance from any state or local government agency” are asked an open-text question on the program name. If these responses could be coded as leaning more toward private and away from public descriptions, this could produce a data point equivalent to the GovPlan item in the CPS, which could be used in tandem with the premium item (nearly verbatim to the CPS item in both surveys) to help separate public from marketplace coverage. This could reduce underreporting of public coverage and overreporting of private coverage for these federal surveys, which is a very high priority for the research and policy communities.

With regard to the ACS, a recent content test compared two versions of the health insurance module where one condition included the terms “Marketplace” and “healthcare.gov” in the item on directly purchased coverage, and the other condition did not. The version that included “Marketplace” and “healthcare.gov” in the direct purchase item resulted in significantly lower reporting of Medicaid (15.5 versus 17.1 percent) (Hernandez-Viver et al. 2023). This suggests some Medicaid enrollees who obtained their coverage on the marketplace may be reporting the coverage as directly purchased rather than Medicaid. As noted above, a second arm of the CHIME study included the ACS health insurance module, along with the additional items on premium and subsidies (Pascale et al., 2019b). The ACS also includes an “other/specify” category with open-text write-ins, which could be coded as public versus private-leaning and used in a similar manner as the GovPlan item was used in the AR algorithm for the CPS. Future research includes exploiting the AR in the CHIME study to examine how best to combine answers to the laundry-list questions on coverage type with the premium and subsidy items, along with the open-text write-ins, to guide coverage type categorization in the ACS.

In sum, all health insurance modules have been plagued for decades by underreporting of public coverage. The introduction of the marketplace, with all its ambiguities, has made it more difficult to separate public from private, and it seems to have exacerbated underreporting of public coverage. In this research, we exploited AR to guide an algorithm that allowed us to carefully combine existing data points in the CPS to improve classification of both public and private coverage compared to a more simplistic algorithm. Other major federal surveys contain data points akin to those in the CPS, paving the way for the development of a version of the AR algorithm for other surveys to improve coverage type classification.

Supplemental Material

sj-docx-1-jax-10.1177_19367244241245953 – Supplemental material for Categorization of Health Insurance Coverage Type from Survey Questions after Health Reform: The Case of the Current Population Survey

Supplemental material, sj-docx-1-jax-10.1177_19367244241245953 for Categorization of Health Insurance Coverage Type from Survey Questions after Health Reform: The Case of the Current Population Survey by Joanne Pascale, Angela R. Fertig and Kathleen T. Call in Journal of Applied Social Science

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for research, authorship, and/or publication of this article: This research is supported by the U.S. Census Bureau, the State Health Access Data Assistance Center (SHADAC), Medica Research Institute, the U.S. Department of Health and Human Services Office of the Assistant Secretary for Planning and Evaluation (ASPE), and the Robert Wood Johnson Foundation. The content is solely the responsibility of the authors and does not represent the official views of these organizations. This report is released to inform interested parties of research and to encourage discussion. The views expressed are those of the authors and not those of the U.S. Census Bureau. The U.S. Census Bureau reviewed this data product for unauthorized disclosure of confidential information and approved the disclosure avoidance practices applied to this release (DMS #P-600673; CBDRB-FY23-CBSM003-001).

ORCID iDs

Joanne Pascale

Angela R. Fertig

Supplemental Material

The complete questionnaire, including verbatim question wording and skip patterns, is available online.

Author Biographies

Joanne Pascale is a senior researcher in the Center for Behavioral Science Methods at the U.S. Census Bureau. She specializes in questionnaire design and evaluation, record linkage, health insurance, disclosure avoidance, and coverage issues.

Angela R. Fertig is an economist whose primary area of research is examining policies affecting the social determinants of health. She has methodological expertise in applied econometrics, survey methods, and ecological momentary assessment data analysis.

Kathleen T. Call is a professor in the University of Minnesota, School of Public Health’s Division of Health Policy and Management. Her research focuses on health insurance measurement and how well health insurance translates into equitable access to health care, exploring barriers to care including health care discrimination.

References

Abraham

J. M.

Karaca-Mandic

Boudreaux

. 2013. “Sizing Up the Individual Market for Health Insurance: A Comparison of Survey and Administrative Data Sources.” Medical Care Research and Review 70:418–33.

Bhandari

2004. “People with Health Insurance: A Comparison of Estimates from Two Surveys.” Survey of Income and Program Participation (SIPP) Working Paper No. 243, U.S. Census Bureau, Washington, DC. Retrieved February 2, 2016 https://www.census.gov/content/dam/Census/library/working-papers/2004/demo/SEHSD-2004-02.pdf.

Boudreaux

Call

K. T.

Turner

Fried

. 2014. “Estimates of Direct Purchase from the ACS and Medicaid Misreporting: Is There a Link?” SHADAC Brief #38, State Health Access Data Assistance Center, Minneapolis, MN.

Call

K. T.

Davern

M. E.

Klerman

J. A.

Lynch

. 2013. “Comparing Errors in Medicaid Reporting Across Surveys: Evidence to Date.” Health Services Research. 48:652–64. doi:10.1111/j.1475-6773.2012.01446.x.

Call

K. T.

Fertig

A. R.

Pascale

. 2022. “Factors Associated with Accurate and Inaccurate Self-reported Reporting of Public and Private Health Insurance Type Coverage.” Health Services Research. 57: 930–943. doi:10.1111/1475-6773.13874

Cantor

J. C.

Monheit

A. C.

Brownlee

Schneider

. 2007. “The Adequacy of Household Survey Data for Evaluating the Nongroup Health Insurance Market.” Health Services Research 42(4):1739–57. doi:10.1111/j.1475-6773.2006.00662.x.

Congressional Budget Office. 2003. “How Many People Lack Health Insurance and for How Long? A CBO Report.” The Congress of the United States. Retrieved February 2, 2016 (https://www.cbo.gov/publication/14426.

Davern

M. E.

Call

K. T.

Ziegenfuss

Davidson

Beebe

T. J.

Blewett

L. A.

. 2008. “Validating Health Insurance Coverage Survey Estimates: A Comparison Between Self-reported Coverage and Administrative Data Records.” Public Opinion Quarterly 72:241–59.

Fertig

A. R.

Pascale

Call

K. T.

Oellerich

. 2018. “Design and Sampling Strategy for a Validation Study Linking Enrollment Records to Survey Reports: The CHIME Study.” Retrieved April 9, 2024.https://www.census.gov/library/working-papers/2018/adrm/rsm2018-10.html

10.

Hernandez-Viver

Adriana

Keisler-Starkey

Katherine

Stern

Sharon M.

Spiers

Samantha

Massarone

Peter

. 2023. “2022 American Community Survey Content Test Evaluation Report: Health Insurance Coverage Final Report.” U.S. Census Bureau. Retrieved February 6, 2024 (https://www.census.gov/content/dam/Census/library/working-papers/2023/acs/2023_Hernandez-Viver_01.pdf.

11.

Loomis

2000. “Report on Cognitive Interview Research Results for Questions on Welfare Reform Benefits and Government Health Insurance for the March 2001 Income Supplement to the CPS.” Center for Survey Methods Research, Statistical Research Division, U.S. Census Bureau, Washington, DC.

12.

Mach

O’Hara

. 2011. “Do People Really Have Multiple Health Insurance Plans? Estimates of Nongroup Health Insurance in the American Community Survey.” Retrieved April 9, 2024. https://www.census.gov/content/dam/Census/library/working-papers/2011/demo/SEHSD-WP2011-28.pdf

13.

Nelson

D. E.

Thompson

B. L.

Davenport

N. J.

Penaloza

L. J.

. 2000. “What People Really Know About Their Health Insurance: A Comparison of Information Obtained from Individuals and Their Insurers.” American Journal of Public Health 90:924–28.

14.

Noon

J. M.

Fernandez

L. E.

Porter

S. R.

. 2019. “Response Error and the Medicaid Undercount in the Current Population Survey.” Health Services Research 54:34–43. doi:10.1111/1475-6773.13058

15.

Pascale

2008. “Measurement Error in Health Insurance Reporting.” Inquiry: A Journal of Medical Care Organization, Provision and Financing 45(4):422–37. doi:10.5034/inquiryjrnl_45.04.422.

16.

Pascale

2016. “Modernizing a Major Federal Government Survey: A Review of the Redesign of the Current Population Survey Health Insurance Questions.” Journal of Official Statistics 32:461–86. doi:10.1515/JOS-2016-0024.

17.

Pascale

Boudreaux

King

. 2015. “Understanding the New Current Population Survey Health Insurance Questions.” Health Services Research 51:240–61. doi:10.1111/1475-6773.12312.

18.

Pascale

Fertig

A. R.

Call

K. T.

. 2019a. “Assessing the Accuracy of Survey Reports of Health Insurance Coverage Using Enrollment Data.” Health Services Research 54:1099–1109. doi:10.1111/1475-6773.13191.

19.

Pascale

Fertig

A. R.

Call

K. T.

. 2019b. “Validation of Two Federal Health Insurance Survey Modules after Affordable Care Act Implementation.” Journal of Official Statistics 35:409–60. doi:10.2478/JOS-2019-0019.

20.

Pascale

Rodean

Leeman

Cosenza

Schoua-Glusberg

. 2013. “Preparing to Measure Health Coverage in Federal Surveys Post-reform: Lessons from Massachusetts.” Inquiry: A Journal of Medical Care Organization, Provision and Financing 50:106–23. doi:10.1177/0046958013513679.

21.

U.S. Census Bureau. 2023. “Current Population Survey (CPS) 2023 Modernization Efforts.” Retrieved January 21, 2024 (https://www.census.gov/programs-surveys/cps/about/modernization.html.

22.

U.S. Census Bureau. 2024. “QuickFacts Minnesota.” QuickFacts United States. Retrieved April 2024. https://www.census.gov/quickfacts/fact/table/MN/PST045223; https://www.census.gov/quickfacts/fact/table/US/PST045223

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.08 MB