Extending current population survey linkages: Obstacles and solutions for linking monthly data from 1976 to 1988

Abstract

The Current Population Survey (CPS) has been the nation’s primary source of information about employment and unemployment for decades. The data are widely used by social scientists and policy makers to study labor force participation, poverty, and other high-priority topics. An underutilized feature of the CPS is its short-run panel component. This paper discusses the unique challenges encountered when linking basic monthly data as well as when linking the March basic monthly data to the Annual Social and Economic (ASEC) Supplement in the 1976–1988 period. We describe strategies to address linking obstacles and document linkage rates.

Keywords

Data integration linking panel data current population survey

1. Introduction

The Current Population Survey (CPS) is the primary source of information about employment and unemployment in the United States. It has been a key data resource for the social science research and policy making community for decades, providing monthly snapshots of the civilian labor force, with microdata available since 1976. The survey has a short-run panel component that is largely unknown and underutilized by the research community. CPS respondents participate in the survey eight times, answering surveys for four consecutive months, rotating out of the survey for eight months, then answering for four more consecutive months. Linked CPS data present numerous opportunities for charting and understanding dynamics of short-run change over the last half century. Examples of research that could be conducted using linked CPS data include analyzing the demographic and employment correlates of families transitioning in and out of poverty; examining the extent to which engagement in volunteering changes following transitions out of employment among older adults; and studying how families’ employment arrangements are stable or change following the birth of a child. Other investigations might focus on how individuals organize their work and family lives in response to recessions, the effects of policy changes on employment, or the labor force participation of veterans from different wars.

IPUMS1 (ipums.org) is a leading disseminator of CPS data, streamlining and simplifying access to these vital social, economic, and demographic data [1]. IPUMS CPS (cps.ipums.org) delivers Annual Social and Economic (ASEC) Supplement data from 1962 to present and basic monthly survey (BMS) data from 1976 to present along with nearly all topical supplement data. IPUMS codes variables consistently across time, provides access to original unrecoded versions of the variables, documents all variables for ease of use, and enables users to create customized datasets that include only variables from the months and years of data they want to analyze. This effort dramatically reduces redundant effort across researchers.

Previously we have documented linkages across months from 1989 forward [2] and between the March BMS and the ASEC [3]. These linkages can be replicated using the IPUMS-constructed linking keys CPSIDP and MARBASECIDP, respectively; these linking keys are available via IPUMS CPS. While researchers could use original keys to link CPS files over time, IPUMS-constructed linking keys and accompanying documentation provide the research community with a common starting point, save individual researchers hours of effort, and reduce potential for error in the linking process. In this paper, we document the specific challenges linking individuals across months and between the ASEC and March BMS in the 1976 to 1988 period. We outline the solutions we implemented and provide linkage rates against which researchers can check their linkages.

2. CPS rotation pattern and linking

The CPS is a rotating panel household-level survey in which all individuals residing in a household are surveyed for four months, rotate out of the survey for eight months, and then rotate back into the survey for another four months (known as the 4-8-4 rotation pattern). The first time a household appears in the CPS is their first “month-in-sample” (MIS)2 and this is indicated in the data (MIS coded as 1). Each subsequent month that a household is included in the CPS, their MIS value increments by one. A household’s MIS value does not increment in months when the household is out of rotation (the eight-month break). Thus, the final month of the 4-8-4 rotation pattern in the CPS has an MIS value of eight.

The Census Bureau provides a very brief set of guidelines for linking individual observations across CPS data files [4], which are insufficient for creating linkages. While they list the variables needed to link observations over time, the documentation is sparse and provides insufficient detail about how to do the linking. Furthermore, the instructions focus on linking within a set of years when linking keys are stable and provide limited direction on how to bridge changes in linking keys over time. Finally, the documentation [5] also indicates that it is not possible to link some years of data together due to changes in the survey (i.e., 1976 to 1977 and 1985 to 1986).

3. CPSID(P)

To overcome barriers related to the rotation pattern and limited guidance for linking CPS data over time, IPUMS created CPSID and CPSIDP. CPSID is a linking key that accounts for changes in original CPS linking keys and consistently identifies households across time. CPSIDP is a person-level linking key that consistently identifies persons across time. CPSID(P)3 is created by applying Census Bureau rules for linking across time. CPSID(P) eliminates the need for every researcher to navigate the complex CPS rotation pattern, changes in linking keys over time, and additional data quality challenges that arise. CPSID(P) provides a common starting point for the research community, which should increase reproducibility of research using linked CPS data.

The CPS is a household survey, which has implications for the creation of CPSID(P). A unique CPSIDP value, based on original linking keys, is assigned to a single individual each time they appear in the CPS. If one or more of the individuals in the household move out of the household between CPS interviews, individuals who have moved are not followed. If everyone in the household moves, and new people occupy the dwelling, the new people are interviewed the next time the household is included in the CPS. In 1976 to 1988, a household is identified using a household identification number (HRHHID) and a household number (HUHHNUM).4 If an entire household is replaced, the household number (HUHHNUM) value changes. Under these circumstances, IPUMS CPS generates a new CPSID value for the new household [2]. For example, a household that was first observed in the CPS in December of 1981 may have the following CPSID value 19811203287400; the two individuals living in the household would have CPSIDP values of 19811203287401 and 19811203287402, respectively. A third individual who joins the household in January of 1982 would be assigned a CPSIDP value of 19811203287403; the two returning household members would be assigned the CPSDIP values they were first assigned in December.5

We encountered several problems when linking CPS respondents in the 1976 to 1988 period. Broadly, these included issues with 1) inconsistent household identifiers, and 2) duplicated person identifiers within households. We describe these problems in detail along with solutions we employed to extend the IPUMS CPS variables CPSID(P) and MARBASECID(P) to basic monthly and ASEC data in the 1976 to 1988 period.

4. Creating CPSID(P) for basic monthly files

Each BMS data file contains only individuals interviewed in that specific month. In some months, topical supplement data are also collected. Regardless of whether topical supplement data are collected, a file containing only the basic monthly variables (hereafter referred to as the basic monthly-only file) is always released. In months where a topical supplement is also fielded, a second file (the supplement-containing file) is released later that includes the topical supplement variables appended to the end of the records. The basic monthly portions of the basic monthly-only and supplement-containing files should be identical, though this is not always the case in the 1976–1988 period. We detail the differences we encounter and our efforts to reconcile differences between the basic monthly-only and supplement-containing files in Appendix A. We also encounter challenges linking across months in the 1976 to 1988. The two major obstacles to linking monthly files are inconsistent household identifiers and duplicated person identifiers.

4.1 Household identifiers and linking across months

To address problems with the household identifiers that affected linking across months between 1976 and 1988, we made modifications to the original household identifier prior to creating CPSID(P). HHID refers to the original, unmodified household identifier and HRHHID refers to the IPUMS CPS-modified household identifier used for linking. Except in specified months, HHID and HRHHID are the same.

4.1.1 Non-unique household identifiers

In 1976 and 1977, the original household identifier (HHID) does not uniquely identify all households in basic monthly-only files. Through trial and error, we located a set of variables that together uniquely identify households within and across months. We used these to create HRHHID for the basic monthly-only files as follows: we used the first nine digits of the twelve-digit HHID and replaced the last three digits of HHID with information from the first two digits of the third and sixth sets of columns marked as “blank” in the original CPS codebook (i.e., BLANK3 and BLANK6).6

In 1976 and 1977, HRHHID, constructed as just described, yields more plausible households within basic-monthly only files than the original household identifier (HHID). Specifically, households identified using HHID were extremely large and contained multiple household heads; using HRHHID results in smaller households with only one head per household. HHID uniquely identifies households in the supplement-containing files; accordingly, we gauge the quality of HRHHID in the basic monthly-only files by making comparisons with the supplement-containing files from the same month (in months that contain supplements). In 1976, HRHHID from the basic monthly-only files matched HHID in the supplement-containing files for 75% of individuals. For the remaining quarter of the records, the final three digits of HRHHID (which correspond to BLANK3 and BLANK6) matched across files, but the first nine digits of HRHHID (which correspond to the first nine digits of HHID) did not match. In these instances, the first nine digits differ systematically between the basic monthly-only and supplement-containing files, so we made a second adjustment to HRHHID (see Appendix A). In 1977, neither HHID nor HRHHID yields successful matches between basic monthly-only and supplement-containing files, so we are unable to check the validity of our constructed HRHHID household identifier as we did for 1976. However, HRHHID in the basic monthly-only files results in each household containing only one household head, which is almost always consistent with the organization of households in the supplement-containing files and represents an improvement over HHID.

4.1.2 Non-numeric household identifiers

In January 1976 to June 1985, some records contained non-numeric characters in HHID. We made adjustments to the original household identifier that allow us to treat HRHHID as a numeric variable in all months. In 1976 and 1977 basic monthly-only files, a few records in each monthly file, usually less than 10, had a ‘ $-$ ’ in the final position of the 12-digit HRHHID variable. These strings are introduced into HRHHID through the adjustments described in the previous section. In 1978–1985, about 12,000 to 14,000 cases in each month contained a blank space or a ‘ $-$ ’ in the second position of HHID/HRHHID. The December 1978 file included a ‘/’ in the third position of HRHHID for 48 records. We replaced blank spaces with a ‘9’; ‘ $-$ ’ characters with an ‘8’;7 and ‘/’ characters with a ‘4’.8 This character swap preserves the uniqueness of the household identifier within each month of data.

4.1.3 Unlinkable months

Census Bureau documentation indicates that it is not possible to link some months of CPS data across years (1976 to 1977 and 1985 to 1986) due to changes in the survey [5]. We elaborate on these known limitations in the 1970s and remark briefly on our investigation in the 1980s.

Unlinkable months within 1977 and between 1976, 1977, and 1978 are the result of a phased-in sample size increase. Starting with the supplement-containing months of 1977, additional households were included in the CPS, resulting in 15,000 to 25,000 more individuals in these months (known as the D-sample). This sample size increase was extended to include all months in 1978. To prevent identification of these additional individuals in 1977, the Census Bureau altered the procedures for generating HHID so that supplement-containing months that included the D-sample could not be linked to basic monthly-only months without it [6].

Despite HRHHID uniquely identifying households within the basic monthly-only files in 1976 to 1978, this identifier does not always yield linkages across months within years or across years. Linkages are possible within 1976 and 1978 between months that are basic monthly-only and supplement-containing (see Fig. 1a). This is not the case, however, in 1977 due to the D-sample. Figure 1b shows linkages between types of files across years. In 1976, both basic monthly-only and supplement-containing files can be linked to 1977 basic monthly-only files, but not to 1977 supplement-containing files. No linkages are made between 1976 and 1978 even when possible given the 4-8-4 rotation pattern (i.e., MIS 1 in October, November, and December 1976). In 1977, supplement-containing files can be linked link to both 1978 basic monthly-only and supplement-containing files; no 1977 basic monthly-only file will link to either basic monthly-only or supplement-containing files in 1978.

Figure 1.

a. Linkages possible between file types within years, 1976–1978. b. Linkages possible between file types across years, 1976–1978.

We find that no month before June of 1985 can be linked to any month after June of 1985. This is consistent with Census Bureau technical documentation, which provides no explanation of why linkages are not possible [5]. We suspect that this linking barrier may be due to a CPS redesign that began in April of 1984 and concluded by July of 1985 [7]. We find a similar linking discontinuity between September and October of 1985. No month after September of 1985 can be linked to any month before September of 1985. This break in linking is due to a change in Census Bureau confidentiality rules beginning in October of 1985 that allows for the identification of smaller individual metropolitan areas in the public use data [12]. As a result of these two barriers to linking in 1985, July, August, and September of that year can only link to one another and to no other months.

4.2 Person identifiers and linking across months

4.2.1 Non-unique person identifiers

Person identifiers are required along with household identifiers to match individuals across months of the CPS. While person identifiers should be unique within household in a given data file, the same person identifier is sometimes assigned to multiple individuals in the same household in months between 1976 and 1983, resulting in duplicate values of the person line number (LINENO) within households. This presents a problem for linking individuals across months using only person (LINENO) and household identifiers (HRHHID and HUHHNUM).

In the basic monthly data, if two or more records in a household have identical person identifiers, we do not allow linkages across months for those individuals. CPSIDP is a linking key based solely on Census Bureau identifiers. Accordingly, we do not use additional information about the individuals to try to ascertain which record with the duplicated person identifier should be linked to a single record with the same identifier in a surrounding month. When we encounter pairs of records with duplicate LINENO values, we assign them both a new (and unique) CPSIDP value. Consider the following household with the HRHHID value of 003147831701 that appears in three months of the CPS, beginning in December of 1981.

Figure 2.

Household from the December 1981 current population survey file.

The first two records in the household in December of 1981, shown in Fig. 2, have the same identifiers (HRHHID, HUHHNUM, and LINENO). Though the identifiers are the same, there is variation between these records on the other variables, so the records are not complete duplicates. Since the first two records have the same identifiers, we assign them unique CPSIDP values.

This household is also in the January 1982 CPS (see Fig. 3). In January, HRHHID and HUHHUM values match and the LINENO values are 01 and 02, respectively. Without additional information, we do not know which of the people in this household in December 1981 with LINENO $=$ 01 is the same person in January of 1982 with LINENO $=$ 01. Therefore, when we see the individual with LINENO $=$ 01 in January 1982, we assign them a new CPSIDP value incrementing by one from the highest CPSIDP value ever assigned to a person in that household. Note that LINENO $=$ 02 is assigned the same CPSIDP value in both months.

Figure 3.

Household from the January 1982 current population survey file.

As of January 1982, the household with an HRHHID value of 003147831701 and an HUHHNUM value of one has appeared in the CPS twice and has four CPSIDP values. When we see the same household in February of 1982 (see Fig. 4) with LINENO values of 01 and 02, we assign the CPSIDP values to them that they were assigned in January of 1982. This occurs because we create CPSIDP by looking to the previous month for a match and continuing backward in time through all linkable months until a match is found.

Figure 4.

Household from the February 1982 current population survey file.

Our approach is conservative and undoubtedly misses some plausible links across months. However, CPSIDP is a mechanical match using only household and person identifiers to assign CPSIDP values. Duplicate person line numbers are uncommon, representing less than 1% of cases each month from January 1976 to December 1983.

Duplicate person line numbers are not an issue in the 1984 to 1988 period. The challenge in this period is that the CPS basic monthly files include three different versions of demographic variables and the Census Bureau documentation on which version to use for these purposes is unclear (see Appendix B for more details). Importantly, these files include three person line number variables. However, only one of the person line number variables (Item 18A in columns 541–542) uniquely identifies persons within households. We use the variable that uniquely identifies persons within households for linking.

4.2.2 Child records

Children under 14 are not included in basic monthly-only files until 1982. However, children are included in supplement-containing months prior to 1982: October 1976, October 1977, and May 1978 through December 1981 (see Appendix A). Despite inclusion in the data file, we do not attempt link persons under 14 across months in this period, as they do not appear in all months; they have CPSIDP values of 0.

5. Basic monthly survey linkage rates across time

Linkage rates across months in 1976 to 1988 using HRHHID, HUHHNUM, and LINENO as linking keys are lower than in recent years [2] due to comparatively poor data quality as outlined above in the earlier time period. We provide sample sizes and retention rates before and after validating links using AGE, SEX, and RACE for CPS data collected in 1976–1977 and 1987–1988 (see [2] for linkages in 1994–1995 and 2009–2010). Table 1 shows the total number of records in each month-in-sample group from January 1976 to April 1977 and January 1987 to April 1988.

Based on the 4-8-4 rotation pattern of the CPS, 75% of respondents are eligible to link between consecutive months (MIS 1-3 and MIS 5-7 in a given month can link to the next month; MIS 4 and 8 rotate out of the survey). Table 2 shows linkages and retention rates between January and February in 1976 and in 1987. More than 90% of eligible records in January are observed in February and nearly all of them are plausible based on age, sex, and race. Plausible links are those which have the same values for SEX and RACE in all time points and whose AGE does not increase by more than two years.

Linkages are also possible across non-consecutive months. Table 3 shows links two months apart, from October to December, in 1976 and in 1987. Half of individuals in October are eligible to participate in the CPS in December. Of those who are eligible, 73% are linked in 1976 compared to 90% in 1987. Most of these linkages are plausible based on comparisons of age, sex, and race.

The CPS rotation pattern also allows for linking the same month across adjacent years (see Table 4). The individuals in MIS 1-4 in 1976 and 1987 are eligible to participate in the CPS in the same month in 1977 and 1988, respectively. About 75% of eligible individuals are observed the next year and just over 70% of these links are plausible based on age, sex, and race.

Respondents may also be linked across up to eight months of participation in the CPS. Table 5 shows the number of people starting the CPS in January 1976 and January 1987 who appear in the CPS up to eight times. The individuals who are

Table 1
Number of people responding to the CPS, by calendar month, month-in-sample group, and year

	Jan	Feb	Mar	Apr	May	Jun	Jul	Aug	Sep	Oct	Nov	Dec	Jan	Feb	Mar	Apr
	1976												1977
MIS1	12,287	12,138	12,216	12,455	12,626	12,232	11,863	12,120	12,406	15,672	12,330	12,679	14,699	12,218	12,173	12,507
MIS2	12,194	12,509	12,209	12,475	12,565	12,795	12,279	12,094	12,385	15,750	12,641	12,508	14,842	12,780	12,197	12,328
MIS3	12,203	12,169	12,405	12,265	12,423	12,536	12,768	12,328	12,185	15,351	12,575	12,758	14,629	12,771	12,686	12,199
MIS4	12,080	12,128	12,147	12,467	12,251	12,341	12,559	12,681	12,313	15,305	12,343	12,638	14,803	12,491	12,636	12,772
MIS5	12,304	11,836	12,167	12,435	12,505	12,164	12,115	12,502	12,664	15,070	12,227	11,986	14,676	12,401	12,414	12,541
MIS6	12,336	12,481	11,837	12,347	12,432	12,580	12,255	12,189	12,654	16,074	12,218	12,353	14,491	12,482	12,371	12,451
MIS7	12,382	12,306	12,387	11,865	12,253	12,351	12,493	12,268	12,221	15,725	12,874	12,303	14,338	11,986	12,412	12,356
MIS8	12,490	12,351	12,191	12,461	11,866	12,241	12,267	12,515	12,241	15,413	12,664	12,849	14,496	12,349	11,989	12,443
Total	98,276	97,918	97,559	98,770	98,921	99,240	98,599	98,697	99,069	124,360	99,872	100,074	116,974	99,478	98,878	99,597
	1987												1988
MIS1	18,684	18,373	18,547	19,087	18,005	18,088	18,399	18,278	18,679	18,730	18,273	18,594	18,269	18,294	18,455	17,287
MIS2	18,254	18,955	18,482	19,060	19,412	18,290	18,467	18,908	18,645	19,239	18,838	18,673	18,991	18,476	18,570	17,725
MIS3	18,857	18,290	18,912	18,580	19,023	19,321	18,106	18,488	19,122	18,962	19,025	18,814	18,658	18,973	18,381	17,391
MIS4	18,514	18,767	18,127	18,824	18,484	18,840	19,022	18,076	18,496	19,191	18,735	18,829	18,633	18,682	18,939	17,192
MIS5	17,691	18,105	18,677	18,619	18,567	18,466	18,435	18,691	18,421	18,674	18,675	18,167	18,514	18,328	18,582	17,696
MIS6	18,855	17,975	18,119	18,974	18,741	18,623	18,643	18,717	18,928	18,694	18,638	18,879	18,312	18,632	18,319	17,486
MIS7	18,806	18,832	17,915	18,063	18,935	18,591	18,434	18,613	18,745	19,099	18,552	18,535	18,891	18,288	18,491	16,929
MIS8	18,922	18,689	18,764	17,905	17,984	18,801	18,521	18,495	18,637	18,785	18,845	18,338	18,501	18,893	18,200	17,221
Total	148,583	147,986	147,543	149,112	149,151	149,020	148,027	148,266	149,673	151,374	149,581	148,829	148,769	148,566	147,937	138,927

Note: Table reports unweighted sample sizes for the number of people participating in the CPS in each calendar month, by month-in-sample group.

Table 2

Sample size and retention rate, CPS respondents linked across two consecutive calendar months

	Year X		1976			1987
	Jan	Feb	Jan	Feb (all)	Feb (plausible)	Jan	Feb (all)	Feb (plausible)
MIS1	Jan ${}_{X}$	Feb ${}_{X}$	12,287	–	–	18,684	–	–
MIS2	Dec ${}_{X-1}$	Jan ${}_{X}$	12,194	11,434	11,168	18,254	17,652	17,125
MIS3	Nov ${}_{X-1}$	Dec ${}_{X-1}$	12,203	11,385	11,180	18,857	17,241	16,803
MIS4	Oct ${}_{X-1}$	Nov ${}_{X-1}$	–	11,443	11,261	–	17,869	17,437
MIS5	Jan ${}_{X-1}$	Feb ${}_{X-1}$	12,304	–	–	17,691	–	–
MIS6	Dec ${}_{X-2}$	Jan ${}_{X-1}$	12,336	11,423	11,187	18,855	16,562	16,105
MIS7	Nov ${}_{X-2}$	Dec ${}_{X-2}$	12,382	11,346	11,111	18,806	17,585	17,193
MIS8	Oct ${}_{X-2}$	Nov ${}_{X-2}$	–	11,474	11,264	–	17,510	17,097
Total			73,706	68,505	67,171	111,147	104,419	101,760
Retention rate				92.94%	91.13%		93.95%	91.55%

Note: Table reports the unweighted number and percentage of CPS respondents in January of Year X (the shaded box) who responded to the CPS in February of that year. Under “Year X,” entries report the month and year in which respondents were in MIS1. Because of the rotation group structure, not all respondents in January are eligible to respond in February. The column labeled “plausible” omits apparent matches when respondents’ sex or race/ethnicity differs or when their age differs implausibly.

Table 3

Sample size and retention rate, CPS respondents linked across two non-consecutive calendar months

	Year X		1976			1987
	Oct	Dec	Oct	Dec (all)	Dec (plausible)	Oct	Dec (all)	Dec (plausible)
MIS1	Oct ${}_{X}$	Dec ${}_{X}$	15,672	–	–	18,730	–	–
MIS2	Sep ${}_{X}$	Nov ${}_{X}$	15,750	–	–	19,239	–	–
MIS3	Aug ${}_{X}$	Oct ${}_{X}$	–	11,485	11,180	–	17,193	16,533
MIS4	Jul ${}_{X}$	Sep ${}_{X}$	–	11,675	11,449	–	17,544	17,003
MIS5	Oct ${}_{X-1}$	Dec ${}_{X-1}$	15,070	–	–	18,674	–	–
MIS6	Sep ${}_{X-1}$	Nov ${}_{X-1}$	16,074	–	–	18,694	–	–
MIS7	Aug ${}_{X-1}$	Oct ${}_{X-1}$	–	10,959	10,762	–	16,740	16,185
MIS8	Jul ${}_{X-1}$	Sep ${}_{X-1}$	–	11,730	11,499	–	16,672	16,214
Total			62,566	45,849	44,890	75,337	68,149	65,935
Retention rate				73.28%	71.75%		90.46%	87.52%

Note: Table reports the unweighted number and percentage of CPS respondents in October of Year X (the shaded box) who responded to the CPS in December of that year. Under “Year X,” entries report the month and year in which respondents were in MIS1. Because of the rotation group structure, not all respondents in October are eligible to respond in December. The column labeled “plausible” omits apparent matches when respondents’ sex or race/ethnicity differs or when their age differs implausibly.

Table 4

Sample size and retention rate, CPS respondents linked in March across two consecutive years

	March		March			March
	Year X	Year X $+$	1976	1977 (all)	1977 (plausible)	1987	1988 (all)	1988 (plausible)
MIS1	Mar ${}_{X}$	Mar ${}_{X+1}$	12,216	–	–	18,547	–	–
MIS2	Feb ${}_{X}$	Feb ${}_{X+1}$	12,209	–	–	18,482	–	–
MIS3	Jan ${}_{X}$	Jan ${}_{X+1}$	12,405	–	–	18,912	–	–
MIS4	Dec ${}_{X-1}$	Dec ${}_{X}$	12,147	–	–	18,127	–	–
MIS5	Mar ${}_{X-1}$	Mar ${}_{X}$	–	9,277	8,941	–	13,687	12,991
MIS6	Feb ${}_{X-1}$	Feb ${}_{X}$	–	9,212	8,927	–	13,583	12,970
MIS7	Jan ${}_{X-1}$	Jan ${}_{X}$	–	9,412	9,132	–	13,984	13,345
MIS8	Dec ${}_{X-2}$	Dec ${}_{X-1}$	–	9,144	8,843	–	13,636	13,008
Total			48,977	37,045	35,843	74,068	54,890	52,314
Retention rate				75.64%	73.18%		74.11%	70.63%

Note: Table reports the unweighted number and percentage of CPS respondents in March of Year X (the shaded box) who responded to the CPS in March of the next year. Under “Year X,” entries report the month and year in which respondents were in MIS1. Because of the rotation group structure, not all respondents in March are eligible to respond the following March. The column labeled “plausible” omits apparent matches when respondents’ sex or race/ethnicity differs or when their age differs implausibly.

Table 5

Number and percentage of people responding to subsequent CPS surveys among those beginning the CPS in January 1976 and 1987

	All links	Retention rate	Plausible links	Retention rate
	1976
Began in CPS in MIS 1 in Jan 1976 …	12,287	–	–	–
… and also responded in February 1976	11,434	93.06%	11,168	90.89%
… and responded on all 3 occasions through March 1976	10,938	89.02%	10,636	86.56%
… and responded on all 4 occasions through April 1976	10,486	85.34%	10,209	83.09%
… and responded on all 5 occasions through January 1977	0	0.00%	0	0.00%
… and responded on all 6 occasions through February 1977	0	0.00%	0	0.00%
… and responded on all 7 occasions through March 1977	0	0.00%	0	0.00%
… and responded on all 8 occasions through April 1977	0	0.00%	0	0.00%
	1987
Began in CPS in MIS 1 in Jan 1987 …	18,684	–	–	–
… and also responded in February 1987	17,652	94.48%	17,125	91.66%
… and responded on all 3 occasions through March 1987	16,781	89.81%	16,212	86.77%
… and responded on all 4 occasions through April 1987	16,052	85.91%	15,553	83.24%
… and responded on all 5 occasions through January 1988	12,985	69.50%	12,426	66.51%
… and responded on all 6 occasions through February 1988	12,546	67.15%	12,123	64.88%
… and responded on all 7 occasions through March 1988	12,101	64.77%	11,665	62.43%
… and responded on all 8 occasions through April 1988	11,012	58.94%	10,627	56.88%

Note: Separately for people entering the CPS in January 1976 and January 1987, the table reports unweighted sample sizes for the number of people participating in all of the CPS surveys for which they were eligible up through the focal month. For example, among the 12,287 people who began the CPS in January of 1976, there were 10,486 (or 85.34%) who participated in all four surveys between January and April 1976 and 0 who participated in all eight surveys between January 1976 and April 1977 due to the linking discontinuity between samples with and without supplements in these years. The column labeled “plausible” omits apparent matches when respondents’ sex or race/ethnicity differs or when their age differs implausibly.

first observed in January 1976 link well in 1976; 93% are also observed in February 1976 and 85% are also observed in February, March, and April of 1976. However, no individuals who started the CPS in January 1976 are linked to January 1977, as it is a supplement-containing month and such linkages are not possible (as described above). The individuals who begin the CPS in January 1987 link well within 1987; 85.91% are observed in all four months between January and April 1987. Linkage rates decrease after the eight month gap between MIS 4 and MIS 5 (see also [2]). About 59% of respondents who started the CPS in January 1987 are observed all eight times. In both 1976 and 1987, most of the linkages we make are plausible based on age, sex, and race.

Table 6

Number and percentage of people responding to subsequent CPS surveys among those beginning the CPS in January 1976 and 1987

	All links	Retention rate	Plausible links	Retention rate
	1976
Began in CPS in MIS 1 in Jan 1976 …	12,287	–	–	–
… and also responded in ANY of the subsequent	11,775	95.83%	11,574	94.20%
surveys between February 1976 and April 1977
	1987
Began in CPS in MIS 1 in Jan 1987 …	18,684	–	–	–
… and also responded in ANY of the subsequent	18,135	97.06%	17,757	95.04%
surveys between February 1987 and April 1988

Note: Separately for people entering the CPS in January 1976 and January 1987, the table reports unweighted sample sizes for the number of people participating in ANY of the CPS surveys for which they were eligible up through April of the following year. For example, among the 12,287 people who began the CPS in January of 1976, there were 11,775 (or 95.83%) who participated in at least one more survey between February 1976 and April 1977. The column labeled “plausible” omits apparent matches when respondents’ sex or race/ethnicity differs or when their age differs implausibly.

Table 7

Number and percentage of people responding to subsequent CPS surveys among those beginning the CPS in January 1976 and 1988

	All links	Retention rate	Plausible links	Retention rate
	1976
Began in CPS in MIS 1 in Jan 1976 …	12,287	–	–	–
… and also responded in ANY of the subsequent	9,475	77.11%	9,057	73.71%
surveys between February 1976 and April 1977
	1987
Began in CPS in MIS 1 in Jan 1987 …	18,684	–	–	–
… and also responded in ANY of the subsequent	14,676	78.55%	13,855	74.15%
surveys between February 1987 and April 1988

Note: Separately for people entering the CPS in January 1976 and January 1987, the table reports unweighted samples sizes for the number of people participating in any of the CPS surveys for which they were eligible between January and April of the following year. For example, among the 12,287 people who began the CPS in January of 1976, there were 9,475 (or 77.11%) who participated in at least one more survey between February 1976 and April 1977. The column labeled “plausible” omits apparent matches when respondents’ sex or race/ethnicity differs or when their age differs implausibly.

Tables 6–8 show linkage rates for a variety of different linkage scenarios. Table 6 shows those individuals who entered the CPS in January 1976 and January 1987 (MIS 1) and appear in any of the subsequent seven months that their household could have been in the CPS (either MIS 2 or MIS 3 or MIS 4 or MIS 6 or MIS 7 or MIS 8). Most respondents (95% in 1976 and 97% in 1987) appear in at least two months of the CPS. Table 7 shows the percent of individuals in January 1976 (and 1987) who appear in the CPS in MIS 2-4. About three-quarters of individuals who begin the CPS in January 1976 and January 1987 are observed in at least one additional month between February and April of 1976 and 1987, respectively. Finally, Table 8 shows attrition between MIS 4 and 5.

6. Extending CPSID(P) to the Annual Social and Economic Supplement, 1976–1988

The ability to easily link the Annual Social and Economic (ASEC) Supplement with the CPS BMS creates many research possibilities. Information only available in the ASEC may be combined with multiple data points from the BMS or used in combination with CPS topical supplements. To make these linkages easier, we add CPSID(P) to the ASEC files. Research utilizing these linking keys, for example, combines information on union membership from the monthly data with tax and public benefit receipt from the ASEC [8] and analyzes family income and health insurance from the ASEC along with smoking behavior from the Tobacco Use Supplement [9].

This section of the paper details the creation of MARBASECIDH and MARBASECIDP, hereafter MARBASECID(P), for the 1976 to 1988 period.9 MARBASECIDH and MARBASECIDP are IPUMS CPS variables that link the March basic monthly data to ASEC data from the same year and enable the addition of CPSID(P) to the ASEC files. Data quality issues in 1976 to 1988 such as differing numbers of records in the household between the March basic monthly and ASEC files and mismatched or duplicated person identifiers necessitated a methodology distinct from the 1989 forward period, which relied on Census Bureau identifiers only (with a few exceptions). The methodology for creating MARBASECID(P) in 1976 to 1988 is also distinct from our CPSID(P) methodology. Because all records from the March BMS should theoretically appear in the ASEC file for a given year, we are more persistent in our attempts to create linkages between the March BMS and the ASEC than we are for links across months of the BMS. We describe the problems encountered in attempting to link March BMS and ASEC files in this period, detail the solutions we implemented to generate MARBASECID(P), and compare our methodology with alternatives.

6.1 Missing March basic monthly records in the ASEC

Between 1976 and 1988, the ASEC file should include all March basic monthly households plus a Hispanic oversample drawn from the previous November CPS [4]. However, in all years during this period, the March basic monthly file contains individuals who do not appear in the ASEC file. Table 10 shows the total number of

Table 8
Sample size and retention rate, CPS respondents in month-in-sample 4 linked to month-in-sample 5

	Jan	Feb	Mar	Apr	May	Jun	Jul	Aug	Sep	Oct	Nov	Dec	Jan	Feb	Mar	Apr
	Year X												Year X $+$ 1
MIS4	Oct ${}_{X-1}$	Nov ${}_{X-1}$	Dec ${}_{X-1}$	Jan ${}_{X}$	Feb ${}_{X}$	Mar ${}_{X}$	Apr ${}_{X}$	May ${}_{X}$	Jun ${}_{X}$	Jul ${}_{X}$	Aug ${}_{X}$	Sep ${}_{X}$	Oct ${}_{X}$	Nov ${}_{X}$	Dec ${}_{X}$	Jan ${}_{X+1}$
MIS5	Jan ${}_{X-1}$	Feb ${}_{X-1}$	Mar ${}_{X-1}$	Apr ${}_{X-1}$	May ${}_{X-1}$	Jun ${}_{X-1}$	Jul ${}_{X-1}$	Aug ${}_{X-1}$	Sep ${}_{X-1}$	Oct ${}_{X-1}$	Nov ${}_{X-1}$	Dec ${}_{X-1}$	Jan ${}_{X}$	Feb ${}_{X}$	Mar ${}_{X}$	Apr ${}_{X}$
	1976												1977
MIS4	12,080	12,128	12,147	12,467	12,251	12,341	12,559	–	–	–	–	–	–	–	–	–
MIS5 (all)	–	–	–	–	–	–	–	–	–	9,438	9,494	9,379	0	9,493	9,763	9,767
Retention rate	–	–	–	–	–	–	–	–	–	78.13%	78.28%	77.21%	0.0%	77.49%	79.11%	77.77%
(all)
MIS5	–	–	–	–	–	–	–	–	–	9,206	9,255	9,131	0	9,247	9,511	9,523
(plausible)
Retention rate	–	–	–	–	–	–	–	–	–	76.21%	76.31%	75.17%	0.0%	75.48%	77.07%	75.83%
(plausible)
	1987												1988
MIS4	18,514	18,767	18,127	18,824	18,484	18,840	19,022	–	–	–	–	–	–	–	–	–
MIS5 (all)	–	–	–	–	–	–	–	–	–	14,345	14,552	14,148	14,634	14,402	14,786	14,077
Retention rate	–	–	–	–	–	–	–	–	–	77.48%	77.54%	78.05%	77.74%	77.92%	78.48%	74.00%
(all)
MIS5	–	–	–	–	–	–	–	–	–	13,753	14,000	13,632	14,087	13,875	14,183	13,539
(plausible)
Retention rate	–	–	–	–	–	–	–	–	–	74.28%	74.60%	75.20%	74.84%	75.06%	75.28%	71.18%
(plausible)

Note: Table reports the number and percentage of CPS respondents in month-in-sample four who responded to the CPS in month-in-sample five nine months later. Under “Year X,” entries report the month and year in which respondents were in MIS1. The rows labeled “plausible” omit apparent matches when respondents’ sex or race/ethnicity differs or when their age differs implausibly.

individuals in the March basic monthly survey for each year between 1976 and 1988 (Panel A), the number that merge to the ASEC (Panel B), and details about unmerged individuals (Panel C). Panel B shows that, except in 1977, over 98% of individuals in the March basic monthly file are merged with the ASEC; the majority are located in merge stage 1, which we describe in more detail below in the “Strategy for Linking March basic monthly to ASEC” section. For those unmerged, we differentiate between individuals whose household (using HRHHID) is or is not in the ASEC. In the first instance, the same HRHHID value is in both the ASEC and the March BMS. Either the household in the ASEC contains fewer persons than the same household in the March BMS or we are unable to confidently match records within a household across files due to duplication or mismatch of variables (described in the next section). The number of basic monthly records for which the HRHHID does not appear in the ASEC is less than 100 per year in 1976 to 1985 and is in the thousands in 1986 to 1988. In the 1986–1988 period, the Census Bureau scrambled household identifiers in the ASEC file for privacy reasons,10 and we do not attempt to link the basic monthly and ASEC in these years.

6.2 Person identifiers and auxiliary linking keys

We encounter difficulties uniquely identifying records within files and in matching records across BMS and ASEC files from 1976–1988. This is not the case in the 1989 forward period where HRHHID and LINENO are sufficient to uniquely identify almost all records within and link across March BMS and ASEC files [2].

6.2.1 Difficulty uniquely identifying records

With the exception of the 1982 and 1983 March BMS files, no March BMS or ASEC files contain records that are complete duplicates between 1976 and 1988. However, even though entire records are unique in most of the files, we are often unable to find a single set of variables to uniquely identify all records in households that appear in both the March BMS and the ASEC file for a given year. The 1989-onward method of simply using linking keys to match March BMS and ASEC data is insufficient. Furthermore, we are unable to identify a single set of variables to use as linking keys that allow us to link all March basic monthly records to their ASEC counterparts for all years in the 1976–1988 period.

Figure 5.

Household 509037594903 in the 1982 March BMS and ASEC.

Figure 6.

Household 202962182216 in the 1985 March BMS and ASEC.

Consider the following example household (HRHHID $=$ 509037594903) in Fig. 5. This household is in the March BMS and the ASEC. The household contains several individual records that have the same values for both LINENO and demographic variables in the March BMS, though they are not completely identical records. Of the eight records in the household, only person records with LINENO values of 1 and 2 are uniquely identified using LINENO and demographic characteristics, while those with LINENO values of 3, 4, and 5 appear twice. Duplication is not a problem in this household in the ASEC. Because the records with LINENO 1 and 2 in the March BMS and ASEC file have unique linking keys, we successfully link these records. This same household in the ASEC file contains only one record each with LINENO values of 3, 4, and 5; these records also have the same demographic characteristics as their duplicated March BMS counterparts. However, in this instance, we attempt to link only the first two persons in the household to the ASEC file. Because linking keys and demographic characteristics do not uniquely identify the additional persons in the BMS file and it is unclear which BMS records should link to the ASEC, we do not attempt to link these records.

In other cases, person identifiers (LINENO) do not match across the March BMS and ASEC files and the demographic variables are insufficient for uniquely identifying individuals when the person identifier is omitted. For example, consider the following household (HRHHID $=$ 202962182216) in 1985 shown in Fig. 6. This household contains eight persons in each of the March BMS and ASEC files; LINENO uniquely identifies persons in the March BMS file, but not in the ASEC. Without LINENO as a linking key, the remaining demographic variables do not uniquely identify all records. In this instance, the two 11-year-olds and the two 12-year-olds (LINENO values 7 and 8 and 5 and 6, respectively in the March BMS file and LINENO values 3 and 4 and 0 and 4, respectively in the ASEC file) cannot be unambiguously matched.

Figure 7.

Household 046112033020 in the 1976 March BMS and ASEC.

Figure 8.

Household 50007137050 in the 1976 March BMS and ASEC.

6.2.2 Unmatchable records

There are two ways that records may not be linked even if identifiers are unique in 1976 to 1988. Both occur because LINENO values do not match across BMS and ASEC files, which undermines the utility of LINENO as a linking key.11 For example, in this household from 1976 (HRHHID $=$ 046112033020) shown in Fig. 7, the two sixteen year-olds can be uniquely identified within their household within both the March BMS and the ASEC file. Notice, however, that the order of the people in the household varies across the BMS and ASEC and the LINENO values are not consecutive in the BMS. Because of the LINENO mismatch, the individuals aged 16 and 18 will not be matched across the March BMS and ASEC files even though they may well represent the same people.

Second, some individuals are uniquely identified by demographic variables and line number and appear in the same order within the household, but person identifiers are different across the March BMS and ASEC files. This is another instance where despite having unique person line numbers linkages across March BMS and ASEC files are not possible. We illustrate this situation in a household from 1976. The households contain the same number of people with the same age, sex, race, and work hours, but have different LINENO values in the March BMS and the ASEC. This data quality issue is not present in the data from 1989 forward but presents a major obstacle to linking March BMS and ASEC records from 1976–1988.

7. Strategy for linking March basic monthly to ASEC

Given the problems detailed above, we apply a multi-stage merging process to maximize linkage rates while minimizing spurious matches. Demographic information is indispensable from 1976 to 1988 for making matches between the March BMS and ASEC, though it is only used in a handful of cases in the 1989–2019 period. Reliance on demographic and other auxiliary information for linking represents a departure from our general approach of creating mechanical links for CPSID(P) and MARBASECID(P). However, the unique challenges during this period necessitate a different approach. We identify matches between the March BMS and ASEC in up to six stages: four in 1976–1981, five in 1982–1987, and six in 1988. We use person identifiers and demographic information as linking keys in some stages and exclude them in other stages.

7.1 Preliminary steps

For an individual to be matched across BMS and ASEC files, the household identifier must appear in both files. Before attempting to link, we exclude those individuals whose HRHHID value does not appear in both the BMS and ASEC files (see Table 10). Because there are no persons under the age of 14 in the March BMS from 1976 to 1981, we also exclude individuals under age 14 in the ASEC from linking to the March BMS. Individuals excluded from the matching are assigned non-linking MARBASECIDP values and CPSIDP values of 0 in the ASEC file.12

We use original Census variables for linking March BMS to ASEC with three exceptions, HRHHID in the 1976 BMS, Race in the 1988 ASEC, and hours worked in the 1982–1988 BMS. The adjustment made to the 1976 BMS household identifier is described in the “Problem 1: Household Identifiers and Linking across Months” section above. In 1988, race codes are different in the March BMS and ASEC; we standardize race codes by recoding values of 3, 4 or 5 in the ASEC (“American Indian or Aleut Eskimo”, “Asian or Pacific Islander”, and “Other”) to 3, which represents “Other” in the March BMS. In the “hours worked” variable (AHRSWORKT) in all March BMS files from 1982–1988, there are two varieties of missing values, one for adults who were not working last week and one for children. In the ASEC files from these years, all NIU cases have a value of 0. We recode the two missing values in the March Basic Monthly files to 0 before matching. This missing value code harmonization is available in the IPUMS CPS variable AHRSWORKT.

7.2 Linking methods

Before each linking stage, we set aside all records for later use that are not uniquely identified by that stage’s linking keys (Table 9). After linking, we combine the set-aside records with the unlinked records (separately for the March BMS and ASEC); these files serve as the inputs for the next linking stage. We detail the number of linkages between the March BMS and the ASEC at each stage for every year from 1976 to 1988 in Table 10. The linking stage in which a given record was linked is available to researchers via IPUMS CPS in the variable MARBASECSTAGE, which is available on both the March BMS and ASEC files from 1976 to 1988.

Table 9
Multi-stage merge linking keys, 1976–1988

Merge stage	Linking keys	Years
1	HRHHID, HUHHNUM, LINENO, AGE, SEX, RACE	1976–1988
2	HRHHID, HUHHNUM, AGE, SEX, RACE	1976–1988
3	HRHHID, HUHHNUM, AGE, SEX, RACE, HOURS	1976–1988
4	HRHHID, HUHHNUM, LINENO	1982–1988
5	HRHHID, HUHHNUM, NUMPER	1982–1988
6	HRHHID, NUMPER, AGE, SEX, RACE	1988

All BMS files in the 1976–1988 period contain multiple versions of the person identifier and age, sex, and race variables, which differ slightly from one another (see Appendix B). We use the version of these variables that yields the highest match rate with the ASEC. These variables are available via IPUMS CPS in LINENO,

Table 10

Multi-stage merge of March basic monthly and ASEC files, 1976–1988

			1976	1977	1978	1979	1980	1981	1982	1983	1984	1985	1986	1987	1988
A. Total BMS
		N	97559		113992	113689	133914	133488	153195	153431	152614	152705	149440	147543	147937
B. Merged BMS
		N	97511		113935	113661	133889	133417	153017	153201	152418	152466	147383	145594	145041
		%	99.95%		99.95%	99.98%	99.98%	99.95%	99.88%	99.85%	99.87%	99.84%	98.62%	98.68%	98.04%
	Merge stage
		1	91988		106720	113606	133828	133394	152888	152869	151402	151338	145956	144140	143058
		2	5505		7191	55	59	21	48	187	873	980	971	953	387
		3	18		24	0	2	2	2	12	14	8	10	28	0
		4	–		–	–	–	–	78	128	120	122	440	457	1246
		5	–		–	–	–	–	1	5	9	18	6	16	12
		6	–		–	–	–	–	–	–	–	–	–	–	338
C. Unmerged BMS
		N	48		57	28	25	71	178	230	196	239	2057	1949	2896
	HHID in ASEC but
	individual is not
		N	36		29	18	10	21	145	168	155	189	205	187	469
	HHID not in ASEC
		N	12		28	10	15	50	33	62	41	50	1852	1762	2427

AGE, SEX, and RACE. The ASEC only has one person identifier and one set of demographic variables in all years.

Stage 1: Stage 1 records are those that are uniquely identified by and linked using HRHHID, HUHHNUM, LINENO, AGE, SEX, and RACE. The majority of March BMS records are linked to the ASEC in this stage (Table 10).

Stage 2: In the second stage of linking, we exclude LINENO to mitigate the problem of non-consecutive and mismatched LINENO values across March BMS and ASEC files. We retain and attempt to link unlinked stage 1 records that are uniquely identified in each file using HRHHID, HUHHNUM, AGE, SEX, and RACE. The result is hundreds or thousands of additional matches (Table 10).

Stage 3: Stage 3 linking adds the number of hours worked last week (HOURS) to the Stage 2 linking keys. We link a handful of records in this stage (Table 10).

Stage 4: In stage 4, we use only household and person identifiers to link records, which is consistent with our approach to assigning MARBASECIDP in 1989 forward. This approach accounts for many successful merges between 1982 and 1988 (Table 10).

Stage 5: In this stage, we link singletons who are in households that have the same number of individuals in the March BMS and the ASEC and where all other household members have already been linked. Stage 5 uses household identifiers and the number of persons in the household (NUMPER) as linking keys.13 These singletons have not merged in previous stages due to mismatch of either LINENO or demographic variables but are the only remaining possible matches. For example, in the household shown in Fig. 9 (HRHHID $=$ 80079928707), the three year-old in the March BMS and the zero year-old in the ASEC are the only records in this household that are unmatched after stage 4. Despite differences on LINENO and AGE, we link them in stage 5, as the only remaining household member that is in both files but still unlinked. This merge stage links individuals in 1982–1988.

Stage 6: Stage 6 linking addresses a specific problem with household number (HUHHNUM) in 1988. There are many households in 1988 where HUHHNUM does not match across the March BMS and ASEC files, but whose members have the same values for AGE, SEX, and RACE across files. Most of these households have an HUHHNUM value of 0 in the ASEC file, despite HUHHNUM generally having a minimum value of 1. In this case, we eliminate HUHHNUM as a linking key and add number of persons in the household (NUMPER) as a linking key. This yields 336 additional linked records. Figure 10 shows an example of persons merged between March BMS and ASEC files in stage 6.

Table 11

Comparison between multi stage and single stage merges, 1976–1988

1976	97,559	97,511	94,820	2,691	91,988	5,523
Year	Total BMS	Multi-stage merge	Single-stage merge, identifiers only	Additional matches from multi-stage merge	Single-stage merge, including demographic
variables	Additional matches from multi-stage merge
1977
1978	113,992	113,935	110,211	3,724	106,720	7,215
1979	113,689	113,661	110,190	3,471	113,606	55
1980	133,914	133,889	129,438	4,451	133,828	61
1981	133,488	133,417	129,002	4,415	133,394	23
1982	153,195	153,017	149,583	3,434	152,888	129
1983	153,431	153,201	151,444	1,757	152,869	332
1984	152,614	152,418	150,181	2,237	151,402	1,016
1985	152,705	152,466	150,086	2,380	151,338	1,128
1986	149,440	147,383	145,047	2,336	145,956	1,427
1987	147,543	145,594	143,275	2,319	144,140	1,454
1988	147,937	145,041	144,043	998	139,068	5,973

Figure 9.

March BMS-ASEC merge stages for household 80079928707.

Figure 10.

An example of persons linked between March BMS and ASEC in stage 6.

Table 12

Comparison between multi stage and single stage merges, 1976–1988

		Single-stage merge, identifiers only					Single-stage merge, including demographic variables					Multi-stage merge
		BMS duplicates	Successful merges		Valid merges (age, sex, race)		BMS duplicates	Successful merges		Valid merges (age, sex, race)		Successful merges		Valid merges (age, sex, race)
Year	Total BMS	$n$	$n$	%	$n$	%	$n$	$n$	%	$n$	%	$n$	%	$n$	%
1976	97,559	250	94,830	97.20%	91,870	94.17%	14	91,988	94.29%	91,988	94.29%	97,511	99.95%	97,511	99.95%
1977
1978	113,992	593	110,222	96.69%	106,434	93.37%	12	106,720	93.62%	106,720	93.62%	113,935	99.95%	113,935	99.95%
1979	113,689	392	113,226	99.59%	113,226	99.59%	18	113,606	99.93%	113,606	99.93%	113,661	99.98%	113,661	99.98%
1980	133,914	652	133,185	99.46%	133,185	99.46%	12	133,828	99.94%	133,828	99.94%	133,889	99.98%	133,889	99.98%
1981	133,488	598	132,818	99.50%	132,818	99.50%	22	133,394	99.93%	133,394	99.93%	133,417	99.95%	133,417	99.95%
1982	153,195	939	152,137	99.31%	152,058	99.26%	80	152,888	99.80%	152,888	99.80%	153,017	99.88%	152,938	99.83%
1983	153,431	1725	151,436	98.70%	151,310	98.62%	109	152,869	99.63%	152,869	99.63%	153,201	99.85%	153,068	99.76%
1984	152,614	0	150,806	98.82%	150,681	98.73%	0	151,402	99.21%	151,402	99.21%	152,418	99.87%	152,289	99.79%
1985	152,705	0	150,673	98.67%	150,541	98.58%	0	151,338	99.10%	151,338	99.10%	152,466	99.84%	152,327	99.75%
1986	149,440	0	145,635	97.45%	145,184	97.15%	0	145,956	97.67%	145,956	97.67%	147,383	98.62%	146,937	98.33%
1987	147,543	0	143,843	97.49%	143,368	97.17%	0	144,140	97.69%	144,140	97.69%	145,594	98.68%	145,121	98.36%
1988	147,937	0	144,701	97.81%	143,058	96.70%	0	143,058	96.70%	143,058	96.70%	145,041	98.04%	143,783	97.19%

8. Comparing multi- and single-stage linking methods

Our multi-stage linking process yields more matches across March basic monthly and ASEC files than a single-stage linking approach. Table 11 compares the number of linkages resulting from single-stage linking using HRHHID and LINENO; single-stage linking using HRHHID, LINENO, AGE, SEX, and RACE; and our multi-stage approach. Our multi-stage approach yields more linkages than both single-stage linkage approaches and higher quality linkages as we illustrate next.

9. Validating March basic and ASEC linkages

Table 12 shows the number of records from our multi-stage linking process that match on AGE, SEX, and RACE in every year. Note that because AGE, SEX, and RACE were linking keys in the first three stages, the number of linked records that match on AGE, SEX, and RACE is extremely high across all years. AGE mismatches are most common followed by SEX and RACE.

Table 13
Validation on demographic characteristics broken out by merge stage

	Merges by stage	Matches by characteristic					Merges by stage	Matches by characteristic
		Age	Sex	Race	Age, sex, and race			Age	Sex	Race	Age, sex, and race
	1982 ( $N=$ 153195)						1983 ( $N=$ 153431)
Stage 1	152888	152888	152888	152888	152888	Stage 1	152869	152869	152869	152869	152869
Stage 2	48	48	48	48	48	Stage 2	187	187	187	187	187
Stage 3	2	2	2	2	2	Stage 3	12	12	12	12	12
Stage 4	78	35	68	53	0	Stage 4	128	52	87	114	0
Stage 5	1	0	1	1	0	Stage 5	5	0	4	5	0
Stage 6	0	0	0	0	0	Stage 6	0	0	0	0	0
N matches	153017	152973	153007	152992	152938	N matches	153201	153120	153159	153187	153068
% matches	–	99.97%	99.99%	99.98%	99.95%	% matches	–	99.95%	99.97%	99.99%	99.91%
	1984 ( $N=$ 152614)						1985 ( $N=$ 152614)
Stage 1	151402	151402	151402	151402	151402	Stage 1	151338	151338	151338	151338	151338
Stage 2	873	873	873	873	873	Stage 2	980	980	980	980	980
Stage 3	14	14	14	14	14	Stage 3	8	8	8	8	8
Stage 4	120	48	80	107	0	Stage 4	122	36	95	113	0
Stage 5	9	0	9	9	0	Stage 5	18	2	17	18	1
Stage 6	0	0	0	0	0	Stage 6	0	0	0	0	0
N matches	152418	152337	152378	152405	152289	N matches	152466	152364	152438	152457	152327
% matches	–	99.95%	99.97%	99.99%	99.92%	% matches	–	99.93%	99.98%	99.99%	99.91%
	1986 ( $N=$ 149440)						1987 ( $N=$ 147543)
Stage 1	145956	145956	145956	145956	145956	Stage 1	144140	144140	144140	144140	144140
Stage 2	971	971	971	971	971	Stage 2	953	953	953	953	953
Stage 3	10	10	10	10	10	Stage 3	28	28	28	28	28
Stage 4	440	59	401	418	0	Stage 4	457	58	415	441	0
Stage 5	6	0	6	6	0	Stage 5	16	1	15	16	0
Stage 6	0	0	0	0	0	Stage 6	0	0	0	0	0
N matches	147383	146996	147344	147361	146937	N matches	145594	145180	145551	145578	145121
% matches	–	99.74%	99.97%	99.99%	99.70%	% matches	–	99.72%	99.97%	99.99%	99.68%
	1988 ( $N=$ 147937)
Stage 1	143058	143058	143058	143058	143058
Stage 2	387	387	387	387	387
Stage 3	0	0	0	0	0
Stage 4	1246	199	881	1102	0
Stage 5	12	1	7	11	0
Stage 6	338	338	338	338	338
N matches	145041	143983	144671	144896	143783
% matches	–	99.27%	99.74%	99.90%	99.13%

Table 13 shows validation broken out by linking stage for 1982–1988. Stage 4, where we use only household and person identifiers, validates most poorly.

10. Discussion

Generating a linking key that performs well for merging across months of the CPS and to ASEC files from 1976 to 1988 is an involved process. Different numbers of people across files and within households, duplicate and unlinkable records, and different coding schemes in linking keys across files amplify the complexity of this endeavor. This documentation along with linking keys in the IPUMS CPS data are intended as a resource for the research community. Our goal is to facilitate linkages between CPS files and to supplement existing documentation about CPS linking. As part of this effort, we also strive to be transparent about our processes and to provide researchers with the flexibility to retain and drop linkages we have made based on their comfort with the linkages we have created. We have described the many problems we encountered, the several steps we took to resolve problems, and the multi-stage methodologies we used to create and add a single linking key to all CPS files from 1976 to 1988. By creating CPSID(P) and MARBASECID(P), IPUMS saves time, eliminates duplication of effort, reduces errors for individual researchers, and provides the research community with a common starting point for linking across CPS data files.

The primary issues we encountered in linking across BMS files to create CPSID(P) were with household and person identifiers. Our description details the adjustments we made to household identifiers based on extensive investigation and a series of checks to ensure the quality of the adjustments. Our approach does not overcome the problem of duplicate person identifiers for linking across BMS files.

Despite our efforts, BMS linkage rates in the 1976 to 1988 period are lower than observed in the 1989 forward period [2]. Some linkages are completely impossible. Particularly complex is 1977, in which linkages between basic monthly-only and supplement-containing months are not possible, though some 1977 months can be linked to adjacent years. Basic monthly-only months in 1977 may only be linked to 1976 and the supplement-containing months in 1977 may only be linked to 1978. Extending existing Census Bureau documentation about linking problems between 1985 and 1986, we find that the break occurs between June and July of 1985; no months may be linked across the break, though linkages are possible prior to and following the break.

We also created linkages between March BMS and ASEC records. The creation of MARBASECID(P) allows us to add CPSID(P) to the ASEC and enable researchers to use ASEC data linked to any month of the CPS. We expected the ASEC to contain all individual records in the March BMS, but this was not always the case. In addition, we encountered problematic identifiers; to address this issue, we implemented a multi-stage process to identify matches between the March BMS and the ASEC and make information about the merge stage available to researchers in the IPUMS CPS variable MARBASECSTAGE. We were unable to link the 1977 March BMS and ASEC, meaning that the 1977 ASEC cannot be linked to any other months of CPS data using CPSID(P). Though no documentation clarifies the exact reason, we strongly suspect that this is related to the “D sample” which was added to the CPS for the purpose of measuring the efficacy of a jobs training program (see more information in Appendix A).

Our approach for creating CPSID(P) and MARBASECID(P) in the 1976 to 1988 period deviated from that employed in the 1989 forward period. In general, we followed the rules elaborated previously [2] for creating CPSID(P) while addressing the challenges described above. Creating linkages between the March BMS and ASEC and constructing MARBASECID(P) in 1976 to 1988 deviated considerably from our approach in 1989 forward. Using the same logic in 1976 to 1988 as we did with the more recent data misses many linkages, which reduces the value of being able to use the ASEC in combination with monthly CPS data. We have demonstrated that our multi-stage linking approach is superior to the single-stage approaches. It strikes a balance between maximizing the number of possible links while applying a consistent linking algorithm across years from 1976 to 1988.

In short, this work serves as a resource for the community of researchers who wish to leverage the underutilized panel component of the CPS. It sheds light on some of the mysteries encountered when attempting to systematically link CPS data across months for a period of more than two decades. It also serves to catalog the challenges we encountered and solutions we employed when creating MARBASECID(P) and CPSID(P) in the 1976 to 1988 period. With this information, researchers can make informed choices when balancing the risks and rewards associated with leveraging the panel component of the CPS back to 1976.

Footnotes

IPUMS is an organization that provides census and survey data from around the world integrated across time and space. IPUMS originally stood for Integrated Public Use Microdata Series, but as of 2016, we no longer treat IPUMS as an acronym given the growth in our data collection beyond microdata and access conditions in some instances that limit usage. For additional information about IPUMS, please see https://ipums.org/what-is-ipums.

We use MIS to indicate month-in-sample; this information is contained in the IPUMS CPS variable MISH.

Hereafter, we use CPSID(P) to refer to both CPSID, the household-level identifier, and CPSIDP, the person-level identifier.

A note on terminology: Prior to 1989, there are often variables without proper variable names in codebooks; rather, variables were referenced as “items” such as “Item 18A. Line no.” For convenience, we refer to the variables that should uniquely identify households and persons within households in the original census CPS files as HRHHID and HUHHNUM and LINENO, respectively. These are IPUMS CPS variable names.

This is described in detail elsewhere [].

BLANK3 is found in columns 25–27 in the original data. Only the first two columns, 25–26 are used to uniquely identify households; the third column is blank for all records. BLANK6 is found in column 107 in the original data.

We used an inductive approach to replace spaces and ‘ $-$ ’ with numbers taking care to ensure that the numbers we selected would result in the same number of unique HHID values before and after the character swap. Converting HHID to be numeric does not change the number of linkages that result; numeric characters are simpler for researchers to use.

We looked for the individuals in adjacent months with problematic HHIDs; based on our analysis, we replaced this character with a 4.

The details of generating MARBASECID(P) for the 1989 forward period are described elsewhere [].

This is based on a conversation with staff at the U.S. Census Bureau, not official documentation.

See Appendix B for more information on versions of the person identifier in 1984–1988. Only one version of the line number variable exists in the ASEC files.

As a result of this data quality issue, it will not be possible to distinguish ASEC oversample records from unlinked March basic monthly records in 1976–1988 and those under the age of 14 in 1976–1981.

We generate the count of individuals per household used for linking (NUMPER) based on HHID values. It may differ from the IPUMS CPS variable NUMPREC for some households in the ASEC files during this period. It is necessary to use this generated count rather than the variable from the original data file or NUMPREC due to the fact that some households as defined by HHID are split across multiple households as defined by HHSEQ in the ASEC files. For more information on split households in the ASEC files, see Appendix C.

Supplementary data

The supplementary files are available to download from https://dx-doi-org.web.bisu.edu.cn/10.3233/ JEM-210480.

References

Flood

King

Rodgers

Ruggles

Warren

. Integrated Public Use Microdata Series, Current Population Survey: Version 8.0 [dataset]. Minneapolis, MN: IPUMS, 2020. doi: 10.18128/D030.V8.0.

Drew

Flood

Warren

. Making full use of the longitudinal design of the Current Population Survey: Methods for linking records across 16 months. Journal of Economic and Social Measurement. 2014; 39(3): 121-144.

Flood

Pacas

. Using the annual social and economic supplement as part of a current population survey panel. Journal of Economic and Social Measurement. 2017; 42(3-4): 225-248.

U.S. Census Bureau. Current Population Survey Design and Methodology Technical Paper 66. 2006. Washington, D.C. Government Printing Office.

U.S. Census Bureau. Current Population Survey, March 1994 Technical Documentation [codebook]. 1994.

McIntire

. The Mechanics of Matching CPS Microdata: Problems and Solutions. Current Population Survey as a Longitudinal Data Base. Bureau of Labor Statistics. 1980; 5-6.

Creighton

Wilkinson

. Redesign of the sample for the current population survey. Employment and Earnings. 1984; 31(5): 7-10.

Sojourner

Pacas

. The relationship between union membership and net fiscal impact. Industrial Relations. 2019; 58(1): 86-107.

Thrul

Riehm

Cohen

Alexander

Vernick

Mojtabai

. Tobacco Control Policies and Smoking Cessation Treatment Utilization: A Moderated Mediation Analysis [Preprint]. 2020 [cited 2021 January 15]; 25. Available from doi: 10.1101/2020.10.16.20213900v1.

10.

Hanson

. The Current Population Survey: Design and Methodology, Technical Paper 40. 1978; Washington, D.C. Government Printing Office.

11.

U.S. Census Bureau. Current Population Survey, October 1986: School Enrollment [codebook]. 1987.

12.

U.S. Census Bureau. Current Population Survey, October 1985: School Enrollment [codebook]. 1986.

Extending current population survey linkages: Obstacles and solutions for linking monthly data from 1976 to 1988

Abstract

Keywords

1. Introduction

2. CPS rotation pattern and linking

3. CPSID(P)

4. Creating CPSID(P) for basic monthly files

4.1 Household identifiers and linking across months

4.1.1 Non-unique household identifiers

4.1.2 Non-numeric household identifiers

4.1.3 Unlinkable months

4.2.1 Non-unique person identifiers

5. Basic monthly survey linkage rates across time

Table 1 Number of people responding to the CPS, by calendar month, month-in-sample group, and year

6.1 Missing March basic monthly records in the ASEC

Table 8 Sample size and retention rate, CPS respondents in month-in-sample 4 linked to month-in-sample 5

6.2.1 Difficulty uniquely identifying records

7. Strategy for linking March basic monthly to ASEC

7.1 Preliminary steps

7.2 Linking methods

Table 9 Multi-stage merge linking keys, 1976–1988

9. Validating March basic and ASEC linkages

Table 13 Validation on demographic characteristics broken out by merge stage

Footnotes

Supplementary data

References

Table 1
Number of people responding to the CPS, by calendar month, month-in-sample group, and year

Table 8
Sample size and retention rate, CPS respondents in month-in-sample 4 linked to month-in-sample 5

Table 9
Multi-stage merge linking keys, 1976–1988

Table 13
Validation on demographic characteristics broken out by merge stage