Evaluating Alternative Migration Estimation Techniques for Population Estimates and Projections

Introduction

Timely population estimates are critically important to planners and policy makers at all levels of government, and yet the planning methods literature has paid little attention to how these estimates are produced or their accuracy. Population estimates can be found in nearly every planning and policy document that either describes past population trends or uses an extrapolation-based approach to project future population levels. These projections, in turn, help determine anticipated demand for resident services and infrastructure—schools, roads, water treatment facilities, and health services, to name but a few (Rayer 2008). The population estimates produced by the U.S. Census Bureau’s Population Estimates Program also provide the control totals for other common data series, such as the American Community Survey and the Current Population Survey, and are used to construct other official statistics such as per capita incomes, unemployment rates, and mortality rates (Long 1993). Finally, population estimates are used to distribute billions of federal program dollars each year. The General Accounting Office estimates that in fiscal year 2008 the federal government allocated more than $300 billion in formula-based funds that were at least partially allocated according to Census population data (Goldenkoff 2009). A study conducted by the Census Bureau using 2007 values estimates that amounts closer to $440 billion, with roughly 46 percent specifically listing “population estimates” or “the latest census data” as a criterion used in funding formulas (Blumerman and Vidal 2009). ¹

Given the high stakes involved in the uses of population estimates within the planning domain, attention to both the accuracy of estimates and the evaluation of alternate estimation methods is critical. Although there are numerous ways to model population change, the most widely used demographic approaches for both estimation and projection share a common foundation in the component-of-change framework. Under this framework, population change can only occur through one of four components: births, deaths, in-migration, or out-migration (Shryock, Siegel and Associates 1973). We may also classify migration into whether international or domestic. Among these components of change, domestic migration is often considered the most volatile and often dominates population change, especially in fast-growing or declining areas (Klosterman 1990; Smith 1986). Such volatility also makes the domestic migration component notoriously difficult to estimate with accuracy. As such, our ability to produce reliable population estimates and forecasts hinges on our ability to getting the domestic migration component right.

Planners are typically more familiar with component-based models for population projections than with annual population estimates. While this paper examines migration in the context of estimates, not projections, its evaluation applies readily to both. The primary difference between estimates and projections is that projections typically divide the population by age and sex cohorts (i.e., the cohort-component approach), whereas annual population estimates typically do not. This greatly increases the burden of data gathering in cohort-component approaches, which require collecting vital statistics (birth and death) and migration data specific to each age-sex cohort. This level of detail is often lacking for small sub-state areas. Detailed counts of annual births and deaths are usually procured from state vital records offices. However, some states, such as Massachusetts, suppress reporting by age and sex cohorts to avoid potential disclosure violations in sparsely populated counties. Even if released, one must always exercise caution deriving estimates from data with few observations. Procuring small-area migration data is typically a bigger problem than vital statistics records. Internal Revenue Service (IRS) migration statistics on county inflows and outflows are not classified by age and sex. The summary files of the American Community Survey (ACS) break down the total population by age and sex, but not the population of in- and out-migrants. Cohort-specific migration rates can be estimated from the ACS Public Use Micro-sample (PUMS) files, but the spatial units reported in the PUMS files (i.e., PUMAs) typically do not conform to county boundaries.

Another major difference is the length of the forecast horizon. For population estimates, the forecast horizon is typically a single year, possibly two. In fact, current-year population estimates are actually short-term forecasts because the data required to produce them typically lags the estimation period by a year or two. For projections, forecast horizons are much longer—five, ten, twenty, even thirty years. This longer horizon, in turn, may warrant different assumptions of whether and how past trends for each component are likely to continue into the future.

This article evaluates demographic approaches to subnational population estimates, with an eye toward improving the accuracy of population estimates, and by extension, population projections. The greater simplicity of the population estimates model is actually a virtue for developing an effective evaluation strategy. With fewer moving parts, it is far easier to isolate a single component of the estimates model, such as migration, and evaluate its solitary impact on forecast accuracy and bias under different assumptions. The shorter time horizon also allows us to evaluate our estimates against real population counts—something rarely done for population projections. For this reason, an evaluation strategy based on population estimates can help us to better understand and ultimately improve the models we used to project regional population change as well.

Our evaluation focuses on the domestic migration component of population change. We consider three alternate approaches to modeling domestic migration: (1) a net migration approach closely resembling the method currently used by the Census Bureau’s Population Estimates Program; (2) a biregional gross domestic migration approach that uses a single rest-of-the nation source as the baseline population for prospective in-migrants; and (3) a multiregional gross migration method that calculates in-migration rates between pairs of counties. We construct annual estimates for nearly every county in the United States from 2000 to our target year of 2010 and use the 100 percent population count of the 2010 Decennial Census as an objective baseline to assess which of the three approaches produces the most accurate estimates according to our review of five evaluation metrics.

We conclude with a preliminary investigation into the county characteristics associated with greater or lesser estimation error or bias. No estimation method is likely to produce superior estimates in all circumstances. Rather than simply picking the method that produces the lowest average error across all types of counties, it is perhaps more valuable for practitioners to know which method is more appropriate to their region. For example, past studies show that prediction error is often highest for small counties as well as those experiencing rapid growth. Given these circumstances, a method that works well for small and/or fast-growing counties might be preferred over a method that has smaller error averaged over all counties in the nation. In addition to size and growth, we examine a number of sociodemographic characteristics that may be also associated with greater error—elderly populations, persons living in poverty, group quarters residents, and college students.

Alternative Migration Estimation Techniques

There are many options for estimating the domestic migration component of population change—potentially leading to vast differences in population estimates and projections. The Census Bureau and many state and local analysts typically estimate domestic migration using a net migration rate approach. Under this approach, the migration rate is measured simply as the number of in-migrants minus the number of out-migrants, divided by the same-county population at the start of the prior year.

Despite its simplicity and intuitive appeal, net migration has a serious conceptual flaw. The problem lies in the estimation of a combined (or net) migration rate for in- and out-migrants (Isserman 1993; Rogers 1990; Smith 1986). Ideally, the migration rate should measure the probability of any individual moving into (or out of) a county. Therefore, the denominator of the migration rate must represent the population “at risk” of migration. This is a reasonable assumption for out-migration—a county’s residents in the previous year could potentially move out—but not for in-migration (Plane and Rogerson 1994). The existing residents of a county are not at risk of moving into the county—they already live there. In fact, the population at risk of moving into a county is literally everyone except its existing residents.

While this may seem like a trivial distinction, it has the potential to create large differences in population estimates and forecasts by making growing places seemingly grow faster and declining places decline faster (Isserman 1993; Rogers 1976, 1990; Smith 1986; Smith and Swanson 1998). Consider the case of the fast-growing area. Such places are characterized by a population growth rate that exceeds the nation, typically due to relatively higher in-migration than out-migration. Positive net migration contributes to a growing baseline population. The growing base produces greater numbers of net positive migrants, which, in turn, leads to an increasingly larger base. Yet, the “true” population at risk of in-migration comes not from the local area, but from the rest of the nation where population growth is more modest. The result is a cumulative and self-reinforcing process that produces increasingly biased projections with each passing year under the net-migration model. ² A more theoretically valid approach is to estimate separate in- and out-migration rates and apply each to the appropriate at-risk population—existing same-county residents for out-migration and residents of the rest of the nation for in-migration. This is known as the biregional gross migration approach.

Gross migration–based estimates might be further improved by modeling the flow of migrants between specific pairs of counties, rather than constructing a single in-migration rate for all residents living elsewhere in the nation. This type of multiregional gross migration model also acknowledges that domestic migration is often dominated by a relatively small number of origins and destinations (typically large and fairly nearby counties), and presumably would lead to models that are more sensitive to changing population dynamics of highly interconnected areas.

In an applied use, the prospective benefits of implementing a particular migration estimation methodology must be weighed against the costs. The argument favoring the net-migration approach is one based mainly upon practical considerations: the realistic time, data, and other resource constraints facing professional planners and demographers (Smith and Swanson 1998). The biregional gross migration method is only marginally more difficult to compute than the net migration model. It only requires the additional steps of splitting the migration component into two distinct calculations (in and out), collecting population component data at the national level, and subtracting the values for the target county to produce an in-migration rate using the rest of the nation as the base. Building a true multiregional model, however, is both data and computationally intensive (Sweeney and Konty 2002). It requires collecting data on the components of change for all U.S. counties, as well as a modest level of programming skill to connect the flows of migrants from one county to another.

It is also not entirely clear that gross migration models offer sufficient improvement in accuracy to warrant the increased time and cost (Smith and Swanson 1998; Tayman and Swanson 1996). While there is a strong conceptual argument favoring gross migration, there has been relatively little evaluation to document which method produces the most accurate population estimates or forecasts. There are several studies that examine the divergence of long-term forecasts produced under different migration assumptions against one another, but not against actual Census counts. Overall, they show that net and biregional gross migration methods produce fairly similar estimates over relatively short time spans (Smith 1986) but diverge the as the forecast horizon expands (Wilson and Bell 2004). The few studies that do compare estimates or projections directly against actual Census population counts are limited to a relatively small number of places and do not consider a multiregional gross migration model in their evaluation. Nor do they profile the sociodemographic characteristics of counties where one model may work better than another. Nevertheless, their results do suggest that the biregional gross model approach yields more accurate results, overall (Isserman 1993; Smith and Swanson 1998). Smith and Swanson (1998) also find that both biregional and net models tend to overpredict the population of fast-growing states, and that this upward bias is considerably higher for the net migration model.

This study extends previous work by evaluating annual population estimates using a far more extensive database—one covering nearly all counties in the United States—while testing the three different domestic migration scenarios: the net migration method currently used by the U.S. Census Bureau and most states; a biregional gross migration model similar to those considered by Smith and Swanson (1998) and Isserman (1993); and a multiregional model of intercounty migration flows that has not previously been compared to actual population counts. We also use a more extensive set of evaluation metrics, including not only standard measures of error and bias but also a model’s propensity to produce extreme outliers and its ability to produce more accurate estimates for a larger number of counties. Lastly, we offer a preliminary investigation into the correlates of bias and error, considering not only size and recent population growth but also demographic characteristics such as the share of the elderly and college student populations.

Calculating Net and Gross Migration Rates

Our population estimates follow a standard components-of-change framework, mirroring that used by the U.S. Census Bureau (2010) for the production of its annual (July 1st) Population Estimates series. ³ Our basic population balancing equation is:

Pop i , t = Pop i , t − 1 + Births i , t − 1 − Deaths i , t − 1 + NetIntMig i , t − 1 + NetDomMig i , t − 1 ,

where in a given year (t) the population of jurisdiction (i) can only change from its prior year value (t – 1) through births, deaths, net international migration (NetIntMig), and/or domestic migration (NetDomMig). This study is mainly concerned with different specifications of the domestic migration component of population change. Estimates for births, deaths, and international migration are exogenous to our models, using figures reported by the U.S. Census Bureau population estimates program. As only the domestic migration component varies across our different model specifications, the remaining components of change are essentially “held constant” across the different model specifications.

There is no direct source of information on the number of domestic migrants coming into or going out of a county. If there were, then domestic migration could be entered directly into the population equation and there would be no need for alternate estimation methods. Instead, domestic migration must be estimated from migration rates developed from symptomatic indicators, which are then multiplied against the population at risk of migration from the previous year. We estimate migration rates using Internal Revenue Service (IRS) data on the total number of tax exemptions (i.e., filers and their dependents) that changed the address of their primary county of residence from the previous year’s tax return. IRS exemptions cannot be used as a direct measure of migration. Not everyone files a tax return, and some are new filers that cannot be matched to an earlier return. However, it is assumed that the rate at which exemptions move between counties are representative of actual migration patterns.

Under the net migration approach, domestic net migration (NetDomMig) is estimated by first calculating a net migration rate and then applying this rate against the base population of potential migrants (i.e., the “at-risk” population), or:

NetDomMig i , t − 1 = ( InEx i , t − 1 − OutEx i , t − 1 TotEx i , t − 1 ) AtRisk i , t − 1 .

The net migration rate is simply the number of exemptions moving into county i (InEx) during year t – 1 minus the number exemptions moving out of county i (OutEx) divided by the total number of exemptions (TotEx). In this case, the at-risk population is estimated as the number of same-county residents plus half of the births, deaths, and net international migration that occurs during t – 1, ⁴ or

AtRisk i , t − 1 = Pop t − 1 + .5 ( Births i , t − 1 − Deaths i , t − 1 + NetIntMig i , t − 1 ) .

The biregional gross migration approach explicitly distinguishes the forces generating domestic migration and includes separate elements for in-migration (InDomMig) and out-migration (OutDomMig) in the balancing equation, or:

Pop i , t = Pop i , t − 1 + Births i , t − 1 − Deaths i , t − 1 + NetIntMig i , t − 1 + InDomMig i , t − 1 − OutDomMig i , t − 1 .

Calculating gross out-migration is very similar to calculating net migration. It is simply the share of IRS exemptions moving out of the target county i between the previous and current year multiplied by the at-risk population, or:

OutDomMig i , t = ( OutEx i , t − 1 TotEx i , t − 1 ) AtRisk i , t − 1 .

Calculating the gross in-migration rate is a bit more complicated, as the population at risk of moving into the target county i are persons living elsewhere in the United States. In our biregional gross migration model, we calculate gross in-migration by treating all potential counties of origin as a singular “Rest of the Nation” (RON), or:

InDomMig i , t = ( InEx i , t − 1 TotEx RON , t − 1 ) AtRisk RON , t − 1 ,

where InEx is the number of IRS exemptions moving into the target county i during the past year from all other counties (RON). The total exemptions (TotEx) used in the denominator of the in-migration rate and the components of the migration base are constructed by summing over all counties, excluding i.

A Multiregional Model of Gross Migration Flows

Gross migration estimates can be further refined if we consider migration as a dynamic process involving the flow of people between specific pairs of origin and destination counties. The landscape of domestic migration is highly uneven, with a small number of counties often accounting for a large portion of the migratory flow into or out of any given county. Distance is one important factor, as people may change houses or apartments but not their jobs or their desire to retain ties with family, friends, and other established social networks. Analyzing ACS mobility data from 2002 to 2009, Census Bureau researchers estimate that typically between 30 and 40 percent of intercounty moves take place within 50 miles of the previous county of residence (Ihrke, Faber, and Koerber 2011). The cumulative total of roughly 50 to 60 percent occurs within 200 miles. A considerable volume of migration research emphasizes the importance of employment opportunities in longer distance moves (Greenwood 1985); thus, we may also observe higher migration rates between counties with similar occupational and/or industrial profiles. Recent work also emphasizes increasing importance of quality of life preferences and amenities in the migration decision (Rupasingha and Goetz 2004). So while our biregional gross migration model may produce consistent migration rates by treating the rest of the nation as a singular entity, it may be less sensitive to changes occurring in those few counties that account for the bulk of migration flows.

In principle, at least, the gross migration framework can be easily modified to accommodate multiple origins and destinations. The basic population estimation equation remains the same as it was under the biregional gross-migration model (see equation 4). The out-migration equation also remains the same as it was in the biregional gross-migration specification (equation 5), although the number of out-migrants does change with the county’s population, which is partly a function of in-migration in earlier years. Only the equation for in-migration (equation 6) changes, as it now must be adjusted to account for flows between specific pairs of counties. In the case of in-migration, the target county i is the destination of migrants originating in county j. The in-migration from j to i (InEx _ji ) is multiplied against the population at risk of migration for each j and summed over all j to estimate the number of in-migrants into i. As a practical matter, the in-migration equation must also include a RON component because the IRS suppresses county-to-county migration estimates in cases where such flows are small. This aggregate residual component can sometimes be sizable, especially for small counties with few gross flows in either direction. The construction of the RON component in the multiregional model is similar to how in-migration was modeled in the biregional gross migration model, except it now excludes both i itself as well as all the j counties that were covered in the county-to-county flows. The resulting equation is:

InDomMig i , t = ∑ j ≠ i [ ( InEx j i , t − 1 TotEx j , t − 1 ) AtRisk j , t − 1 ] + ( InEx RONi , t − 1 TotEx RON , t − 1 ) AtRisk RON , t − 1 .

Thus, total domestic migration into target county i is estimated as the sum of in-migrants coming from all other j counties for which county-to-county flow data are available, as well as a residual component reflecting the in-migrants originating in counties that are not covered by the county-to-county flows.

Data and Estimation

In order to evaluate the three different methods for calculating domestic migration, we estimate county population change for each year from 2001 to 2010—using the 2000 Census Bureau estimates as our launch year population. Like the Census Bureau’s, our estimates only cover the resident household population and exclude persons living in institutional group quarters such as students living in dormitories, military personnel living in barracks, nursing homes residents, prisoners, etc.

Annual data on births, deaths, and international migration in the inter-censal years are taken from the Vintage 2009 County Components of Change files provided by the Census Bureau’s Population Estimates Program. Annual data on births and deaths are compiled by the Population Estimates Program, namely, from state vital statistics records. International migration is estimated largely from the American Community Survey data (U.S. Census Bureau 2010). As discussed previously, annual domestic migration rates are derived from IRS tax statistics. The most recent IRS data available at the time of this study cover moves between 2009 and 2010.

We produced annual population estimates for all counties in the United States that had consistent population and migration data over the decade. We excluded from the evaluation 29 counties that experienced major boundary changes between 2000 and 2010. While we could not produce consistent population estimates for these 29 counties, they are still factored into the migration rates and population estimates of other counties through the RON component of the gross and multiregional migration models. Their exclusion does not affect the net migration-based population estimates, which are based purely on within-county components of change.

Evaluation Metrics

We test the three migration-based population estimates (net, biregional gross, and multiregional gross) against the 100 percent household population count of Census 2010 for the 3,117 counties that had consistent boundaries over the preceding decade. Our evaluation is based upon five metrics selected to represent different aspects of estimation accuracy and bias. The root mean squared error (RMSE) and the mean absolute percentage error (MAPE) both measure the average accuracy of our estimates over all counties. The RMSE is the approximate average number of persons over- or underestimated by each method. It is calculated by (1) taking the difference of each county’s estimate (E_i) from the Census 2010 counts (C_i); (2) squaring this difference to convert it to an absolute value in order to correct for the canceling effects of positive and negative net migration values; (3) dividing by the total number of counties (n); and (4) taking the square root to covert the mean squared error back into population units or:

RMSE = ∑ i ( E i − C i ) 2 n .

Because it is based upon the absolute magnitude (and not the rate) of error, the RMSE tends to be more heavily influenced by prediction error in larger, more populous counties.

The MAPE is another common metric of average prediction error. Like the RMSE, the MAPE also corrects for the canceling effects of positive and negative net migration values, this time by taking the absolute differences between estimated and actual population counts, or:

MAPE = 1 n × ∑ i ABS ( E i − C i ) C i .

Because it is based on percentages, the MAPE is more likely to be influenced by less populous counties that are more prone to larger percentage errors even if the absolute number of miscounted persons is rather small.

To measure bias, we include the mean algebraic percentage error (MALPE). The MALPE is the mean percentage error taken over all counties, or:

MALPE = 1 n × ∑ i ( E i − C i C i ) .

The MALPE is similar to the MAPE in that it is based on percentages and not actual counts, and likewise tends to be more influenced by smaller counties. But unlike the MAPE, the MALPE does not account for the cancelling effects. It is most commonly used to indicate the direction and relative magnitude of over- or under-prediction.

The RMSE, MAPE, and MALPE are all measures of the average or typical error. Yet, a model that produces a slightly higher average error might actually be preferred over a model with a lower average error, if the former produces fewer cases where the actual population was missed by a large amount. Therefore, we also calculate the share of counties with an absolute percentage error greater than 10 percent over the Census count as an additional measure of accuracy. Conversely, a small number of counties with extreme errors may also heavily influence mean error-based evaluation metrics and may not properly identify which model is the best for any particular county. Therefore, we also include, as our final summary metric, the proportion of counties where each approach produced the most accurate estimates. This measure reveals how often each approach produced the most accurate estimates relative to the others. For example, if the net migration model produced a lower absolute error than either the gross or multiregional approaches in 25 out of 100 counties, we say that it was most accurate for 25 percent of counties.

Evaluation Results

All three estimation methods perform rather well, with generally similar estimates and degrees of predictive accuracy. Of the three methods, the biregional gross migration model is the preferred model in three of our five evaluation metrics (Table 1). On average, the biregional model missed the actual 2010 Census count by only 7,429 persons (RMSE)—roughly three percent of the Census 2010 count (MAPE). The net and multiregional models miss the census count by a near equal number of persons according to their RMSE values, although the net migration model comes in with a noticeably lower MAPE that is only 0.1 percentage points off that of the gross migration method. Somewhat surprisingly, the multiregional gross migration approach produced the highest MAPE of the three models—despite its greater complexity and use of more detailed data on county to county migration flows.

OPEN IN VIEWER

Table 1.

Evaluation of Alternative Migration Estimation Methods.

Evaluation Metric	Net	Biregional	Multiregional
Root mean squared error (RMSE)	8,060	7,429	8,004
Mean absolute percentage error (MAPE)	3.1%	3.0%	3.7%
Mean algebraic percent error (MALPE)	–1.3%	–1.3%	0.6%
Extreme percent error (±10%)	3.1%	2.7%	5.5%
Share of counties, most accurate a	27.1%	31.0%	41.4%

Note: Preferred value for each metric is in bold.

a

There were 17 counties where two methods produced identical estimates, resulting in ties for the most accurate: 13 counties where gross and net estimates were tied and 4 where the multiregional and biregional migration models were tied.

According to the MALPE (our measure of bias), both the net and biregional gross migration models tend to err on the side of underprediction, while the multiregional model is prone to slight overprediction. This contrasts with Smith and Swanson (Smith and Swanson 1998) who found that the net and biregional models both overestimated Census counts, with the net migration model producing considerably higher estimates of the two. We find essentially no difference in the relative bias of these two methods.

The biregional and net migration models not only have similar degrees of estimation error and bias, they also over- or underestimate the same set counties. The Pearson and Spearman rank-order correlation coefficients between the net and biregional models are both above .97 (data not shown). The pattern of errors produced by the multiregional model is more distinct. It has only modest positive correlations with the other models, with Pearson and Spearman coefficients in the .23 to .50 range.

While the multiregional model also generally tends to over- or underestimate the same counties as the others, the magnitude and pattern of over- and underprediction is not as consistent. There are numerous examples where the multiregional model overestimated population in counties that were underestimated under the net and biregional models, and vice versa. Whereas the net and biregional models produced over- or underpredictions uniformly for nearly all counties (96%), the multiregional model agreed with the bias of the other two models only approximately 60 percent of the time. However, relatively few of these instances of divergent predictions involve relatively large errors; all but 8 percent are within ±5 percent of the actual census count.

The biregional gross migration model also produces the fewest “extreme” errors, while the multiregional migration model produces far more. Nearly 5.5 percent of the multiregional estimates missed the Census 2010 benchmark by ±10 percent, whereas fewer than 3 percent of counties had errors this large when based on the net or the biregional migration approach.

Our final evaluation metric identifies the method that had the lowest absolute error for each county and then tallies that number across all counties. The results were rather unexpected. Despite its higher average error, the multiregional approach produced the most accurate estimates for the greatest number of counties. In just over 40 percent of the cases, the multiregional model produced the most accurate population counts. The biregional model comes in second, producing the most accurate estimates in 31 percent of counties (slightly higher if you consider ties). This suggests that the higher average error found for the multiregional model may not be systemic across all counties, but rather heavily influenced by counties with extreme errors (>±10%). If these extreme outliers are removed, the MAPE for the multiregional model is reduced to 3.4 percent—slightly higher but far more consistent with the other two models.

Investigating the Characteristics Associated with Estimation Error and Bias

We close our study with an exploratory investigation of some of the sociodemographic attributes of counties associated with accuracy and bias in each of the three models, drawing upon previous studies and our own critical assessment of the data sources to help guide and interpret our analysis.

Population size and growth are both known to influence forecast accuracy (Rayer 2008; Smith 1987). Small areas are prone to greater error, in part because they experience more erratic year-to-year swings in their change components. Rapidly growing and declining places are also difficult to predict. It is widely believed that the net-migration approach will exaggerate growth in fast-growing areas and population loss in declining areas relative to gross migration approaches (Isserman 1993; Rogers 1976, 1990; Smith 1986; Smith and Swanson 1998). Fast-growing regions are generally more likely to have higher in-migration relative to out-migration. And if we accept that the entire nation is a more accurate reflection of the population at risk of in-migration, then net-migration-based estimates will be upwardly biased because the true pool of potential in-migrants is not growing as fast as assumed by the model. The opposite holds in the case of declining regions. Smith and Swanson (1998) offer some evidence that net-migration-based estimates are indeed inflated relative to the biregional gross migration model. However, their study is limited to ten states and uses a different source for migration data (the 1980 Census). Thus, the relationship between growth, accuracy, and bias warrants additional investigation.

Unlike previous studies, we include measures representing the socioeconomic characteristics of groups most likely to be undercounted by IRS data. These groups include the elderly (modeled here as the population 65 years and older), many of whom are not required to file income tax returns if their sole source of income is Social Security. The elderly are also typically less mobile than younger age cohorts. While the elderly are underrepresented in IRS exemption data, they are included in our estimates of the at-risk population. Therefore, we expect counties with large shares of elderly residents to have upwardly biased in- and out-migration rates. This will result in greater estimation error, but it is difficult to predict whether this would lead to over- or underestimation, on balance. The poor (represented by county poverty rates) are also less likely to file tax returns and are generally less likely to relocate long distances than more affluent households. College students are also likely to be poorly represented by tax-based migration data and thus also may contribute to greater error. Many college students file as dependents on their parent’s tax return or identify their parent’s home as their place of residence, while the Census often records their county of primary residence as where they attend school.

Lastly, we consider areas with a large share of the population living in the non-college Group Quarters (GQ) as possibly having more erroneous estimates. ⁵ While the GQ population is not included in our estimates, they still may constitute a source of error because they do factor into the calculation of the migration rate. Some GQ residents file tax returns, although presumably at far lower filing rates than the general public, and persons living in GQs are typically less mobile, given that this population includes residents of nursing homes, prisons, orphanages, and the like. This makes IRS-based migration rates lower than they would be if GQ residents were excluded from tax statistics, particularly in counties with a large share of GQ residents. The lower rate then gets applied to the household-only population, resulting in fewer numbers of in- and out-migrants.

Our investigation proceeds by grouping counties according to their socioeconomic characteristics, calculating evaluation metrics within each group, and examining patterns and trends in the magnitude and direction of error across groups. For the sake of efficiency, we focus our attention on four evaluation metrics: the MAPE, the MALPE, the share of counties with estimation error in excess of 10 percent, and the share of counties where each model performed best. We classified counties into four population-size categories and five growth categories. For the remaining characteristics, we divided all counties equally according to the quartile values of the characteristic variables. Our results are summarized in Table 2.

OPEN IN VIEWER

Table 2.

Evaluation of Model Accuracy, Bias, and Fit by County Socioeconomic Characteristics.

Percentile/Class	MAPE			MALPE			Error (>10%)			Percent Best Fit
	Net	Biregional	Multiregional	Net	Biregional	Multiregional	Net	Biregional	Multiregional	Net	Biregional	Multiregional
Population in 2000
<20k	4.1	4.0	4.3	–2.3	–2.3	0.5	5.9	5.3	8.2	23.2	28.8	47.3
20k to 100k	2.5	2.4	3.3	–0.9	–0.9	0.7	1.3	1.0	3.9	28.7	31.3	39.5
100k to 500k	1.8	1.7	3.2	–0.2	–0.1	0.7	0.2	0.2	2.7	35.2	35.7	29.1
≥500k	1.9	1.7	2.4	0.8	0.7	0.0	0.9	0.9	2.7	25.0	37.5	37.5
Population growth rate, 2000 to 2010
<–10%	5.0	4.7	5.4	–0.8	–0.8	3.1	10.7	9.3	13.7	20.0	36.1	42.0
–10% to 0%	3.0	2.9	3.1	–1.5	–1.4	1.1	3.0	2.1	3.5	22.5	30.7	46.1
0% to 10%	2.7	2.6	3.2	–1.3	–1.3	0.2	1.0	1.0	3.0	27.7	29.9	41.8
10% to 25%	3.0	3.0	4.1	–1.3	–1.4	–0.3	3.3	3.4	8.9	32.4	30.5	37.0
≥25%	3.8	3.6	5.5	–1.2	–1.2	1.2	7.7	7.2	10.9	34.4	34.8	30.8
65 years of age and older
<25th	3.1	3.0	4.9	–0.4	–0.4	0.7	4.4	4.0	10.4	35.0	35.3	29.4
25th to 50th	2.6	2.5	3.4	–0.7	–0.7	0.9	1.4	1.3	4.4	25.3	35.2	38.9
50th to 75th	2.8	2.8	3.0	–1.7	–1.7	0.4	2.4	2.3	3.1	25.2	27.9	46.3
≥75th	3.6	3.5	3.4	–2.5	–2.5	0.5	4.1	3.3	4.2	22.8	25.8	50.8
Poverty rate
<25th	2.3	2.1	3.7	–0.7	–0.4	3.1	1.0	0.8	3.3	31.3	40.4	28.2
25th to 50th	3.0	2.8	2.8	–1.6	–1.4	0.9	2.7	2.1	2.6	19.1	31.2	49.2
50th to 75th	3.2	3.1	3.2	–1.8	–1.9	–0.4	3.1	2.8	4.2	25.4	25.5	48.1
≥75th	3.8	3.8	4.8	–1.1	–1.4	–1.1	5.5	5.3	11.9	32.4	26.9	39.9
Enrolled in college
<25th	4.1	3.9	4.0	–2.4	–2.3	0.8	5.8	5.1	6.4	22.0	27.7	49.3
25th to 50th	3.1	3.0	3.6	–1.4	–1.4	0.9	2.6	2.6	4.9	25.8	31.5	42.5
50th to 75th	2.7	2.6	3.5	–0.8	–0.7	1.7	2.2	1.8	3.9	29.9	32.1	37.5
≥75th	2.4	2.3	3.5	–0.8	–0.8	–0.9	1.8	1.4	6.9	30.6	32.8	36.2
Population in group quarters, no college
<25th	3.3	3.2	4.5	–0.7	–0.8	1.4	4.0	3.6	7.6	32.9	32.9	34.1
25th to 50th	2.6	2.6	3.5	–1.1	–1.1	0.6	2.1	1.8	5.0	27.3	34.1	37.7
50th to 75th	2.7	2.6	3.1	–1.2	–1.2	0.6	1.7	1.4	3.7	24.9	30.6	44.2
≥75th	3.6	3.5	3.5	–2.3	–2.2	–0.2	4.6	4.1	5.8	23.2	26.5	49.4

Note: Preferred value for each metric is in bold. MAPE = mean absolute percentage error; MALPE = mean algebraic percentage error.

Discussion

Domestic migration is arguably the most dynamic component of population change for most counties, yet there remains some debate over the best method for estimating the domestic migration component of population estimates and projections. This study evaluates the accuracy of county-level population estimates produced under three different approaches for estimating domestic migration: (1) a net-migration approach, similar to that used by the U.S. Census Bureau and many state data centers; (2) a biregional gross migration approach where in- and out-migration are estimated separately in the population change equation; and (3) a multiregional gross migration approach that tracks the flows of migrants between specific pairs of counties to estimate in- and out-migration.

Overall, our results favor the biregional gross migration–based method as a general approach for estimating population change. The biregional model produces lower average prediction errors than either the multiregional approach or the net migration approach commonly used in practice. The biregional model seems like a particularly good candidate for the U.S. Census Bureau and for agencies in large states or regions that must use a single methodology to estimate population for many different types of areas. It is also less prone to extreme estimation errors, making it appealing in circumstances where the costs of large misses are high.

Despite its use of more detailed data and increased complexity, the multiregional model had the highest average error (MAPE) of the three methods tested. Yet our results do not completely rule out the value of a multiregional approach. The MAPE of the multiregional model is still within a single percentage point of the top performing biregional model, is less prone to consistent under-estimation than the net and biregional, and produces more accurate estimates than either the biregional or net-migration model for the greatest number of counties. But when the multiregional model misses the mark, it tends to miss by a lot—resulting in more counties with extreme errors and raising its overall prediction error.

Although an entity like the Census Bureau may opt to select a method that produces the lowest average error across counties of all types, regional planners and state demographers might do better knowing which estimation method is most appropriate to the particular circumstances faced in their jurisdiction. For example, the models that do well for growing areas may not be the same as those experiencing recent decline. We take a preliminary look at this issue through an exploratory analysis of the association between estimation error and a variety of socioeconomic indicators.

Small, fast-growing, and rapidly declining counties are notoriously difficult to estimate—all three models work best for larger counties that are relatively stable. However, when we compare the three models against one another, we find the multiregional approach produces more accurate estimates for a greater number of small counties and in cases of slow growth or recent population decline. These counties are also often characterized by large elderly populations, more group quarters, and counties with higher poverty rates. In short, the multiregional model tends to do better in places where the resident population is generally less mobile. But even in these circumstances, it is still somewhat prone to extreme errors. By contrast, the biregional model nearly always has the lowest average error—regardless of the size, growth, or the demographic structure of the county considered. The biregional model also tends to produce more accurate estimates among counties experiencing rapid population growth and among those with large student populations.

The benefits of implementing a more accurate model must ultimately be weighed against the costs—namely the time and effort spent developing, implementing and testing a new approach (Smith and Swanson 1998; Tayman and Swanson 1996). On the one hand, the biregional and net-based migration estimates produce closely similar estimates—with mean absolute errors differing by less than a single percentage point. On the other hand, the costs of implementing the biregional model are also small. Both net and biregional models are easy to develop using existing secondary data sources and standard spreadsheet software. The Census Bureau or state demographic offices could easily make a switch to a biregional gross migration approach with very little disruption. The multiregional model is far more data intensive and its successful implementation would likely require a moderate degree of database programming skill. And although it produces more accurate estimates for a larger number of counties, most of these are stable and/or gradually shrinking counties. It somewhat begs the question of whether the increased complexity of the multiregional approach is warranted if it tends to work best in circumstances where migration is likely to be less important as a component of population change.