An Introduction to Multilevel Regression and Post-Stratification for Estimating Constituency Opinion

Getting the Survey Information

Estimates of opinion in small areas has to be based on survey responses. Researchers may be in the fortunate position of being able to commission original survey research. Or, researchers may have access to the raw data from an existing large survey. Finally, researchers may have access to several smaller surveys which they can combine.

If researchers are commissioning an original survey, they will be able to specify both what background characteristics are recorded, and how they are recorded. Researchers will want to ensure that background characteristics are recorded in ways that either match the post-stratification frame, or can be recoded so as to match. Sometimes this can be difficult. In the UK, information on educational attainment is often provided according to NVQ Levels, where Level 4 is equivalent to a post-secondary qualification, Level 3 is equal to the highest type of secondary qualification, and so on. Almost no one knows about these categories, and so respondents must instead be asked whether they have a university degree, or A-levels, or other named qualifications, and these responses must be recoded to match the different NVQ levels.

Any researcher commissioning an original survey will have to give thought to the necessary sample size. The answer to the question ‘how large a sample is required?’ is almost always ‘the largest you can afford’. More practically, the answer will depend on the number of small areas for which estimates are required. Kastellec et al. (2016) report that MRP produces ‘reasonably accurate estimates of [US] state public opinion using as little as a single large national poll–approximately 1,400 survey respondents’, or 28 respondents per small area. The required number of respondents will increase with the number of small areas, but it will increase at a lower rate than this. Hanretty et al. (2017) created estimates of attitudes to same-sex marriage and general left-right orientation for the 632 constituencies in Great Britain using information from between 8000 and 12,000 respondents, or between 13 and 19 respondents per small area.

A more common situation is where researchers have information from an existing large national survey. The British Election Study online panel has information from a very large number of respondents. Because it is a general purpose social scientific study, it also contains information on a large number of respondent characteristics, including characteristics that could be used in post-stratification. Researchers do, however, need to recode the information from the BES so that the values of BES variables match the values of variables in the post-stratification frame.

If researchers do not have access to a single large sample, but only multiple smaller samples, then this work of recoding variables becomes much more difficult. Where the samples have been collected at different times or using different methods, researchers may need to model these extra factors, and pick a ‘preferred’ method and time for the purposes of post-stratification.

Getting the Constituency Information

Very often descriptions of MRP do not discuss the process of gathering constituency information. This is unfortunate. The accuracy of MRP estimates can be more strongly affected by the inclusion of good constituency predictors than by the details of the post-stratification frame (Buttice and Highton, 2013; Toshkov, 2015: 459). Fortunately, researchers in the UK can benefit from the resources compiled by the British Election Studies team, who include a very helpful database of constituency-level information.

The choice of constituency variables to include will be guided by the opinion being modelled, and by the variables that previous research has identified as mattering for that opinion. Occasionally, compiling this information can be difficult if different sub-units of a country report information in different ways. The proportion of the population without a passport was a strong local-authority level predictor of the Leave vote in the 2016 EU referendum – but this information, derived in England and Wales from the 2011 census, is not available for Scottish constituencies.

Estimating the Model

Most researchers will be familiar with regression models, which I define as any model which tries to relate measures of a dependent variable, or outcome, to measures of one or more independent variables, or predictor variables, by means of an equation. Regression models exist for outcomes of various types (continuous outcomes, dichotomous outcomes, categorical outcomes), and in principle if you can model an outcome using a regression model, you can produce estimates of that outcome as part of MRP. (In practice, categorical outcomes can be tricky: Kastellec et al. (2015: 792))

Multilevel regression models are models where the parameters in the model apply to different levels. In MRP models, these levels are usually hierarchically organised. That is, the model uses information about respondents (level 1 information), but also information about the small areas in which respondents are located (level 2 information), and possibly also information about broader groupings of small areas like regions (level 3 information).

One key part of MRP models is the effect associated with each small area, which some researchers describe as a random intercept. ¹ These constituency effects are drawn from a common distribution. This allows for small area estimates to be idiosyncratic, given what we know and can measure about the people who live in them and their other characteristics. Crucially, it allows for these idiosyncrasies to borrow strength from one another. Because random intercepts are drawn from a common distribution, information about a different small area can affect our estimate of the effect associated with one respondent’s area. If evidence from another area suggests that the effect associated with that area is very large, it can mean that the distribution of area effects generally contains very large values. This might in turn mean that we estimate a larger value of the area effect for the area we started with.

Multilevel regression is common enough to feature in most statistical environments, but complex enough to present some difficulties in implementation. In addition, since multilevel regression for the purposes of MRP is often estimated through Bayesian methods, researchers who only know about frequentist methods will have to familiarise themselves with a different terminology.

Getting the Post-Stratification Frame

In the context of MRP, a post-stratification frame is a large rectangular data frame which contains, for each small area, all the possible combinations of respondent characteristics, together with either the count or the proportion of residents of each area who have those characteristics.

For example, the post-stratification frame used in Hanretty et al. (2018) contains information on the proportion of constituency residents according to gender (male/female), housing tenure (rents/owns), sector of the economy (private/public), marital status (married/not married), age group (eight different groupings), educational level (six different groupings), and social grade (four different groupings). There are therefore 3072 (2×2×2×2×8×6×4) voter types for each constituency, and 1.9 million (3072 × 632) rows in the data.

It is much easier to use an existing post-stratification frame than it is to construct one oneself. The construction of post-stratification frames has become more complicated over time, and the state of the art involves not just combining several different sources of information, but also running complicated imputation models, and using techniques (like raking, or iterative proportional fitting) familiar to survey researchers but not familiar to applied social scientists.

The first applications of MRP were able to use existing ‘analytic’ post-stratification frames made available by Census authorities. For example, Park et al. (2004) modelled presidential vote choice using four individual-level characteristics: sex, ethnicity (African-American or other), age (four categories), and education (four categories). The 1990 US Census provided the joint distribution of these four variables (‘we know from the census that there were 66,177 adults who lived in Alabama, were male, not black, aged 18–29, and did not have a high school diploma’). Census authorities can release this relatively detailed information at the level of US states because the total numbers in each cell are still large.

Where the number of individual level variables is larger, or where the ‘small areas’ are smaller, it is not possible to use these ‘analytic’ post-stratification frames. For example, the most complicated UK Census tables available at constituency level are three-way tables (e.g. approximated social grade by sex by age, Table LC6124EW).

Consequently, researchers have turned to ‘synthetic’ (researcher-created) joint distributions. These synthetic distributions can be produced in simple or elaborate ways. The simple way of producing a joint distribution is to use information on the marginal distributions of variables (which Census authorities do release at small area level), and assume that these variables are independent. Thus, the proportion of women who are aged 25–34 is equal to the proportion of women, times the proportion of people aged 25–34. This is a poor way of producing joint distributions, because important social and political variables are often associated with one another. In the UK, the proportion of people aged 55–64 with a university degree is substantially lower than the proportion of people aged 55–64 times the proportion of people with a university degree, because university education has become more popular over time. However, Leemann and Wasserfallen (2017) were able to show (by simulation and example) that if we are interested only in the small area estimate, then these errors in the joint distribution tend to cancel out.

The elaborate way of producing a joint distribution for each small area is to take some existing joint distribution (perhaps the joint distribution at national level, or the joint distribution from the survey), and ‘rake’ these joint distributions to match the known marginal distribution at the level of the small area. Earlier research stuck with Census information. Hanretty et al. (2018), for example, took information from the Census Sample of Anonymised Records, and raked these to the known constituency marginal distributions (see Appendix B of Hanretty et al., 2018).

Perhaps emboldened by Leemann and Wasserfallen (2017), more recent research has started with joint distributions which include non-Census variables like past vote behaviour, and has raked to the constituency results of recent elections. For example, Lauderdale et al. (submitted) created a post-stratification frame for the 2017 general election which included information on 2015 vote, 2016 vote, age, qualifications, and gender.

Making Predictions and Aggregating

If researchers are able to estimate a model, and able to obtain a post-stratification frame, then the last stage should be comparatively easy. All that is required is to generate predicted values for each row in the post-stratification frame using the parameter values from the estimated regression model. These might be predicted probabilities (for a dichotomous outcome) or predicted values (for a continuous outcome). These predicted values can then be multiplied by the proportion given in the post-stratification frame, and added together to give the proportion or count at the level of the constituency.

Where estimates of uncertainty are required (and estimates of uncertainty are always useful) then predicted values can be repeatedly generated using different draws from the posterior distribution of the model.

In principle, multilevel regression and post-stratification could be used to generate an estimate of opinion for the country as a whole, rather than for multiple small areas. Multilevel regression and post-stratification would therefore perform the same function as sample weighting, but would allow more complicated sets of variables to be used as ‘weights’. Some researchers have had success in using MRP in this way to correct for samples known to be unrepresentative. In principle, MRP could also be used to generate estimates of other arbitrary combinations of respondent characteristics. It would be possible to generate estimates for young home-owners, or Labour leavers, or other groups.

Questions Which Pick Out Different Things in Different Areas

Sometimes, surveys asks questions about named individuals or named organisations. If respondents to a survey are asked for their opinion on Theresa May, we can be fairly confident that they have in mind the same person, rather than some other person who happens to share the name of the Conservative Prime Minister.

Where respondents have in mind the same object, we can assume that their opinions are governed by the same process. Individual respondents may have very different views about Theresa May, but any particular respondent’s view would change in the same way if (counterfactually) they aged 10 years, or changed gender, or relocated to a majority-Conservative constituency.

Sometimes, however, surveys ask questions in ways that lead respondents to picture different individuals or organisations. Surveys might ask about rates of satisfaction with ‘your local MP’, or ‘your local police service’. It would be wrong to model these kinds of opinions using MRP, because the same process would not govern the responses given. In some areas, having voted Conservative in the last election would cause you to be more favourable towards your local MP; in other areas (most obviously, constituencies not held by the Conservative party), it would cause you to be less favourable towards your local MP.

There are borderline cases between these two extremes. Levels of trust in one’s local MP might be satisfactorily modelled using MRP, if we are prepared to claim that trust is a general personality disposition which is strongly affected by respondent characteristics, and which extends to local MPs without being strongly affected by their party label.

Unpredictable Beliefs

MRP can also go wrong when it is used to model unpredictable beliefs. In order for MRP to work, the opinion in question has to be credibly related to constituency and/or individual-level variables. Opinion regarding Theresa May can credibly be related to both constituency and individual level variables, because Theresa May is a Conservative party politician; attitudes towards Conservative party politicians are strongly structured by attitudes to the Conservative party, and these in turn are strongly structured by different social and constituency characteristics.

It would not be possible in the same way to model beliefs about the authorship of Shakespeare’s plays, not just because few people have attitudes about the authorship of Shakespeare’s plays, but because even among those people who do have strong opinions regarding Edward de Vere or Christopher Marlowe, these opinions are not obviously structured by social or constituency characteristics.

There are many cases in between these two extremes: attitudes in relation to some supermarkets, or some other types of popular culture, for example, might be strongly structured by education and class. It is hard to say in advance when the relationship is sufficiently strong to enable MRP to be carried out. In the case of the example given later, estimates which are on the face of it plausible can be constructed even when the individual level model offers a relatively poor fit to the data.

Rare Outcomes

One particular issue with MRP has emerged quite clearly in the context of election forecasting. Generally, it is difficult to use MRP to provide estimates of opinions held with low frequency (less than 1% to 2%). This may result from attenuation bias (Lauderdale et al., submitted), or from general difficulties in explaining rare events with logistic regression models (King and Zeng, 2001).

Non-Representative Samples

MRP can do well given some kinds of non-representative samples, and can do poorly when given other kinds of non-representative samples. Whether MRP does well or poorly depends on the ways in which the sample is non-representative.

If the sample is non-representative because certain characteristics are over- (under-)represented, and those characteristics are included in the post-stratification frame, then MRP can do well. Wang et al. (2015) used data from a survey of Xbox users to predict the results of the 2012 presidential election. Xbox users are not a representative sample of the population: they are much younger, and much more likely to be male. However, those demographic characteristics are present in most post-stratification frames.

If the sample is non-representative because certain characteristics are over- (under-represented), and those characteristics are not included in the post-stratification frame, then MRP will not do well. For example, suppose that an online panel has much higher levels of political interest than the general population, and that higher levels of interest are positively associated with the belief we want to study. MRP will therefore over-estimate the prevalence of the belief in question at the level of the small area, for just the same reason that a national poll would over-estimate the prevalence of the belief at the national level.

MRP can therefore be regarded as a functional equivalent to survey weighting. Like survey weighting, it depends on the variables used, the relationship between those variables and the opinion being modelled, and the relationship between unmeasured variables and the opinion being modelled.

When Demography is Destiny

I noted earlier that MRP will fail to generate good estimates of local opinion when it is asked to model unpredictable beliefs. The converse is also true: MRP will do well when asked to model very predictable beliefs. This is particularly true where demographic characteristics have a very strong relationship with the outcome in question.

The recent EU membership referendum in the UK is a good example of this. Voting behaviour in the referendum was strongly affected by education: individuals with higher educational qualifications were more likely to vote to Remain. This relationship held at both the individual and aggregate level. Because of this, even relatively simple multilevel models, which included no post-stratification element, did well, and not much less than much more complicated models (Lauderdale et al., submitted).

When There is a Past Benchmark

In the case of the EU membership referendum, there was no good past benchmark for opinion on EU membership. The previous 1975 referendum on the UK’s membership of the European Community was too long ago and counted on different boundaries. For general elections, however, past results provide a very good benchmark as to the relative ordering of constituencies. This is the reason why uniform national swing works well.

MRP models of vote behaviour therefore generally include lagged versions of party vote share as a small area level predictor. More recent MRP work has moved to using past party choice as an individual level predictor. In this way, MRP builds not just on uniform national swing, but on the alternative approach of using transition matrices (McLean, 1973). Some MRP models can therefore make claims like ‘x% of Labour Leavers stuck with the party in 2017’.

When Good and Abundant Constituency Predictors are Used

Several authors have already noted how good constituency predictors can improve the accuracy of MRP estimates considerably (Buttice and Highton, 2013; Hanretty et al., 2018; Toshkov, 2015: 459). Despite this, relatively few constituency predictors are used. This is particularly perplexing where the number of small areas is large. Consider the British case. If we were estimating a linear regression of party vote share, then one popular rule of thumb (Harrell, 2014) would suggest that we could estimate up to 62 coefficients, or 1 coefficient for every 10 constituencies. Yet applications often use far fewer constituency-level predictors, between 6 (Hanretty et al., 2018) and 14 (Lauderdale et al., submitted). This may be because we are already estimating 632 constituency coefficients drawn from a common distribution.