Abstract
Asset ownership is frequently used to assess the welfare status of households in rural areas of developing countries. Researchers often want to know the prior status of households or how that status has changed over time. In a case study in the Brazilian Amazon, we compare recall data with contemporary reports on assets from a panel survey. We consider multiple dimensions of the consistency of retrospective and contemporary data and seek to identify characteristics that lead to more accurate recall. We find that although retrospective data provide some information on past assets owned by households, they do not provide a highly accurate measure of either individual asset ownership or counts of types of assets owned. Consistent with previous studies, we find that items with greater salience are recalled more accurately. We also find that wealthier households exhibit upward bias when recalling assets owned in a previous period.
Introduction
Understanding the welfare and behavior of rural households in developing countries often requires information on the circumstances of those households in an earlier time period. For example, such data can be used to model the determinants of dynamic processes such as land cover change (Mertens et al. 2000), technology adoption (Moser and Barrett 2003), or evolution of household welfare (Deininger and Okidi 2003). Alternatively, the data may be used to generate a baseline to identify the impacts of an event such as a financial crisis (Sunderlin et al. 2001), policy reform (Uchida et al. 2009), or a household-specific event such as migration (Boucher et al. 2009).
One way to obtain data on household characteristics, behavior, or preferences in multiple time periods is to undertake panel surveys. However, panel surveys are expensive (Moser and Barrett 2003), subject to attrition (Alderman et al. 2001), and require a long-term research effort before any results are obtained (Beckett et al. 2001). Further, if an event is unexpected, it may not be possible to collect baseline data. These challenges have encouraged researchers to instead ask respondents to recall past events or circumstances of interest. However, the consequences of substituting retrospective data for true panel data are poorly understood, particularly in developing countries (Rindfuss et al. 2003).
This article focuses on retrospective data regarding asset ownership by rural households in Brazil. Asset ownership is frequently used to assess the welfare or poverty status of households in developing countries for several reasons. First, assets are not as subject to short-term fluctuations as income and consumption and therefore provide information on households’ structural income levels and underlying welfare (Filmer and Pritchett 2001). Second, it is more straightforward to measure asset ownership than alternative indicators such as household income, agricultural profit, or consumption expenditure (McKenzie 2005). Third, asset ownership can provide an indication of the vulnerability and potential mobility of households (Adato et al. 2006; Moser 1998).
Despite the importance of data on past asset ownership, there is a notable absence of evidence about whether retrospective reports provide a good measure of the assets that a household actually owned in an earlier time period. Corresponding to this, there is also a lack of information on the types of assets that can be more easily recalled, the conditions in which recall is most likely to be accurate, and the quality of recall data (especially in developing countries; Beckett et al. 2001).
In this article, we examine whether recall data on asset ownership are a valid substitute for contemporary data (collected at the time in question) in a case study in the Brazilian Amazon. Using survey data from a panel of households interviewed in 1997 and 2006, we evaluate whether retrospective data from the second wave of the survey are sufficient to characterize the wealth of households in the earlier time period. This question is important because there are clear cost savings from using recall data, and in some cases it may be the only option. In the latter situation, it is also important to identify the circumstances in which recall data are most likely to be accurate—either particular categories of assets or reports by households with particular characteristics. Finally, when using retrospective data to analyze household welfare or behavior, a critical question is whether recall errors are likely to just reduce the precision of the estimated relationships or are likely to bias results.
We begin by identifying some of the lessons from existing literature on the quality of retrospective data and describing our case study. The subsequent sections compare retrospective reports of assets to panel survey data. We measure the accuracy of retrospective data relative to the panel data in multiple ways, starting with the overall correlation between actual and recalled assets and the rate at which respondents make errors when recalling their assets. We then investigate how the accuracy of the recall data varies depending on the type of assets recalled, the household characteristics, and the data collection conditions. Finally, we draw conclusions about the extent to which and the circumstances in which recall data provide a valid substitute for contemporary reports of asset ownership.
Prior Research on Quality of Recall Data
There is a significant literature analyzing the quality of retrospective data in various contexts, for example, previous illnesses (Paganini-Hill and Chao 1993), smoking (Shiffman et al. 1997), unemployment (Dex and McCulloch 1998), and female marital and fertility history (Beckett et al. 2001).
This literature proposes various determinants of recall accuracy. First, the length of time between the interview and the event being recalled is likely to be important. Based on a review of the literature, Sudman and Bradburn (1973) find that respondents are more likely to overstate items or events when the recall period is short and are more likely to forget items or events when the recall period is long. A second factor found to consistently affect recall is the size or salience of the event or item being recalled (e.g., Mathiowetz and Duncan 1988; Neter and Waksberg 1964). Other features of the recall task that can influence accuracy include exactly what the respondent is asked to recall. For example, respondents tend to have more difficulty recalling the dates of events than their occurrence (Shiffman et al. 1997). Furthermore, a significant body of literature on the psychology of recall suggests that objective facts are more accurately recalled than subjective states or opinions (e.g., Schwarz and Sudman 1994; Tourangeau et al. 2000).
Finally, a number of studies investigate the relationship between demographic characteristics of the respondent (such as age, education, ethnicity, or gender) and accuracy of recall, with mixed results. However, the complexity of the recall task is often more important than demographic variables (Mathiowetz and Duncan 1988; Schaeffer 1994). The present circumstances or views of the respondent can also influence the way they recall events (Ross 1989).
The literature suggests a number of hypotheses for our case study. Relative to other studies, we might expect more accurate recall because asset ownership is an objective fact (rather than a subjective state or opinion) or less accurate recall because of the relatively long recall period (nine years). Across asset types in our study, we expect rare or valuable assets to be easier to recall because they tend to be more salient for the respondents. Alternatively, if respondents tend to err in the direction of cultural norms (Bernard et al. 1984), then we would expect more errors of commission for commonly owned assets and errors of omission for less common assets.
Household wealth in the later period may affect recall in two ways. First, poorer households may have better recall because, for them, all assets are more salient. Second, wealthier households may overestimate their past assets if their memory is influenced by their current circumstances. Finally, given mixed results on the role of demographic characteristics in previous studies, we test but do not have specific hypotheses about the effect on recall of demographic characteristics such as age, education, family size, and gender.
Case Study Data
To evaluate recall of assets, we use a panel data set collected in Brazil in 1997 and 2006. We interviewed a random stratified sample of 321 households in 22 communities in March 1997. In July 2006, we implemented a second wave of the panel survey, including 229 of the original households. Due to missing data on one or more assets, we use data on 222 of the 229 households interviewed in both years to characterize recall accuracy.
We use 199 observations with complete demographic data in our final regression model. Where possible, both the male and female household heads were interviewed, and the data reflect their consensus answers. In households with only one head, that individual and an adult of the other gender were interviewed, except in households with only male or only female adults. Interviews were conducted at the respondents’ homes, where many assets could also be directly observed by interviewers. The respondents live in communities on the banks of the Tapajós river in the Amazonian state of Pará, and the river is the primary form of transportation between the communities and regional market centers. Households make their livelihoods primarily from shifting cultivation, hunting, fishing, and collection of forest products (Pattanayak and Sills 2001).
The interviewers asked respondents whether at least one item in eight different categories of assets was in working condition in the house or belonged to the respondent’s family (in Portuguese: “Está funcionando em sua casa ou pertence a sua família?”) currently (in the 1997 and 2006 surveys) and nine years ago (in the 2006 survey). The respondents were not given any specific cues to assist recall of 1997 in particular. The asset categories were identified during focus groups and key informant interviews as indicators of socioeconomic status and/or important production inputs. In most cases, households owned either one or zero assets in each category. We also asked whether a household had electricity for any part of the day. Table 1 lists the assets in order of increasing frequency of ownership in 1997.
Assets and Recall Accuracy (N = 222).
Note: hh = household.
**Reject null hypothesis of no relationship at 5% significance level. ***Reject null hypothesis of no relationship at 1% significance level.
For purposes of the analysis, we treat the assets listed by the households in 1997 as the gold standard against which the retrospective reports are compared. We therefore label all variations between the two sources as “recall error.” Error in the 1997 responses is possible but unlikely because the assets in question are large, well defined, and in many cases would have been visible to the interviewer. However, there are at least two other possible explanations for discrepancies. First, if a household purchased or sold assets sometime during 1997, there would have been two potentially correct answers about asset owner in 1997. Second, changes in household composition could also explain some of the discrepancies.
Assessing Reliability
The reliability or quality of recall data can be assessed in multiple ways. We consider first the number of recall errors, divided into errors of omission (respondents forget assets that their households actually owned) and errors of commission (respondents recall assets that their households did not own). Second, we quantify the degree of association between actual assets and recalled assets with the kappa coefficient. Because we find considerable variation in recall accuracy across different assets, we next examine how recall varies by value of asset and ownership rates.
When using recall data to answer specific research questions, the degree of accuracy required depends on the planned analysis. In many cases, the researcher does not need to know specifically whether a household owned a bicycle or a radio but rather uses the data to measure changes in overall household wealth or in a household’s relative ranking within a village. This places fewer demands on the recall data. We examine the extent to which recall data can provide (1) an accurate count of total assets types, a measure of overall household wealth; (2) an accurate indication of the direction of change in asset ownership over time; and (3) an accurate measure of relative household wealth. To the extent that these measures are reliable, recall data will be useful even if there is some error in the recall of individual assets.
The extent to which recall data replicate contemporary data may also depend on the context in which the data are collected and the characteristics of the survey respondents. We therefore estimate models of the number of errors made by respondents as a function of both the characteristics of the household and the circumstances of the data collection. This allows us to identify situations in which recall data are more or less likely to correspond to contemporary data.
Overall Accuracy of Recall
On average, in 1997 respondents reported that their households owned 3.5 out of the 8 possible types of assets. When asked in 2006 to recall their asset holdings in 1997, each respondent made about two errors relative to the contemporary reports of asset ownership in the 1997 survey. As shown in Table 1, these were fairly evenly split between errors of omission and errors of commission, although with some variation as discussed subsequently.
For each asset, we measure the extent to which recalled values differ from actual values using the kappa coefficient (K; Cohen 1960):
where p o is the proportion of observed agreements and p c is the proportion of agreements expected by chance. The proportion of chance agreement is Π1Π2 + (1 − Π1)(1 − Π2), where π1 is the proportion of respondents who actually had the asset and π2 is the proportion of respondents who recalled having the asset. Thus, kappa measures the agreement between two sources of data, beyond what would occur by chance. Values closer to 1 indicate relatively strong agreement; 0 indicates the level of agreement that would be obtained by mere chance. Based on the z-tests, the relationship between contemporaneously recorded assets and recalled assets is significantly greater than would be observed by chance (Table 1).
These significance tests are important as minimum assessments of whether or not the recall data have any validity. However, the degree of association, as indicated by the magnitude of K, varies widely across asset categories. Rates of errors in recall also vary widely, from around 6% for recall of television ownership to 40% for ownership of a radio. The potential for errors of omission is bounded by the number of households that owned the asset in 1997; the errors of commission are bounded by the number who did not own the asset. There are generally more errors of omission in our data set and those are less informative for rare assets. Nevertheless, it is worth noting that for every type of asset, the majority of households (at least 59%) correctly recalled whether they had owned at least one of those assets in 1997.
Relationship between Asset Type and Error Rates
To evaluate the variation in recall accuracy, we next consider accuracy by type of asset recalled characterized by value and frequency of ownership. Table 2 contains Pearson correlation coefficients for the average value of each asset type against the percentage error rates and kappa coefficients for those asset types. It also shows the correlation between the percentage of households that own each asset in the recalled period against the percentage errors and kappa.
The high positive correlation between percentage of ownership and recall error indicates that household members are better able to recall whether they owned unusual assets as compared to assets that are commonly owned. This applies to both error of omission and error of commission, although the relationship is weaker in the latter case. The relationship between frequency of ownership and the kappa coefficient is less strong still but also suggests that there is greater agreement between actual and recalled ownership for assets owned by few households.
The difference in the degree of correlation between value or ownership frequency and recall accuracy as indicated by the two alternative measures (percentage error and kappa) reflects the biases inherent in each measure. Assets owned by a very small proportion of the sample will tend to have lower errors of omission, and correspondingly lower total error rates, regardless of how well they are recalled. Kappa coefficients have an opposite bias. Chance agreement increases toward one as the proportion with the asset goes to zero, so the possible degree of observed agreement beyond chance gets smaller. Given these effects, the actual relationship between accuracy of recall and ownership levels lies between about 0.8 (the correlation with percentage total error) and 0.4 (the correlation with the kappa coefficient), whereas the actual relationship between recall and asset value lies between −0.6 and −0.1.
The strong correlation (ρ = −.68) between asset value and ownership rates means that we cannot completely separate the influences of asset value and ownership rates on accuracy of recall. However, our results indicate assets that are widely owned and of low value tend to be recalled less accurately than rarer and more valuable assets.
Validity of Recall Data for Analysis
So far, we have considered the degree to which respondents to household surveys make errors in recalling the specific assets they owned at a particular time period in the past. However, this information is typically not collected for its own sake but for analyzing the behavior or welfare of households. Thus, in this section, we consider how errors in recall affect the interpretations that may be drawn when the data are used for different types of analysis. This allows us to determine whether some potential uses of recall data are subject to greater bias than others.
Recall data on assets are commonly used as an indicator of the past wealth levels of households. For this purpose, asset data are generally combined into some form of aggregate index (e.g., Vyas and Kumaranayake 2006). This means that the accuracy of recall of specific individual assets is less important than the combined impact of all recall errors on aggregate measures of wealth. A simple and frequently used aggregate index is the total count of assets owned by the household, or, as households typically own no more than one asset of a particular type (e.g., one television or one vehicle), a count of asset categories owned (e.g., Spears 2011). We therefore compare the total count of asset categories recalled with the total count of asset categories reported in 1997.
Recall data on assets may also be used to determine the relative position of a household in a village or other sampling frame. For example, households’ recalled asset counts may be used to categorize them as relatively wealthy or relatively poor. In this case, the validity of substituting recall data for panel data will depend largely on the degree to which any recall errors affect how households are ranked relative to each other. To evaluate the validity of our recall data for this purpose, we treat the total count of asset categories as a measure of overall household wealth and estimate the Spearman’s rank correlation coefficient between the retrospective and contemporary counts. A third potential use of recall data is to measure changes in assets over time. In this case, errors matter most if they obscure whether assets have increased, decreased, or remained constant over time.
The results in Table 3 suggest that when asset data are used in aggregate form to categorize households in terms of their absolute or relative wealth or their changes in wealth over time, recall data again show moderate correlation with actual wealth positions as reported in the earlier time period. In general, there is greater correlation in aggregate wealth than in individual asset ownership. High-value assets show slightly higher correlation between recall and contemporaneous reports. This suggests that estimates of household wealth based on retrospective data are not necessarily improved by asking about a wide range of assets. A better strategy may be to allocate more interview time (e.g., by providing cues about the recall time period or repeated questioning) to aid in the recall of a few significant assets.
Pearson Correlation between Error Rates and Asset Types (N = 222).
Impact of Recall Errors on Alternative Measures of Household Wealth (N = 222).
Impact of Household and Interview Characteristics on Recall
In the previous section, we considered which asset types are most likely to be recalled accurately. Recall accuracy may also be affected by the characteristics of the respondent household and the circumstances in which the recall questions are asked. We test the influence of demographic characteristics and household wealth. We also include measures related to the circumstances of the interview, namely whether the same individual was interviewed in both periods, whether one of the respondents was female, and whether more than one person participated in the interview. If discrepancies between actual and recalled assets are primarily driven by different respondents, they may not be true recall errors. All demographic and interview characteristics are from the 2006 survey as that is when the recall exercise took place. Table 4 provides definitions and descriptive statistics for these explanatory variables.
Variable Definitions (N = 199).
aAge of male household head, substituted with female household head when there is no male head of household.
We estimate a pooled multinomial logit model, with three possible outcomes: Error of commission, error of omission, or accurate recall. The standard errors are clustered by respondent, and the model includes fixed effects for assets (not reported in the table, but broadly consistent with the results reported earlier on asset types and error rates). In Table 5, the first column shows the determinants of errors of commission and the second column determinants of errors of omission.
Multinomial Logit Model of Determinants of Recall Error (N = 199).
Note: The boldface values are statistically significant coefficients. *Significant at 10%. **Significant at 5%. ***Significant at 1%.
The most notable finding in relation to the household characteristics is that people in wealthier households are less likely to forget assets that they owned in 1997 but are also more likely to recall assets that they did not actually have. As in much of the literature, other household characteristics do not have a significant impact on recall accuracy.
Turning to the characteristics of the interview, we find that interviewing a single respondent on their own increases the likelihood of errors of commission. As expected, interviewing the same individual in both time periods reduces the rates of both errors of omission and commission.
Overall, we find that there are systematic biases generated by both the socioeconomic status of the household and the circumstances in which the recall data are elicited. Of particular concern is that households that are wealthier in the later period tend to overestimate their previous assets, which would downwardly bias any estimates of changes in assets over time. The finding that interviewing the same person in both periods reduces both types of error may indicate that some of the differences in responses are due to different perceptions of asset ownership, perhaps related to changes in household composition, rather than recall errors. Thus, the differences between panel data and recall data serve as reminders that both are based on particular constructions of the household over time.
Conclusions
This article considers multiple dimensions and ways to judge the reliability of retrospective data on asset ownership of rural households in a developing region. Such recall data are important alternatives to costly and time-consuming panel surveys. We investigated the circumstances in which retrospective data will tend to provide relatively better or worse approximations of actual patterns of asset ownership as reported in a panel survey.
Our first conclusion is that even without prompts or contextual details to aid the recall process, the majority of respondents did remember whether their households owned particular types of assets nine years earlier. However, our second conclusion is that although retrospective data clearly provide some information on the assets that a household owned at a specific time in the past, they do not provide a highly accurate measure (see Bernard et al. 1984). The literature suggests that accuracy of recall could be improved by asking about assets in a memorable time period (e.g., during an election campaign). We did not provide any such cues to our respondents, so our findings on recall accuracy may represent a lower bound. Our empirical results also suggest that recall can be improved by interviewing more than one person. Respondents were less likely to recall assets that their household had not actually owned in this context, although there was no impact on errors of omission.
Across different assets, there is wide variation in recall accuracy. Assets that are valuable and not commonly owned tend to be recalled more accurately than assets that are inexpensive and widespread. This suggests that rather than asking households about many minor assets, it may be more informative to gather detailed information (e.g., model or age) or to double-check (e.g., by checking consistency across repeated questions) on a few significant assets.
Data on assets are often used to create an index of wealth or socioeconomic status. Again, there is a weak relationship between wealth indices based on retrospective versus panel data. The relationship is stronger for indices based only on high value (and less common) assets, which tend to be recalled more easily than lower value assets.
Among the determinants of recall error, a key variable is the overall wealth of the household. We found that wealthier households made more errors of commission when recalling the specific assets they owned in the earlier period, and poorer households were more likely to make errors of omission. The finding that current wealth affects recall of asset ownership is of particular concern because it is likely to bias any subsequent analysis of wealth dynamics and distribution.
In sum, although it is possible to use retrospective asset data where no baseline data have been collected, the results are likely to be quite different, due both to imprecision in the measures of previous assets and to their relationship with current household wealth. One avenue for addressing these drawbacks is to focus on the most significant assets, perhaps gathering additional information on that smaller set of assets. Another avenue is to interview multiple household members. Previous research suggests that the type of recall task generally affects recall accuracy. We expect asset ownership to be more straightforward to recall than information on prices, land areas, or output quantities. If this is the case, the cautions we express in relation to the use of retrospective data on assets are likely to apply at least as much to other types of retrospective data collected from rural households in developing countries.
Footnotes
Acknowledgment
We thank (without implicating) Jill Caviglia-Harris and Dan Harris for helpful suggestions. Cody Burnett assisted with data processing. We are grateful for funding from the USDA Forest Service International Programs, the Amazon Institute for People and the Environment (IMAZON), and the National Science Foundation (grant #0752904).
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: funding from the USDA Forest Service International Programs, the Amazon Institute for People and the Environment (IMAZON), and the National Science Foundation (grant #0752904).