Explaining System Missing

Abstract

Missing values are nearly inescapable within social science research. The problem of missing data is especially troubling in longitudinal and intensive longitudinal studies when participants miss an entire collection cycle compared to item nonresponse. The present study examines some of the contextual influences of missed study participation using unobtrusive measures of experience in conjunction with the experience sampling method (ESM). A sample of 66 participants generated 2,940 observations across a 7-day study, yielding a missing response rate of 34%. Multilevel binary logistic regression was used to estimate the probability of missing the study participation signal based upon study time and physical activity states. Results indicate that the probability of missing collection signals increases throughout the duration of the study and with high levels of physical activity. The strongest predictor of missed participation occurred when participants’ activity monitors were set to “asleep” mode. Implications of these findings and recommendations for future ESM studies are discussed.

Keywords

advanced quantitative methods longitudinal methodology applied social psychology research methods assessment

Missing Data and Experiential Sampling

Experience sampling method (ESM) is becoming a popular way to study emotion (Goetz, Bieg, & Hall, 2016), cognition (Franklin, Smallwood, Zedelius, Broadway, & Schooler, 2016), personality (Wrzus, Wagner, & Riediger, 2016), and behavior (e.g., Lopez, Milyavskaya, Hofmann, & Heatherton, 2016) in the context of everyday life (Beal, 2015). The implementation and utilization of new technology has been at the heart of ESM research since its earliest applications (Csikszentmihalyi & Larson 1987; Csikszentmihalyi, Larson, & Prescott, 1977). Recent technological advances have increased the viability of ESM as a research tool allowing a diverse range of research questions (e.g., see Bolger & Laurenceau, 2013). Despite the growing use of ESM, the identification and treatment of missing data may limit the generalizability of ESM results (Silvia, Kwapil, Eddington, & Brown, 2013).

Missing data are an inevitable part of research in the social sciences (Roth, 1994). This aspect of research is particularly concerning for longitudinal and intensive longitudinal studies that entail a sampling schedule (Graham & Donaldson, 1993; Hedeker & Gibbons, 1997; Hektner, Schmidt, & Csikszentmihalyi, 2007; Little, 1995). Participant attrition in longitudinal studies can affect both the internal and external validity of the research study by increasing measurement errors. The error component of an observed score can be theoretically broken into random and systematic error (Crano, Brewer, & Lac, 2015). In longitudinal studies, purely random participant attrition can increase random error, potentially leading to reduced statistical effects (Graham & Donaldson, 1993). In contrast, systematic errors inflate or deflate measurement scores in a systematic manner and may lead to biased results in a study. In a longitudinal intervention study, for example, those for whom the intervention was unsuccessful could drop out of the study leading to inflated effects for the intervention. The challenge for researchers is to identify and mitigate instances of systematic bias within longitudinal research and to distinguish those systematic errors from the random deviations associated with human variability. The unique quality of longitudinal or intensive longitudinal designs is discontinuous participation since no further information is collected from the individual to explain the missed measurement period¹ (Acock, 2005). It becomes a researcher’s responsibility to understand the mechanisms leading to missing data in order to adjust for missing responses.

Missing data are typically classified into one of three categories: missing completely at random (MCAR), missing at random (MAR), or not missing at random (NMAR; Graham & Donaldson, 1993). Data MCAR occurs when the probability of missingness is unrelated to any observed or unobserved variables, has no systematic missingness, and can be ignored in most research studies. MCAR is a rarity in most research designs because missingness often has some systematic cause. Missing data that only occurs on observed variables are called MAR (Little, 1995). For example, if the probability of answering a question regarding weight is dependent on sex, then the data can be classified as MAR. MAR is easily handled in research by ignoring the missing outcome and controlling for the variable that causes the missingness (Gelman & Hill, 2007). The last form of missingness occurs when data are NMAR. Data are not considered MAR when there is a systematic cause for missing data that is not measured elsewhere in the study. For example, missingness on a question regarding preference for a political candidate could be dependent on social desirability (or lack thereof). If one candidate was perceived as reprehensible, it could cause participants to underreport support for that candidate. So long as social desirability is unmeasured, then the missing responses for candidate favorability would be classified as NMAR. Data NMAR can cause biased estimates and interpretation based on the unknown causes of the missingness. Analyses with data NMAR need to explicitly model the missingness, which requires many assumptions about the causes of missingness (Buhi, Goodson, & Neilands, 2008). For this reason, it is better for researchers to include more variables in a study to catch possible sources of missingness.

ESM and the Problems of Missing Data

ESM was developed by Mihaly Csikszentmihalyi to assess the psychological experience of daily life (Hektner et al., 2007). The concept underlying ESM was to take snapshots of individuals’ daily experience in real time. ESM avoids some potential problems of systematic bias that may be present in retrospective reports of experience and has the benefit of measuring within-person change that is absent in traditional survey research.

Originally, ESM required individuals to carry an electronic pager that was signaled via radio broadcast stations at prespecified times throughout the day. Study participants were asked to complete a short survey whenever the pager was signaled. The surveys were stored in a booklet that participants were asked to carry with them throughout the study. ESM studies typically last for only 7 days due to the high participant demand.

A problem with the traditional ESM method was that it was possible for participants to fake their data by completing the booklet of survey forms immediately before returning the booklet and pager to the researchers. Modern technology has limited the potential for participants to complete all of the survey forms at one time. Increasingly, ESM studies are conducted using electronic data collection devices such as PDAs or cell phones (Hektner et al., 2007). The benefits for researchers using these technologies to conduct ESM studies are the ability to measure when the signal was sent, the exact time at which the participant completes the ESM survey, and the ability to block participants from completing the experiential survey after a certain period of time (e.g., participants must complete the survey within 15 min of being signaled).

Missingness in intensive longtitudinal designs is a reality of research (Bolger & Laurenceau, 2013). Previous researchers have documented rates of missingness as low as 10% in clerical and managerial workers and up to 30% for high school students and blue-collar workers in the United States using pen-and-paper booklets (Csikszentmihalyi & Larson, 1987). Asakawa (2004) found that 27% of signals were missed while conducting a study using a sample of Japanese college students. Missingness typically occurs when an individual does not complete an entire survey when they are signaled. This form of missingness is assumed to occur at random (MAR), but there is typically no method to assess this assumption since all predictors and outcome variables are also missing. As previously mentioned, data MAR is not a problem for statistical analyses as long as the variables that predict missingness are included in the model. Data NMAR poses a significant threat to ESM researchers because it may bias statistical models as well as miss important phenomenological aspects of experience.

The challenge for researchers is to obtain variables that predict missingness when no information is collected at the time when the survey was missed. Available information (typically demographic information) collected in previous waves of a study can be used to help identify and model missingness (Acock, 2005). For example, Dunton and colleagues (2014) state that they assessed whether missing data were associated with any demographic characteristics of their sample prior to the main analysis of their study. Multilevel binary logistic regression can be used to model missing versus nonmissing responses within individuals to identify sources of missingness (Gelman & Hill, 2007). Typically, only between-subject analyses could be completed on missing signals for intensive longitudinal studies because self-report data are not obtained.

Self-report data are not the only information that can be collected from participants throughout a study. Previous ESM researchers have used unobtrusive data collections to monitor human behavior without impeding participant’s activities (e.g., Mehl, Pennebaker, Crow, Dabbs, & Price, 2001). Dunton and colleagues (2014) utilized both unobtrusive measures of physical activity and self-report assessments to study the effect of physical activity on children’s mood. The Electronically Activated Recorder was developed to unobtrusively record 30 s of ambient sound relating to a participant’s daily experience (Mehl, 2007). Other researchers have studied the effects of physical activity in a wildness protection area using GPS and accelerometer data collected through cell phone tracking devices in conjunction with self-report questionnaires (e.g., Doherty, Lemieux, & Canally, 2014). Unobtrusive measures of experience can be synchronized with self-reported information to assess human behavior in naturalistic settings.

This study utilizes wrist-worn accelerometers to assess why participants may not respond to an ESM signal. Studies examining the factors that contribute to missed ESM signals have identified that participant response decreases throughout the length of the study (Courvoisier, Eid, & Lischetzke, 2012; Silvia et al., 2013). It is hypothesized that participants will be more likely to miss an ESM signal later in the study. Silvia, Kwapil, Eddington, and Brown (2013) identified that participants who were highly enthusiastic about what they were doing were less likely to respond to the next ESM signal. Based on this finding, we hypothesized that individuals who are highly physically active at the moment they are signaled would be less likely to respond. Finally, those who had set their accelerometer to sleep mode would be less likely to respond to the study signal.

Method

Participants and Design

The present study utilized the mobile ESM application Personal Analytics Companion (PACO) that runs exclusively on Android and iOS smartphones. Participants received a Fitbit Flex, a water-resistant triaxial accelerometer worn around the wrist, to passively collect information about their physical activity and sleep throughout the study period. Word of mouth and social networking were used to recruit participants. Participation was entirely voluntary and a US$20 incentive was provided to participants at the completion of the study.

A sample of 75 participants was recruited for the study. Two participants did not enroll in the ESM portion of the study; seven participants were excluded for failures related to the Fitbit (e.g., lost device, device was not worn, and data were lost or corrupted). A final sample of 66 participants provided 2,940 observations over a 7-day period. Nine participants did not complete the end-of-week survey that contained demographic information. The sample consisted of slightly more females (63%) than males and was primarily White (58%), well-educated (80% with bachelor’s degrees or above), working professionals (58%) or students (30%), who were on average 31 years old (SD = 13 years) based on the available demographic information (n = 57). Participant characteristics are displayed in Table 1.

Table 1.

Participant Demographic Information.

Female	63%
Age	31 (±13)
Race
White	58%
Asian	18%
Black	7%
Hispanic	9%
Native	2%
Other	7%
Occupation
Student	30%
Paid work	58%
Unemployed	7%
Volunteer	5%
Education
High school	7%
Some college	12%
Bachelor’s	47%
Master’s	28%
Professional degree	5%

Note. n = 57. Demographic variables do not reflect the total sample used for statistical analysis (N = 66). Nine participants did not complete the end-of-week survey that contained demographic markers. Demographic information was not a useful predictor, so the larger sample was used for statistical analysis.

Procedure

Participants were randomly signaled on their smart phone via the mobile application PACO 6–7 times per day for a full 7-day period between the hours of 8 a.m. and 10 p.m. When a participant was signaled during the study, they were required to open the mobile application and complete the experience sampling form within 15 min of being signaled. Participants were allowed to adjust the time frame to best suit their waking schedule but were asked to maintain a 14-hr participation window. Each participant received a Fitbit Flex prior to their 7-day participation period and received instructions for operation (i.e., participants were trained on how to sync the Fitbit to upload data, charge the battery, and switch between sleep mode and physical activity mode). Training sessions were conducted either in person or over the telephone by a member from the research team. Participants were instructed to wear the Fitbit for the duration of the 7-day period, only to take it off if it needed to be recharged or to avoid submerging it in water for extended periods of time. Following the completion of the 7 days, individuals were given a link to complete an online survey that would provide demographic information.

Measures

Missingness

Missingness is represented by a dummy coded variable where 1 represents a missed ESM response and 0 represents a nonmissing ESM response. Throughout the study, 34% of responses were classified as missing (n = 994). This rate of missingness is similar to the rates of missingness obtained in recent studies using electronic data collection (e.g., 30%; Silvia et al., 2013).

Study time

Study time was measured continuously throughout the study. The measure for study time began with the first ESM signal and concluded following the completion of a poststudy survey. For interpretability, study time was scaled to 24 hr such that an increase of 1 scale point corresponds with an increase in 24 hr.

Physical activity states

Two measures were collected using the Fitbit activity tracker. First, participants set the Fitbit to “asleep” mode when they were going to sleep and “awake” mode when they awoke. Sleep was dummy coded with 1 representing the Fitbit in “sleep” mode and 0 awake. When the Fitbit was in the awake mode, the Fitbit recorded the number of steps taken per minute. Cumulative physical activity in the 5 min leading up to the ESM signal was used to assess participant physical activity at the time of the signal. Physical activity was aggregated such that 1 represents an average of 100 steps per minute for the 5 min leading up to the signal. The American College of Sports Medicine finds that 100 steps per minute measured with a pedometer was a rough approximation for moderate to vigorous physical activity (Garber et al., 2011; Marshall et al., 2009). Using this approximation, an increase from 0 to 1 in the present study is associated with the difference between being sedentary and moderate to vigorous activity.

Statistical Analyses

Multilevel binary logistic regression was used to predict missed ESM signals using demographic information, length of time in the study, whether participants’ accelerometers were awake or asleep, and their physical activity during the 5 min leading up to the missed signal. A multilevel analysis was chosen based on the nested structure of the data (Bolger & Laurenceau, 2013). Several nested models of increasing complexity were tested. In all models, individual intercepts were allowed to vary based on individual differences in probability of missing an ESM signal. After assessing the measured variables as fixed effects, individual differences in the effect of physical activity were also assessed. Previous research has shown that more active individuals are less responsive to effects of exercise (Reed & Ones, 2006); similarly, we would expect individual differences in the amount that exercise affects the probability of missing an ESM signal. The statistical program R was used to test the multilevel structures using the “lme4” package (Bates, Maechler, & Bolker, 2016).

Results

Preliminary Analysis

Prior to the main analysis, participants excluded from the study were assessed for differences that could explain their failure to comply with the study protocol. First, participants who had failures related to the Fitbit were compared to those who did not have any Fitbit problems. Those who did not have Fitbit data were more likely to be male, χ²(1) = 5.57, p = .018, but otherwise not statistically different in age, t = .054, p = .957, race, χ²(5) = 6.66, p = .247, occupation, χ²(3) = 1.87, p = .600, or level of education, χ²(5) = 4.38, p = .357, nor were they more or less likely to miss the ESM signal, b = 0.30, p = .422. Those who missed the end-of-week demographics survey were marginally more likely to miss the ESM signal, b = 0.64, p = .060, but were no more physically active in the time leading up to the ESM signal, b < 0.01, p = .945, after controlling for person-level variance. The within-subject means and standard deviations for the variables used in the main analysis are shown in Table 2 as well as the within-subject bivariate correlations.

Table 2.

Within-Subject Descriptive Statistics and Correlations.

	Missed Signal	Days in Study	Asleep	100 Steps per Minute
Missed signal	0.35 (0.43)
Days in study	0.07***	3.53 (2.02)
Asleep	0.13***	0.02	0.03 (0.09)
100 Steps per minute	0.04*	>0.01	−0.06**	0.07 (0.14)

Note. Within-subject means (standard deviations) for each variable are shown in the diagonal.

*p < .05. **p < .01. ***p < .001.

Main Analysis

Several models were specified to test the causes for missing an ESM signal (Table 3). The baseline model consisted of random intercepts only.² The null model indicates that there is a 27% probability that a participant will miss an ESM signal, with a 95% confidence interval (CI) ranging from 25% to 29%, b = −0.76, SE = 0.12, Akaike information criterion (AIC) = 3,464.9, −2 log likelihood (−2LL) = −1,730.4. The within-person variability suggests that the individual probability of missing an ESM signal ranges from 7% to 75%, but the within-person variability makes this estimate unstable, unstructured covariance (UN)(1,1) = 0.90, SD = 0.95. Accounting for this individual variance, the null model is able to classify 71.26% of cases correctly as missing or not missing. The second model specified included the demographic variables with fixed effects for age and gender and random intercepts for race, education level, and occupation (see Table 1 for demographic characteristics). The model fit was not an improvement over the intercepts-only model using the participants who provided demographic information, $χ_{diff}^{2}$ (5) = .09, p = .999. Demographic variables do not improve the model prediction and were dropped from further analyses, and the larger sample of N = 66 was used for model comparisons.

Table 3.

Multilevel Logistic Analysis for Predicted Missed ESM Signal.

Fixed Effects	Model Ø		Model 1		Model 2		Model 3
Fixed Effects	Estimate	SE	Estimate	SE	Estimate	SE	Estimate	SE
Intercept	−0.76	0.12	−1.31	0.13	−1.15	0.15	−1.16	0.15
Days in study			0.08	0.02	0.08	0.02	0.08	0.02
Asleep					1.92	0.31	1.90	0.32
100 steps per minute					0.69	0.27	0.70	0.44
Random Effects	Estimate	SD	Estimate	SD	Estimate	SD	Estimate	SD
Subject ID	0.90	0.95	0.91	0.95	0.91	0.96	0.96	0.98
100 steps per minute							4.46	2.11
AIC	3,464.9		3,451.2		3,408.4		3,401.7
−2 log likelihood	−1,730.4		−1,722.6		−1,699.2		−1,693.8
Classification accuracy	71.26%		71.67%		72.21%		73.33%

Note. Sixty-six participants provided 2,940 observations with 34% missing data (n = 994). Classification accuracy is the total model accuracy for identifying an ESM signal as missed or not missed. ESM = experience sampling method; SE = Standard Error; AIC = Akaike information criterion.

Number of days in the study was added to the model as a continuous fixed effect while adjusting for individual-level variance. The overall model was a significant improvement over the intercepts-only model, AIC = 3,451.2, −2LL = −1,722.6, $χ_{diff}^{2}$ (1) = 15.74, p < .001. The amount of time a participant was in the study was a statistically significant predictor of missingness while ignoring all other variables, b = 0.08, p < .001. These results suggest that for each day a participant is in the study, the odds of missing an ESM signal are 1.09 times higher than the odds the preceding day, 95% CI [1.04, 1.13]. Including the amount of time participants were involved in the study increased the probability of correctly identifying whether an ESM signal was missed or not to 71.67%.

The measures of physical activity states were added to the model as fixed effects and produced a statistically significant improvement in the model with only the amount of time spent in the study, AIC = 3,408.4, −2LL = −1,699.2, $χ_{diff}^{2}$ (2) = 46.78, p < .001. The largest effect was whether the Fitbit activity tracker was set to sleep mode or not. When the Fitbit was set to sleep, the odds that a participant would miss the ESM signal were 6.80 times higher, 95% CI [3.67, 12.60], b = 1.90, p < .001. After accounting for time in the study and sleep status, physical activity was also a significant predictor of missingness, b = 0.69, p = .009. Participants who were walking at a moderate to vigorous pace during the 5 min prior to the signal (100 steps per minute average) were 2.00 times more likely to miss the signal than those who were sedentary, 95% CI [1.19, 3.36]. Including all of the measured variables as fixed effects at the time of the missed signal increased the probability of correctly identifying whether an ESM signal was missed or not to 72.21%.

In the final model, the slope for the effect of physical activity was allowed to vary randomly for each individual to test for between-subject differences in the susceptibility for missingness resulting from physical activity. Allowing the slopes of physical activity to vary within each person provided a statistically significant improvement in the overall model, AIC = 3,401.7, −2LL = −1,693.8, $χ_{diff}^{2}$ (2) = 10.71, p = .005. Additionally, the correlation between participants’ baseline probability to miss a signal was positively correlated with the slope of physical activity, r = −.12, suggesting that individuals who were more likely to miss a signal at the beginning of the study were less likely to miss a signal as a result of being physically active. Accounting for individual susceptibility to physical activity in addition to the other variables in the model was able to increase the model classification to correctly identify 73.33% of cases.

Discussion

The present study sought to test whether missed sampling signals are MAR. Missingness is an assumed aspect of intensive longitudinal designs. The benefit of intensive longitudinal designs is in the ability to sample a cross-section of participants’ lived experiences. Continuous and unobtrusive measurement of experience allows researchers to access the moments when self-report data are inaccessible. Physical activity data that were continuously collected throughout the study were used to understand the moments when participants missed the ESM signal. Results from this analysis suggest that certain experiences may require specialized sampling protocols if they are the primary focus of the study.

This study utilized continuous activity monitoring in conjunction with self-report surveys sampled randomly throughout each day of the study. Participants who were involved in moderate to vigorous physical activity were more likely to miss the ESM signal. By measuring the effects of physical activity in conjunction with a study, physical activity can be added as a covariate in prediction models for other variables in future ESM studies to help account for potentially biased results from missed sampling experiences. An additional analysis suggested that the relationship between physical activity and missing the ESM signal is weaker for people who were more likely to miss the ESM signal in general, that is, those who were already more likely to miss the ESM signal were less sensitive to the effects of being physically active. This finding could reflect a ceiling in participant nonresponse, that is, those who are going to miss the most ESM signals missed them regardless of their activity levels.

Based on these findings, researchers seeking to explore the experiences surrounding moderate to vigorous physical activity should utilize study protocols that would minimize the probability of losing sampled experiences. One method that could be employed for ESM research is event-triggered protocols in which participants are asked to self-report their experience immediately after completing a specific task or activity (Hektner et al., 2007). With increased usage of cell phones in ESM studies, researchers may have the ability to access the accelerometer information collected by the phone in conjunction with GPS data to create the signal following a specific event (e.g., when a participant returns to their starting location after a run). Audio and GPS tracking software have been successfully incorporated into behavioral research (e.g., Doherty et al., 2014; Mehl et al., 2001). There are many potential research applications for unobtrusive measures of behavior as a supplement to traditional self-reported measures of experience (Conner & Mehl, 2015). Researchers have the opportunity to incorporate these technologies into study designs that allow for new methods to observe and study human behavior.

Study Design and Missing Data

In addition to the information for participant nonresponse collected through continuous monitoring of physical activity, other contextual effects associated with the study and study design were associated with missing the ESM signal. The accelerometers that were used in the study (Fitbit) had an asleep/awake function that could be used for individuals to monitor their sleep patterns. Using this setting, when participants had set the accelerometer to asleep mode, they were least likely to respond to a signal for study participation. This finding is consistent with previous studies that have found that participants who are feeling more awake tend to be more likely to respond to the study signal (Courvoisier et al., 2012). Consequently, when participants are awoken from sleep by the ESM signals, they are generally unhappy when completing the corresponding self-report survey (Hektner et al., 2007).

Participant fatigue is always a potential problem in ESM studies. The present study found that the probability of missing the ESM signal was higher the longer the study progressed. As the study continues, participants may become fatigued with the study procedures and may decide to ignore the ESM signal. This finding is consistent with previous research on participant nonresponse (Courvoisier et al., 2012; Silvia et al., 2013). A less pessimistic perspective would suggest that, throughout the study, participants become less sensitive to the ESM signal and fail to recognize when they are requested to participate in the study. This process has been observed using several different ESM paradigms (Hektner et al., 2007). Additionally, the present study allowed participants a 15-min window in which to respond to the ESM signal. Increasing the allowable time to complete the survey would allow participants to complete whatever activities that were keeping them from the study and to report on the effects of those activities. This is a design feature that ESM researchers should consider when designing their own studies.

The traditional method to assess systematic missingness in ESM studies requires the analyst to create dummy “missing” variable codes and to test for differences in participation based on demographic variables. These procedures were not found to be an effective control for missingness. In the present study, using a sample of students and professional workers, participant demographics (age, gender, race, occupation, and education level) provided no useful information when predicting missingness after accounting for individual differences. These findings are consistent with previous ESM studies attempting to account for missingness (e.g., Dunton et al., 2014). These findings suggest that typical approaches for checking systematic missingness do not capture the mechanisms of missingness relevant to studies of daily experience.

There are several ways to mitigate missed sampling experiences based on the present findings. For starters, selecting a sampling schedule that optimizes participants’ waking hours would be preferable to a single fixed schedule. Participants in the present study were most likely to miss a signal when their Fitbit activity monitors were set to the sleep mode. While the present study offered participants the opportunity to adjust their sample schedule, it may have been better for researchers to have all participants manually set their study availability instead of having a default schedule already in place. Additionally, researchers utilizing an ESM study should balance the sampling frequency with the duration of the study and with the time required to respond to each ESM signal. Previous texts have provided similar recommendations (e.g., Hektner et al., 2007); however, the present research would add to this by drawing attention to the participant fatigue that can be measured using a missingness analysis throughout a study. Best practices for statistically modeling ESM data include time as a covariate (Bolger & Laurenceau, 2013). The present study reaffirms this practice and suggests that conclusions drawn from the end of an ESM study may be less stable than those drawn from the beginning and middle.

While the present study suggests minor refinements to ESM for future research, limitations regarding the study should still be recognized. The study participants were primarily White and highly educated. Individuals coming from those populations are likely to have different experiences compared to others in the general population. Second, several procedural decisions at the outset of the study created the ability for participants to miss ESM signals. In particular, a 15-min window was provided for participants to respond to the signal. It is possible that a longer window would have resulted in lower rates of missed ESM signals. Additionally, having the signals beginning shortly after 8 a.m. as the default and stopping after 10 p.m. may have created an artificial window for waking hours for many participants and resulted in missed or unsampled momentary experiences. Finally, using mobile phones for signaling study participation has potential malfunctions due to a lack of cellular reception or limitations in mobile battery life. Both of these issues may have resulted in additional missed ESM signals that were unrelated to the study procedures and outside of the researchers’ control. These limitations should be considered in conjunction with the findings presented, not to negate the general findings but to qualify the generalizability of the current research.

Missing data is a reality of the social sciences. Understanding the mechanisms of missingness allows for researchers to appropriately plan for and statistically model cases of missing responses. Missing responses for intensive longitudinal designs are assumed to be MAR (Bolger & Laurenceau, 2013); however, the current study suggests that there are systematic causes for missingness that typically go unmeasured. Utilizing unobtrusive and continuous measurement devices allows researchers to capture information about a participant’s experience when they are otherwise occupied. These methods can be combined with traditional self-report measures to create a more refined understanding of human behavior. In the present study, wrist-worn accelerometers were used to understand the instances of missed ESM study notifications such that when the accelerometers were set to the asleep mode or when participants were engaged in moderate to vigorous physical activity, they were more likely to miss an ESM signal. Mechanisms of missingness that can be measured for all ESM studies include study duration to control for some of the effects of missingness (Acock, 2005). It would be unreasonable for many researchers to require participants to wear devices to record physical activity, but accelerometer data could be obtained through other sources such as participant’s cell phones. These data sources can supplement traditional self-report information to account for systematic sources of missingness and will help researchers qualify and interpret results generated using the experience sampling methodology.

Footnotes

Acknowledgments

The authors wish to gratefully acknowledge the help of Brian Whiteley, Brian Werter, and their colleagues at NORC at the University of Chicago for their help and cooperation, without which this research could not have been accomplished.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

References

Acock

A. C.

(2005). Working with missing values. Journal of Marriage and Family, 67, 1012–1028.

Asakawa

(2004). Flow experience and autotelic personality in Japanese college students: How do they experience challenges in daily life? Journal of Happiness Studies, 5, 123–154.

Bates

Maechler

Bolker

(2016, April 16). Linear mixed-effects models using ‘Eigen’ and S4. Retrieved from https://cran.r-project.org/web/packages/lme4/lme4.pdf

Beal

D. J.

(2015). ESM 2.0: State of the art and future potential of experience sampling methods in organizational research. Annual Review of Organization Psychology and Organizational Behavior, 2, 383–407.

Bolger

Laurenceau

J.-P.

(2013). Intensive longitudinal methods: An introduction to diary and experience sampling research. New York, NY: Guilford Press.

Buhi

E. R.

Goodson

Neilands

T. B.

(2008). Out of sight, not out of mind: Strategies for handling missing data. American Journal of Health Behavior, 32, 83–92.

Conner

T. S.

Mehl

M. R.

(2015). Ambulatory assessment—Methods for studying everyday life. In Scott

Kosslyn

Pinkerton

(Eds.), Emerging trends in the social and behavioral sciences: An interdisciplinary, searchable, and linkable resource (pp. 1–13). Hoboken, NJ: Wiley.

Courvoisier

D. S.

Eid

Lischetzke

(2012). Compliance to a cell phone-based ecological momentary assessment study: The effect of time and personality characteristics. Psychological Assessment, 24, 713–720. doi:10.1037/a0026733

Crano

W. D.

Brewer

M. B.

Lac

(2015). Principles and methods of social research (3rd ed.). New York, NY: Routledge.

10.

Csikszentmihalyi

Larson

(1987). Validity and reliability of the experience-sampling method. Journal of Nervous and Mental Diseases, 175, 526–536.

11.

Csikszentmihalyi

Larson

Prescott

(1977). The ecology of adolescent activity and experience. Journal of Youth and Adolescence, 6, 281–294.

12.

Doherty

S. T.

Lemieux

C. J.

Canally

(2014). Tracking human activity and well-being in natural environments using wearable sensors and experience sampling. Social Science and Medicine, 106, 83–92. doi:http://doi.org/10.1016/j.socscimed.2014.01.048

13.

Dunton

G. F.

Huh

Leventhal

A. M.

Riggs

Spruijt-Metz

Pentz

M. A.

(2014). Momentary assessment of affect, physical feeling states, and physical activity in children. Health Psychology, 33, 255–263.

14.

Franklin

M. S.

Smallwood

Zedelius

C. M.

Broadway

J. M.

Schooler

J. W.

(2016). Unaware yet reliant on attention: Experience sampling reveals that mind-wandering impedes implicit learning. Psychonomic Bulletin & Review, 23, 223–229.

15.

Garber

C. E.

Blissmer

Deschenes

M. R.

Franklin

B. A.

Lamonte

M. J.

Lee

I. M.

… Swain

D. P.

(2011). Quantity and quality of exercise for developing and maintaining cardiorespiratory, musculoskeletal, and neuromotor fitness in apparently healthy adults: Guidance for prescribing exercise. Medicine and Science in Sports and Exercise, 43, 1334–1359. doi:10.1249/MSS.0b013e318213fefb

16.

Gelman

Hill

(2007). Data analysis using regression and multilevel/hierarchical models. Cambridge, MA: Cambridge University Press.

17.

Goetz

Bieg

Hall

N. C.

(2016). Assessing academic emotions via the experience sampling method. In Zembylas

Schutz

P. A.

(Eds.), Methodological advances in research on emotion and education (pp. 245–258). Switzerland: Springer International Publishing.

18.

Graham

J. W.

Donaldson

S. I.

(1993). Evaluating interventions with differential attrition: The importance of nonresponse mechanisms and use of follow-up data. The Journal of Applied Psychology, 78, 119–128. doi:10.1037/0021-9010.78.1.119

19.

Hedeker

Gibbons

R. D.

(1997). Application of random-effects pattern-mixture models for missing data in longitudinal studies. Psychological Methods, 2, 64–78. doi:10.1037/1082-989X.2.1.64

20.

Hektner

J. M.

Schmidt

J. A.

Csikszentmihalyi

(2007). Experience sampling method: Measure the quality of life. Thousand Oaks, CA: Sage.

21.

Little

R. J. A.

(1995). Modeling the drop-out mechanism in repeated-measures studies. Journal of the American Statistical Association. doi:10.1080/01621459.1995.10476615

22.

Lopez

R. B.

Milyavskaya

Hofmann

Heatherton

T. F.

(2016). Motivational and neural correlates of self-control of eating: A combined neuroimaging and experience sampling study in dieting female college students. Appetite, 103, 192–199.

23.

Marshall

S. J.

Levy

S. S.

Tudor-Locke

C. E.

Kolkhorst

F. W.

Wooten

K. M.

… Ainsworth

B. E.

(2009). Translating physical activity recommendations into a pedometer-Based Step Goal. American Journal of Preventive Medicine, 36, 410–415. doi:10.1016/j.amepre.2009.01.021

24.

Mehl

M. R.

(2007). Eavesdropping on health: A naturalistic observation approach for social health research. Social and Personality Psychology Compass, 1, 359–380.

25.

Mehl

M. R.

Pennebaker

J. W.

Crow

Dabbs

Price

(2001). The Electronically Activated Recorder (EAR): A device for sampling naturalistic daily activities and conversations. Behavior Research Methods, Instruments, and Computers, 33, 517–523.

26.

Reed

Ones

D. S.

(2006). The effect of acute aerobic exercise on positive activated affect: A meta-analysis. Psychology of Sport and Exercise, 7, 477–514. doi:http://doi.org/10.1016/j.psychsport.2005.11.003

27.

Roth

P. L.

(1994). Missing data: A conceptual review for applied psychologists. Personnel Psychology, 47, 537–560. doi:10.1111/j.1744-6570.1994.tb01736.x

28.

Silvia

P. J.

Kwapil

T. R.

Eddington

K. M.

Brown

L. H.

(2013). Missed beeps and missing data: Dispositional and situational predictors of nonresponse in experience sampling research. Social Science Computer Review, 31, 471–481. doi:10.1177/0894439313479902

29.

Wrzus

Wagner

G. G.

Riediger

(2016). Personality-situation transactions from adolescence to old age. Journal of Personality and Social Psychology, 110, 782–799. doi:10.1037/pspp0000054