Imputing Data for the Fragile Families Challenge: Identifying Similar Survey Questions with Semiautomated Methods

Step 0: Initial Data Preparation

Two major steps were taken to prepare the data for the subsequent imputation steps. First, all data that could be cast as numeric float values were converted from text to those numeric values. Similarly, categorical (multiple-choice) variables were cast as integer representations when possible. Features that could not be placed into either data type were removed. Second, we removed columns that contained zero variation in the data (were all one value) or were all null valued. As a result of these two operations, 2,519 features were removed from the original training data.

Step 1: Cross-Path Imputation

We refer to a path in the survey as a result of a particular skip pattern. It is a sequence of questions that depends on answers to prior questions or criteria identified during the interview. For example, mothers being surveyed will answer a different set of questions on the basis of their current relationships with their children’s fathers. An inspection of the Fragile Families surveys showed a number of either same or similar questions asked in separate and exclusive paths. This means that a null value appearing for one question in the survey is actually answered in a differently labeled question in the same survey that has either the same or similar question text. For example, in the mother baseline survey, questions B5, B11, and B22, are all the same: “I’m going to read you some things that couples often do together. Tell me which ones you and [baby’s father] did during the last month you were together.” In any given survey, only one of these questions would be answered depending on the path the interview took following these criteria:

Cross-path imputation is a method to replace question responses that were skipped or not answered by using response values from the same or similar questions elsewhere in the survey but within the same section of questions. This automated algorithm was applied to all surveys, regardless of respondent. Sections of a particular survey are organized to cover major topics such as “income” and “current partner.” Similarity is measured using the Levenshtein edit distance ¹ (Bird, Klein, and Looper 2009) between variable labels of survey questions in the same section. These variable labels were encoded by the FFC. Labels with an edit distance of less than 10 were identified and placed into a set. The mean value, in the case of multiple related responses, was then used to replace the original null value. Although a sample of questions were validated by hand and found to be correct, we did observe that there were false positives (see Appendix A3).

Steps 2 and 4: Cross-Caregiver Imputation

In each survey wave (birth, year 1, year 3, etc.), there were a number of same or similar questions when comparing across caregivers. For example, there were questions appearing in the mother survey for a given wave that also appeared in the father survey or the primary caregiver survey. If only one parent was interviewed, the corresponding questions on the other surveys would be null. Two assistants manually identified these questions in each wave. A third person checked the results for validation. Once the related questions were determined, if a question was missing a value, the corresponding survey value was substituted.

Cross-caregiver imputation was performed twice, before and after cross-year imputation. The strategy was to impute within a particular wave in the first iteration of cross-caregiver imputation. The remaining data were imputed after cross-year imputation to capture cases in which only one or no parents would respond after a given wave in the survey.

Step 3: Cross-Year Imputation

Both the mother and father surveys showed monotonic increases in missing values with each successive wave. Although the mother, father, and primary caregiver surveys varied the questions in each wave, there were many similarities. One option was to carry the last completed survey data forward; however, this potentially overrepresents the particular value of the last recorded year, resulting in skewing of the imputed data set. Mode imputation was another option, but we wanted to capture the variation in recorded values across survey waves in the imputation that would otherwise be lost by choosing the most common value. For example, if an eviction was recorded in an earlier survey, we wanted to make sure that it had some influence on the value of the imputed surveys. Although no option is perfect, we chose to take the mean value for a given feature across all of the surveys that were completed. We term this “cross-year” imputation.

Question labels could have different text yet essentially ask for the same information, as shown in Figure 1. The Levenshtein edit distance and simple thresholding produced unreliable matches. Therefore, an algorithm was designed to suggest groups of related questions to a user in each of the survey waves. The algorithm output used the same Levenshtein edit distance from the NLTK toolbox (Bird et al. 2009), and two thresholds were used to identify candidate questions. The first threshold was a hard threshold that determined the preliminary admittance into the set of potential candidates. The user would not see any questions outside of the set. To reduce the chances of false negatives, this primary threshold was set high so that less similar question text would be admitted. However, if the user had to manually remove these entries, that would take more time, which was limited. Therefore, a second lower threshold marked suspect question text with an asterisk. The user need only remove the asterisk for questions that are to be included and put an x next to questions that should not be included. This made it more efficient to scan the sets of question labels and have the user validate. An example of this process is shown in Figure 1. Once completed, the imputation process was conducted sequentially across waves starting with the first two collection waves.

OPEN IN VIEWER

Figure 1.

Two examples of candidate groups of related questions identified by our algorithm. Questions without markers will be clustered, and their values will be averaged to fill in missing data. Questions denoted with an asterisk were labeled as not part of the set automatically, and questions with an x were manually labeled. This procedure was a simple way to find related questions without resorting to much manual effort writing code or comparing surveys.

Step 5: Mode Imputation and Finalization

Our final step was simply to fill in the remaining missing data with most frequent values appearing for a given feature: mode imputation. We did this because most of the input features were interpreted by us to be either categorical or ordinal. This is evidenced by the number of unique values shown in Appendix A4. In retrospect, we would have handled this better by giving different simple imputation assignments to the different continuous or categorical variables. Last, a final pass at removing any columns that displayed zero variation in the data was performed as outlined in step 0.

Model Selection

After completing preprocessing and feature engineering, the result was an imputed set of training data that could be applied to multiple types of models. Because of time and resource constraints, we wanted to quickly test numerous types of models. Therefore, we imposed a self-limited set of models that were easily implementable and available in well-tested packages. We chose the scikit-learn package (Pedregosa et al. 2011) written in Python. With more time, our modeling efforts would have explored relevant social science theories and created heterogeneous models to account for different data types. One model was trained for each outcome variable separately. We experimented with several types of models, both classifiers and regression models. Validating using training data, we found that the linear and logistic regression models tended to perform the best with respect to mean square error (MSE) and were chosen for the final submission. Our final choice of models for the Challenge is listed in Table 2. We found that the L1 regularized linear regression (i.e., least absolute shrinkage and selection operator; Tibshirani 1996) produced the best results using K-fold (10 folds) cross-validation for the GPA, grit, and material hardship outcomes. For the binary variables, eviction, job training, and layoff, we used logistic regression with L1 regularization of the weights.

OPEN IN VIEWER

Table 2.

Summary of Model Selections for Each of the Outcome Variables.

Outcome Variable	Model Type	Number of SelectionK-Folds	Number of Training K-Folds	Number of Features Imputed	Num. Features Engineered	Initial Selection Method
Grade point average	Linear	10	10	28	0	L1-penalty
Grit	Linear	3	10	48	5	L1-penalty
Material hardship	Linear	—	10	41	5	Manual
Eviction	Logistic	—	10	41	5	Manual
Layoff	Logistic	—	10	5	1	Manual
Job training	Logistic	—	10	5	1	Manual

Note: The linear regression model with L1 regularization was chosen for the ordinal outcomes, and the logistic regression with L1 regularization was chosen for the binary outcomes. Models showing values for the number of K-folds for selection used an initial L1 regularized model (i.e., least absolute shrinkage and selection operator) to pare down the large imputed data set. The models not showing values for the number of selection K-folds had features that were selected manually. All models had L1 regularization on the model weights when trained. The number of K-folds during training to perform cross-validation was 10 for each model. Last, the impact of the imputation strategy on the training data for each model is given by the number of features imputed (in total) and the percentage of the total input data imputed. These values are detailed by each individual imputation step also in Appendix A6.

Variable Selection

We used both manual and automated methods for variable selection, as summarized in Table 2. For some outcome variables, we manually identified potentially relevant survey questions and data sources. These manually identified features mostly consisted of the features used to form the outcome variable discussed in the “Feature Engineering” section. If the manually selected subset did not perform well, we used the entire imputed data set for variable selection using L1 regularization. GPA and grit fell into this category. For these models, we selected a subset features that had effectively nonzero weights (i.e., >.00001) in the initial L1 regularized model. A full accounting of the features chosen for each outcome variable is given in Appendix A6.

Challenge Performance Results

The prediction results are summarized in Table 3. Overall, the models in our submission performed better when predicting the material hardship and layoff outcome variables in terms of MSE. These categories both performed better than a mean calculated baseline model and were ranked fifth and ninth, respectively, in the FFC. Distributions comparing our results with the rest of the submissions in the challenge are presented in Appendix A5.

OPEN IN VIEWER

Table 3.

Results of Our Final Submissions Using Both Challenge Ranking and MSE.

Outcome Variable	Outcome Type	Prediction Error	Error Measure	Model Rank	Baseline Error	Model-baseline Improvement (%)
Grade point average	Ordinal	0.365	MSE	36 of 124	0.425	14.04
Grit	Ordinal	0.255	MSE	77 of 111	0.253	−0.96
Material Hardship	Ordinal	0.020	MSE	5 of 113	0.025	21.51
Eviction	Binary	0.055	Brier	33 of 90	0.055	1.07
Layoff	Binary	0.164	Brier	9 of 91	0.167	1.66
Job training	Binary	0.185	Brier	47 of 95	0.185	−0.01

Note: These are the same metrics determined by the Fragile Families Challenge. The baseline results presented are the mean of the training data for a particular outcome variable. Of the models in our submission that performed better than the baseline, material hardship and layoff were ranked in the top 10 of the challenge. Model ranking is determined by unique mean square error (MSE) scores, so models with the same score are given the same ranking. However, relative to the scale of outcome variables, none of our models were good predictors.

Despite FFC rankings, the results of our models and those of the challenge do not indicate particularly predictive models when applied to individual circumstances. This is because error tolerances on individual predictions are not guaranteed with aggregate error measures such as MSE. This is discussed in more detail in Appendix A1. Here, we comment only on the results of our imputation and modeling approach in the context of the challenge. The outcome variables material hardship, layoff, and GPA show improvements over the baseline, and we view this work as a potential first step toward developing imputation techniques for Fragile Families data that exploit the structure of the survey instruments.