Best Practices in Data Collection and Preparation: Recommendations for Reviewers,Editors,and Authors

Type of Research Design

The ultimate purpose of all scientific endeavors is to develop and test theory, and a critical goal is to address causal relations: Does X cause Y? To establish causal claims, the cause must precede the effect in time (Shadish, Cook, & Campbell, 2002; Stone-Romero, 2011). In other words, the research design must be such that data collection involves a temporal precedence of X relative to Y (Aguinis & Edwards, 2014). Another necessary condition for drawing conclusions about causal relations is the ability to rule out alternative explanations for the presumed causal effect (Shadish et al., 2002).

Because information regarding research design issues is critical for making claims about causal relations between variables, submitted manuscripts need to answer fundamental questions such as: Which data were collected and when? Was a control group used? Were the data collected at different levels of analysis? Was the design more suitable for theory development or theory testing? Was the design experimental or quasi-experimental? Was the design inductive, deductive, or abductive?

For example, in their study on the effects of team reflexivity on psychological well-being, Chen, Bamberger, Song, and Vashdi (2018) provided the following information regarding their research design:

We implemented a time lagged, quasi-field experiment, with half of the teams trained in and executing an end-of-shift team debriefing, and the other half assigned to a control condition and undergoing periodic postshift team-building exercises…. Prior to assigning production teams to experimental conditions (i.e., at T0), we collected data on the three team-level burnout parameters and team-level demands, control, and support. We then assigned 36 teams to the intervention condition and the remaining teams to the control condition on the basis of the shift worked (i.e., day vs. night). (pp. 443-444)

As a second example, in a study investigating an early entrant into the U.S. car-sharing industry and factors that facilitate cross-occupational collaboration, Truelove and Kellogg (2016) provided information about their qualitative research design. Specifically, they noted that they used

an inductive, ethnographic approach that is well-suited for developing new theory…. Data collection at Transco’s headquarters began in June 2012 and ended a year later…. We observed 111 meetings: 87 daily “scrum” meetings and 24 cross-departmental meetings in which engineers and marketers discussed progress and challenges on joint projects. This enabled us to track 42 projects attempted during the study’s timeframe…. To complement observational data, we conducted 126 formal, semistructured interviews. (p. 667)

Control Variables

The topic of control variables is directly related to our previous discussion about drawing causal inferences because it is about ruling out alternative explanations for a hypothesized effect (Aguinis & Vandenberg, 2014; Becker et al., 2016; Bernerth & Aguinis, 2016). When it is not practically possible to implement a research design that allows researchers to prevent or rule out the influence of other variables, the use of statistical controls offers an alternative for addressing potential confounds. Stated differently, the use of statistical controls is a way to improve on limitations of the data collection process because it involves measuring variables suspected of having a relationship with either an antecedent or outcome and including them in the analyses, thereby removing potential antecedent-outcome “contamination” (Bernerth & Aguinis, 2016; Spector & Brannick, 2011).

Important questions related to control variables that reviewers should be able to answer include: To what extent do the control variables chosen in a particular study hold a theoretically meaningful relationships with antecedents and outcomes? How and why were the particular control variables chosen? Which control variables were initially investigated but then subsequently dropped from the final analysis? Are the control variables included in the correlation table so that their reliability and correlations with all other variables are reported openly and fully?

For example, in a study examining the processes through which managers become leaders, Chiu, Balkundi, and Weinberg (2017) implemented best practices in reporting that they

included several control variables at both the leader and follower levels (L2 and L1). In order to better capture the magnitude of leader network centralities on leadership perception at T2, we controlled for followers’ leadership perceptions at the prior data collection wave (L1.T1). Also, we controlled for the sex of both leaders and followers because multiple studies have reported the impact of sex on leadership perception and effectiveness…. Further, we controlled for leaders’ personality, because multiple meta-analyses have indicated associations between leadership perception and leader personality. (p. 339)

As a second illustration of best practices, Ryu, McCann, and Reuer (2018) examined the extent to which geographic co-location between a focal firm’s partner and rivals introduces potential indirect paths of knowledge leakage to rivals. In their description, they explained that they

controlled for a number of additional factors that the previous literature has argued affect knowledge misappropriation and spillover concerns and therefore could affect alliance governance and design. The alliance literature has long argued that social networks in which alliance partners are embedded provide controls for opportunistic behaviors and thus might also affect the risk of knowledge losses, as well as the alliance design choices firms make…we controlled for an alliance dyad’s social embeddedness, using variables to capture the partners’ prior ties, indirect ties between the two firms in the dyad, and each partner’s degree centrality. (p. 953)

Sampling Procedures

Organizational research relies on samples rather than populations. Thus, it is important to provide information on the procedures used to select the sample because this is critical for making inferences about the representativeness of the observations and the generalizability of results and conclusions (Aguinis & Lawal, 2012). Moreover, in the case of qualitative research, transparency regarding sampling procedures is particularly pertinent because samples are often nonprobabilistic (Aguinis & Solarino, 2019; Teddlie & Yu, 2007).

Reviewers evaluating manuscripts should be able to answer the following questions about sampling procedures: Which particular firms and/or individuals were targeted, and which were eventually included in the study? What were the procedures used in selecting and recruiting them? In the case of archival data, what specific databases were used? What particular type of sampling procedure was used (e.g., snowballing, convenience, purposeful)?

For example, Kaul, Nary, and Singh (2018) examined the role of nonventure private equity firms in the market for divested businesses. They described their sampling strategy and steps as follows:

We test our hypotheses in a sample of divestments by publicly listed U.S. manufacturing firms from 1997 to 2010. Data on divestments are drawn from the Securities Data Company (SDC) Platinum database. We begin with all divestments in the U.S. manufacturing sector (SIC 2000-3999) listed in the SDC database for our sample years. These are then matched to parent corporations in Compustat, thus limiting our sample to divestments by publicly listed firms. We then match the remaining transactions to the Execucomp database, since we need data on executive compensation and ownership. Our final sample consists of 1,711 divestments. (p. 1277)

As a second example regarding best practices about sampling procedures, Follmer, Talbot, Kristof-Brown, Astrove, and Billsberry (2018) wrote that they

used responses from Phase 1 to design the sampling approach for Phase 2 by evaluating when episodes of misfit were most likely to occur and who was most likely to try to address misfit, rather than quickly leave the organization…we pursued a theoretically driven sampling strategy for Phase 2…. Although leaving is one viable response to misfit, our particular interest was in people who chose to remain in poorly fitting environments. Therefore, we approached people who had recently experienced a change at work and those who had invested in their careers through obtaining a higher-level degree. To gather this sample, we recruited from the LinkedIn and Facebook groups for the alumni associations of a college of business in a large, Midwestern, state university and a small, East Coast, liberal arts college. Respondents qualified for the study if they were currently employed at least part-time and were experiencing misfit due to a recent change at work. (pp. 444-445)

Missing Data Management

Directly related to the aforementioned discussion regarding sampling procedures, it is likely that the resulting data set includes missing data (Newman, 2014; Schafer & Graham, 2002). This is an issue not only in the context of micro research involving individual-level data collected using surveys (Aguinis, Pierce, Bosco, & Muslin, 2009) but also macro research involving firm-level data available in archival sources and publicly or commercially available databases (Ketchen, Ireland, & Baker, 2013).

There are several questions related to missing data management that submitted manuscripts should address. For example, if data imputation approaches were used, what specific techniques were implemented, and which particular software package was used? What are the assumptions in the implementation of these procedures (e.g., data missing at random)? Were some of the initially collected data excluded from analysis, and if yes, why? What was the approach to eliminating firms/individuals from the study based on missing data concerns (e.g., listwise, pairwise)?

As an example of best practices, Kalaignanam, Kushwaha, Steenkamp, and Tuli (2013) examined for which types of firms and contexts it is more or less beneficial to outsource customer-relationship management. Regarding missing data management, they reported that they

were able to collect data on IT expenses for 119 of the 158 observations in our sample. For the remaining 39 cases, where IT expenses are missing in a particular year, we used the following heuristic. We searched the 10-K statements and annual reports for the firm in previous and subsequent financial years and replaced the missing value with IT expense from the previous available year. If IT expenses of a firm were not available for any financial year (11 of 39 cases), we replaced the missing value with the average IT expense in the industry (i.e., four-digit or three-digit SIC code) for the given year. (p. 757)

As a second illustration regarding best practices about missing data management, Antonakis, House, and Simonton (2017) studied whether the relation between intelligence and perceived leadership might be more accurately described by a curvilinear rather than a linear function. Regarding how they managed missing data, they explained the following:

Because data were missing only on one variable (IQ), we could not directly test the assumption of missing completely at random (MCAR)…. Thus, using the full sample we created a variable “missing” coded 0 for when data is complete, or 1 otherwise and regressed the seven fully measured leader individual-difference characteristics (i.e., personality, gender and age) on the variable “missing” and all the fixed effects…the variable “missing” was unrelated both individually and jointly to any of the individual difference measures…. As a further test, we examined the MCAR assumption…by performing two Monte Carlo simulations…. The mean p value of the MCAR test was .86 (SE .003; 95% CI [.85 to .87]). Out of the 5,000 simulations, the test was only significant 18 times (0.36%). Thus, overall, the listwise sample with full observations appears to be MCAR. Still, in reporting, we include too results from the full sample using Stata’s MLMV estimator—maximum likelihood estimator for missing data in the event that data are MAR or missing at random (which is not testable); the MLMV estimator is still consistent under MAR assumptions. (pp. 1008-1009)

Outlier Management

Outliers are data points that lie far from others (Aguinis, Gottfredson, & Joo, 2013). Because their location is so far from other data points, they often have an outsized influence on parameter estimates as well as standard errors, which are used for hypothesis testing and computing p values as well as confidence intervals. Thus, different ways of managing outliers usually alter substantive research conclusions (Aguinis & Joo, 2015).

Submitted manuscripts should address the following questions: What were the specific rules used for defining, identifying, and handling outliers? Were outlying data points error outliers, influential outliers, or interesting outliers? How many outlying data points were deleted from the final sample? What would the results be if the deleted observations had been included? Answering these questions is critical for drawing substantive conclusions and also for future reproducibility attempts.

As an example of best practices regarding outlier management, consider a study by Dineen, Duffy, Henle, and Kiyoung (2017) on how a painful social comparative emotion (i.e., job search envy) transmutes as deviant or normative résumé fraud. In describing how they managed outliers, they reported that they

screened for outliers using Bollen and Jackman’s (1990) conservative criteria for the standardized dFits diagnostic statistic. This statistic offers a balance between identifying studentized residuals and influential cases. We also examined the raw data to ensure that the dependent variable for any identified outlying cases was at least four standard deviations from the mean. Using this combined procedure we eliminated one outlying case, for a final sample size of 334. (p. 302)

As a second example, Neckebrouck, Schulze, and Zellweger (2018) tested competing propositions about the impact of family on employment practices and reported, “After winsorizing our data at the 1% level to reduce the potential influence of outliers, our final data set consisted of 102,094 firm-years of data from 14,961 private firms” (p. 560). In other words, Neckebrouck et al. explained that they windsorized their data and precisely reported the final size of their sample.

Use of Corrections for Statistical and Methodological Artifacts

The development of valid and reliable measures continues to be an important challenge in organizational research (Aguinis, Henle, & Ostroff, 2001; Bliese, 2018). We are still a long way from the type of effective and precise measures that are in use in other scientific fields, such as the electron microscope in biology, the Hubble telescope in astronomy, and the teleseismometer in geology. A clear sign of the need to improve measurement is that reliability estimates of about .80 or higher are usually considered adequate—meaning that 20% of variance in observed scores is completely random. This level of error is completely unacceptable in older and more established fields such as physics as well as applied fields such as engineering. For example, this level of measurement error would surely have prevented the successful landing of the Rosetta orbiter on a distant comet after chasing it in its orbit around the sun at a speed of about 19,000 miles per hour for 12 years (Gilbert, 2016).

Acknowledging the fallibility of observed scores, organizational researchers often implement a variety of procedures to “correct” measures for what are often labeled methodological and statistical artifacts (Le, Schmidt, & Putka, 2009). The goal of these corrections is to learn what scores and relations between scores would look like if measures were error free (i.e., no sampling error, no measurement error, no range restriction; Schmidt & Hunter, 2014). But, there are several questions that researchers need to answer in their submitted manuscripts. For example, for all corrections, what were the procedures used (e.g., artifact distributions), and what were the assumptions involved? Was the measurement error correction applied to the antecedent variable, outcome variable, or both? If a range restriction correction was used, was it for direct or indirect range restriction, or both?

As an example of best practices regarding this issue, Huang, Ryan, Zabel, and Palmer (2014) implemented several corrections in their meta-analysis of the relation between personality and adaptive performance at work. They reported the following:

We corrected for measurement error in the criterion using internal consistency estimates for the overall adaptive performance measure at the study level…we corrected for direct or indirect range restriction at the study level using information from the HPI manual…. For the six samples with unknown validation design, we adopted the direct range restriction formula on these datasets to provide a conservative estimate…. As no interrater reliability estimates were available, the corrected validity estimates in the current study would be the lower bound estimates for operational validity. (p. 6)

Another example of best practices regarding corrections is from Shaffer and Postlethwaite (2012) in their meta-analysis to compare the validities of general noncontextualized personality measures and work-specific contextualized measures. They reported the following:

We analyzed our data using the methods developed by Hunter and Schmidt (2004). This method computes the sample-weighted mean of the observed correlations and observed standard deviations from the original studies and then corrects them for statistical artifacts, including predictor unreliability, criterion unreliability, and range restriction…. We computed separate artifact distributions for each of the Big Five traits based on the data available in our data set. Because it is extremely unlikely that the individuals in our sample were selected top-down based on personality scores, corrections for direct range restriction were not appropriate for use in this study. Therefore, in order to correct for range restriction we used the procedures for correcting for indirect range restriction outlined by Hunter and Schmidt (2004). We computed a separate u_x value for each Big Five trait by combining data from each of the studies in our data set with the normative data provided in the test manuals for the various personality scales that were included in our data set. We also computed estimates of predictor reliability…. Although we did not correct for unreliability in the predictor, predictor reliabilities were used to test Hypothesis 3 and in the process of correcting for indirect range restriction. (p. 12)

Data Transformations

There is increased awareness that in many research domains, variables are not normally distributed; that is, they do not follow a Gaussian pattern in which scores are clustered around the mean, the mean and median are in the same location, and there are fairly small tails to the left and to the right of the mean (Aguinis & O’Boyle, 2014). For example, recent research in the domain of individual performance has provided evidence regarding the prevalence of heavy-tailed distributions due to the presence of star performers (e.g., Aguinis, Ji, & Joo, 2018; Joo, Aguinis, & Bradley, 2017). Similarly, firm performance and firm size (Stanley et al., 1995) as well as firm revenue and revenue growth (Crawford, Aguinis, Lichtenstein, Davidsson, & McKelvey, 2015) also follow non-normal distributions

A common practice is to implement nonlinear transformations to normalize scores (Becker, Robertson, & Vandenberg, 2019). As documented by Becker et al. (2019), researchers also transform their data to reduce heteroscedasticity, linearize relations among variables, change relations between variables from multiplicative to additive, and promote analytical convenience (i.e., to be able to use familiar data-analytic techniques based on the general lineal model; Cohen, Cohen, West, & Aiken, 2003). However, based on their review of 323 articles published between 2012 and 2017, Becker et al. showed that many articles do not include sufficient description of the transformation procedures, their justification, and their effects. So, there is a need to answer the following questions in submitted manuscripts: What were the effects of the transformation? Were hypotheses stated in terms of untransformed scores actually tested with transformed data? Which particular variables were transformed, what was the specific procedure used, and what is the justification for the particular procedure?

As an example of best practices regarding data transformations, consider a study by Courtright, Gardner, Smith, Mccormick, and Colbert (2016) on the relation between ego depletion and abusive supervision. Specifically, they noted:

Given the low mean level of abusive supervision in our sample, we conducted an analysis with a data transformation appropriate for skewed data by squaring the abusive supervision variable (Tabachnick & Fidell, 1996). In doing so, we found results that mirrored those of our original analysis, with support for tested hypotheses remaining statistically significant. Thus, for the sake of interpreting our results in a parsimonious fashion, we report the results of the standard linear regression analyses. (p. 1638)

A macro-level illustration of the procedure used and rationale for transformation is McDonnell and Brayden’s (2013) study examining how firms’ strategic reactions differ based on their reputation when they are targeted by consumer boycotts. The authors provided a detailed description of how they transformed the Fortune reputation ranking score as follows:

Each organization included in the list is given a raw score that ranges from 0 to 10. Scores for roughly 37 percent of the firms in our sample are not reported in the Fortune rankings, however, an indicator that these firms are not central players in the field. The raw scores are also unevenly distributed…most scores vary between 6 and 8. Following prior research…we adopted an ordinal transformation of the raw Fortune scores to account for the skewed distribution of the scores and to demonstrate a firm’s relative reputational position…. To create the ordinal transformation of the reputation variable, we used Stata’s xtile function to evenly divide the raw scores into three quantiles, or tiers, of ranked firms. We recalculated the tiers for each year, using the scores of every firm in the reputation index. Because we recalculated quantile membership for each year, raw scores vary in their distribution among the quantiles, depending on the shape of the distribution of the raw scores that year…. A score of “1” was given to companies in the lowest third of Fortune’s annual index in a given year; companies in the middle tier of the rankings in their year received a value of “2,” and the highest value, “3,” was allotted to all companies in the top third. We assigned unranked firms a value of zero to reflect their peripheral field position…. By opting for an ordinal scale, we acknowledge that variation in the rankings is important, but we also operate under the assumption that firms that are not covered in Fortune’s index lack the same reputational standing as those that are covered. (p. 398)