Does Anyone Care About External Validity? A Call (or Plea?) for More OB/HR Research From Multiple Organizations/Industries,Panels,and Publicly Available Datasets

Are We on the Same Page?

Let us think back to our first doctoral seminars. If you were like me, one of those first seminars was a research methods seminar. Mine was taught by John Hulland (currently at University of Georgia), and I would not be here to this day without him (he knows why). One of the topics we discussed in his seminar was about internal validity and external validity, among the other “validities” you can discuss. Let us all remember that internal validity speaks to causality and external validity speaks to generalizability. Generalizability can come in many different forms. Our field loves to be extremely specific (can you sense my sarcasm here?), so let us get more specific by what I mean when I say generalizability. I am speaking to generalizability about the results of a study. In other words, would the results hold if the same study was conducted with different individuals, professions, organizations, industries, or cultures? If the answer to that question is no or you do not know because of your sample, you have very little (or no) generalizability. If the answer is yes, then at least you have some generalizability. Are we now on the same page about the difference between internal and external validity and what I mean by external validity?

This is Where My Rant Really Begins…

But are we on the same page about the importance of internal and external validity? Since I graduated from my doctoral program in 2011, it seems my field (i.e., OB/HR) has become almost exclusively focused on internal validity when conducting field studies (my research focuses on field studies because I have an acknowledged bias for real employees in real organizations). What happened to caring about external validity?! Now, I know there are both theoretical and empirical reasons for caring so much about internal validity. Robert Ployhart and his crusade to enlighten everyone on change is a great example as to why internal validity matters and how to properly establish internal validity (e.g., Pitariu & Ployhart, 2010; Ployhart & Vandenberg, 2010; Ployhart & Ward, 2011). But it seems to me that the most often cited reason for having two time points in field studies (which is more often the case than having three or more time points from my own reading) is Phillip Podsakoff and friends’ case that it potentially decreases the effects of common method bias (e.g., Podsakoff, MacKenzie, Lee, & Podsakoff, 2003; Podsakoff, MacKenzie, & Podsakoff, 2012). I will not get into the debate as to whether the effects of common method bias are overblown or not (see Spector (2006), for more about that issue), but it seems to me that the quest for establishing internal validity has (1) come at the expense of external validity (e.g., selecting only those entities that will participate over two or more time points may cause a sampling bias in our field—thanks Reviewer 1 for that one!) and/or (2) sent a signal that internal validity is more important than external validity (e.g., authors being held hostage to two or more time points in order to have their papers published).

Is External Validity More Important than Internal Validity?

This brings up the question of whether internal or external validity is more important. You can find many books and articles that argue internal validity should be established before external validity (too many to name here!), implying that internal validity is more important. I actually believe it should be the opposite, in that external validity should be established before internal validity. Assuming your theoretical focus is not about a specific type of employee, organization, profession, industry, or culture, wouldn't you want to know whether the results of a particular study hold true across employees, professions, organizations, industries, or cultures (you preferably should try to have both internal and external validity)? If your results only apply to that specific study sample, why should other employees, professions, organizations, industries, or cultures follow your managerial recommendations? From a theoretical standpoint, why should we take results that only pertain to that specific study sample as theoretical gospel? Although I wish I could say I am the first to point this out, I surely am not. Others have argued for the importance of external validity either directly or indirectly. Examples have included Bergman and Jean’s (2016) argument that research in I/O psychology has focused too much on a particular group of employees and Gary John’s argument (e.g., 2016; 2018) that context is important in understanding what results actually mean.

However, it does not seem that these arguments have had a considerable influence on the way researchers actually conduct their field studies. Let us use the Financial Times 50 list since so many people seem to care so much about this list (I personally do not give a damn about the random lists people choose to follow because I feel lucky that any journal publishes my research). The journals that I would possibly try (almost always in vain!) to publish in on this list often include studies that use field data from very similar employees, or from a single organization, or from a single profession, or from a single industry, or from a single culture. In my eyes, these studies have very low (or really no) external validity. They are basically a case study to me, and as I have often required my students to read in Brown and Keeley (2018), case studies have their drawbacks. Are they useful? Certainly. But are they generalizable? Oftentimes, not so much. At best, we know there might be a linear relationship across two time points or true change (remember Ployhart and his crusade about needing at least three time points!) within that specific group of employees, organization, profession, industry, or culture. And this is assuming nefarious choices regarding how to handle the data and testing were not made, such as selecting control variables that provide the “best” results, rather than using a methodical approach for control variables (e.g., Bernerth & Aguinis, 2016).

You are probably thinking in your mind right now that we do not need to be so concerned with external validity within a single study. That is what literature reviews, meta-analyses, and replication studies are for, right? For literature reviews and meta-analyses, why would you want to wait for at least five to ten years to be able to draw any definitive conclusions about external validity? For replication studies, where are they actually published? Although our field says they are important and are greatly needed, I do not see them regularly published in any of our “premier” journals (I use quotations because that it is a very subjective term to me—refer back to my belief about journal lists). This says to me that replication studies exist, but (1) are not rigorous enough, (2) do not make a “significant” theoretical contribution (what do you believe my quotation marks mean?), or (3) most scary, are not published because they do not align with prior research findings. I would really like to offer my opinion of which of these three is correct, but I will probably be in hot water already for this rant and will therefore keep my opinion to myself. Someone should really start a journal called the Journal of Management Replication Studies though. It would probably be the most cited journal in our field. Back to the topic at hand!

My Three Recommendations for Authors

Assuming that we do not want to wait five to ten years and that replication studies will still not be valued behind closed doors, I strongly believe authors should make a concerted effort to increase the external validity of their studies before worrying about internal validity (am I a heretic in my field now?). There are times when journals request authors to conduct multiple studies, which can potentially increase external validity. But for those authors who have limited resources, such as myself, this may not be a viable possibility and we are stuck with trying to increase external validity within a single study. This means that we need to be extra vigilant in thinking about how we can increase the external validity, in at least an empirical sense and possibly a theoretical sense, from the very beginning of our research projects. Do we want our results to generalize across different employees, different professions, different organizations, different industries, and/or different cultures? My plea to you is to achieve at least one, but aim for more than one. I offer three humble recommendations for doing so.

My first recommendation is to collect data from multiple organizations/industries (mind blown, right?). Not to toot my own horn or anything (but I am going to because being able to write this rant in GOM gives me an inflated sense of self-importance), this is what my co-authors and I often do in our own studies. Using my articles published in journals from the aforementioned Financial Times 50 list that so many people care about, we collected data from six organizations (Scott et al., 2014) and 111 teams across 24 organizations (Schabram, Robinson, & Cruz, 2018; Study 1). Just to prove that my co-authors and I use this approach as much as possible, irrespective of the particular journal “quality,” my articles in “lesser” journals used a similar approach (e.g., Cruz Zagenczyk, Scott, Thoroughgood, & Cheung, 2018; Cruz & Pinto, 2019; Cruz, Zagenczyk, & Hood, 2020). I was once giving a job talk and one of the researchers in the room said I should write an article about the process I follow to collect data using this approach, but the Editor tells me I only have a specific number of pages and I am already over that limit (sorry for those who wanted to know!). Utilizing this approach can at least show that your results generally hold true across different organizations/industries and are not a function of the specific context (remembers Johns!) of a single organization or industry.

This is not the only way you can increase the generalizability of your findings though. Because I now have an inflated sense of self-importance, I am going to toot my own horn about another way my co-authors and I have increased the external validity of a single study: online panels (e.g., Zagenczyk et al., 2013). Online panels can easily increase the external validity of your findings because the data are typically collected from employees who work in a relatively wide range of positions, organizations, industries, and possibly cultures. At the same time, online panel data can give you similar data properties as “traditional” data (Porter et al., 2019; Walter et al., 2019). Like anything else, there are trade-offs using this approach that should be noted. Unlike collecting the data on your own, per my first recommendation, you often do not have the ability to capture nuances beyond the individual-level, such as about the teams and organizations study participants are members of, because of limitations in how panel data are collected. However, online panels can potentially provide you a wider breadth of employees, professions, or cultures than could be obtained by collecting data on your own. Thus, my second recommendation to increase external validity in a single study is to utilize online panel data that are available for purchase through a number of different organizations.

You do not have to collect the data yourself or purchase it though. There are many readily accessible datasets collected by public entities that you can utilize to increase your external validity. These public entities obviously have much larger resources at their disposal and can therefore collect vastly larger amounts of data than individual researchers can collect. These data are typically from employees and teams across many organizations and industries. For example, we obtained access to the 2004 UK Workplace Employment Relations Survey (Department Trade and Industry, 2005) in Cruz and Pil (2011). While we were constrained to the choices made by others of which constructs were and were not included in this dataset, utilizing this dataset provided a degree of external validity that we could have never likely obtained in our own data collection efforts or through online panels. Hence, my third, and final recommendation, is to utilize publicly available datasets.

You Are Not Off of the Hook Reviewers: My Recommendations

Although my work does not always have multiple time points, I can at least say my work has a relatively large degree of external validity and I think they provide a foundation for then attempting to show causality. I know, I know, I should try to do both in the same study, but let us talk about that from a practical perspective. Let us use my Schabram et al. (2018) article as an example. Assuming we are not in a country in which respondents do not “voluntarily” participate in our studies at questionably high rates, it would be very difficult to have all 24 organizations agree to allow their team members to complete surveys across two or more points in time at the same exact time (to control for time effects) and at a sufficiently high enough response rate to have enough data to publish an article, especially for teams data which requires at least two or more respondents (right Michael Kukenberger—perhaps a future GOMusing?). Believe me, I am trying! Thus, I believe reviewers should not continue holding authors hostage to two or more time points if the authors are making concerted efforts to increase the external validity of their findings. This is especially true if cross-sectional data are warranted (Spector, 2019).

Furthermore, if you see authors making a concerted effort to increase the external validity of their findings, do not question the external validity of their findings like you do (I bet that you probably do not) for studies with employees from only one organization using two or more time points. I recently revised and resubmitted a paper to an unnamed journal that had a much higher degree of external validity than over 90% of published papers (and I had two time points—sorry it was not three Ployhart) and the generalizability of the findings were questioned. It was quite simply unbelievable to me. And one might argue that sacrificing some internal validity expectations in the name of external validity may actually lead to greater internal validity in the long run (hmmm…maybe another rant…again, thanks to Reviewer 1 for that one!).

The end of my rant (you are probably saying “finally!”)

So, to reiterate my answer to my own question: YES, we should all care about external validity and to a much larger degree than I have seen in the past 10 years. Just do it (please do not sue me Nike) by following my recommendations. Okay, rant over, at least my rant. Sorry for the length (blame the reviewers, not me, right?). Did I accomplish at least one goal of these rants? We shall see. If I did, please feel free to email me and tell me how great I am. If I did not, or you do not want to tell me how great I am, please do not email me. This will help confirm my bias for the importance of external validity. Who wants to rant next?