The Problem of Research Method Informing Research Questions in Leadership Research

What we teach our students (and learned ourselves) is that research method follows research question. Whether theory-driven or phenomenon-driven, the idea would be that we identify what we want to study and then determine what the most appropriate method is to study this. Drawing on personal observations – this is an opinion essay and I share my observations rather than data to build my argument – I contend that this is not how things typically work in leadership research. What is much more common, in my experience, is that we let research method preferences inform research questions, such that we study what we can study with our prefered methods. It is not the research question that results in the choice of research method, but the prefered research method that informs the research question. This essay is about why this is a problem for the development of leadership research and what we can do to address this issue.

My argument is that because research method preferences guide research questions, and leadership research is characterized by a strong preference for survey methods, this low diversity in research methods used is associated with low diversity in research questions studied. That is, in leadership research, we predominantly study what can be studied in survey research and as a result we seriously limit the consideration set of potential research questions. This is a problem because it means that many important research questions are likely to get little or no attention because they lend themselves less – or not at all – to survey research. To advance the field of leadership research we need to address this issue. We can only do so by stimulating greater diversity in research methods used.

Why Does our Field Have so Little Diversity in Methods Used?

My starting observation is that the reason that leadership research is dominated by studies relying on survey methodology is not because the most important research questions are most appropriately studied with surveys. Rather, surveys have become the preferred research method for a variety of reasons unrelated to the importance of the research questions considered. This methods preference leads many researchers to develop ideas that can be studied in survey research. Before elaborating on why this is a problem and what we can do to address that problem, let us first consider why surveys are the dominant research method – and more broadly why there is low diversity in the research methods used in leadership research.

From the perspective of the researcher, there are several incentives to prefer more common over less common research methods. It stands to reason that we feel more comfortable designing research that involves research methods we are more familiar with. For the methods we are most familiar with, we have the greatest understanding of how to use them, how to secure the data access needed, how to analyze the data, and how to write papers for journal submission. The methods we use may also be associated with the resources available to us. Choice of method may for instance be contingent on access to facilities for laboratory experiments or access to colleagues with experience with the method as potential sources of advice.

To the extent that we build on earlier work – and we typically do – there is also likely to be a shared methods tradition in the particular stream of research we are working in (e.g., survey research on abusive supervision sparked first and foremost survey research on abusive supervision, Mackey, Frieder, Brees, & Martinko, 2017; qualitative research on leader sensegiving primarily inspired more qualitative work on leader sensegiving, van Knippenberg, 2016). Such research traditions in effect provide “role models”, such that one's own research within the particular stream of research can be modeled on the methods used in earlier work within that stream of research. Moreover, such role models often seem to take on a role that is not purely informational, but also normative. Earlier work shapes the understanding of researchers, editors, and reviewers of what a study in that stream of research “should” look like. For instance, many researchers studying leadership in lab experiments have experience with journal rejections based on the argument that leadership cannot be studied in the lab. This belief that lab experiments are inappropriate to study leadership is not supported by the evidence that lab findings generalize to the field (e.g., van Knippenberg & van Knippenberg, 2005), but is consistent with the idea that the dominance of survey research takes on a normative aspect in which survey research is understood to be the more appropriate method. Such descriptive (“what is”) and injunctive (“what should be”) norms have a self-perpetuating effect in that they implicitly convey to researchers what the more appropriate methods are to study the phenomena of interest. When researchers act on these norms and use the dominant method, this reinforces the norm. Such norms also subjectively legitimize that methods education focuses primarily on the methods most used in the field of research, which both helps sustain the norm and helps build methods preferences based on familiarity.

Adopting a new or infrequently used method also comes with the risk that editors and reviewers are less able to appreciate the value of the study when the focus is on phenomena not studied with more common methods (and speaking from experience, a common response of editors and reviewers to methods they are not familiar with is to question their validity). This probably holds especially true when the less common method goes hand in hand with a less common research question – and less common research questions are often the very reason a less common method is adopted. For example, recent neurocognitive research in leadership puts variables and processes on the agenda that were never studied before neurocognition methods arrived on the scene, because the methods are explicitly developed to measure processes that are impossible to assess with other methods (e.g., Waldman, Wang, Hannah, & Balthazard, 2017). From their own experience and those of others, researchers may thus learn that adopting less common research methods may result in an uphill battle in the publication process. This too incentivizes going down the beaten path.

The dominance of survey methodology in leadership research is also understandable from pragmatic considerations. Compared to other methods used with some frequency in leadership research (e.g., lab and field experiments, qualitative research), survey research can typically be done in less time, with less costs, or with less demands in terms of resources, invasiveness for participating organizations and respondents, and the like. Many researchers for instance do not have access to facilities for experimental research, are unable to convince companies to partner on a field experiment, or see the timelines involved in qualitative research as unfeasible in their career trajectory vis-à-vis job market, tenure, or promotion requirements. Survey research simply is the most accessible research method to most researchers and moreover one that allows researchers to produce a relatively high volume of studies within salient time frames such as those for a PhD or the tenure track. This is not to say that we therefore should accept the dominance of survey research, but we should recognize that stimulating more diversity in research methods is easier said than done. The deck is stacked heavily in favor of survey research and only when we create situations in which other methods have an advantage can we expect to incentivize the use of less common methods.

Survey research in leadership undoubtedly “has it easiest”, but the diagnosis I outline here also applies to other research methods encountered with some frequency in leadership research. For instance, even though there is far less experimental research than survey research in leadership, there is a steady flow of experimental research too. This has created its own tradition of sorts and in research streams where experiments are relatively prevalent this created its own role models (e.g., in the social identity analysis of leadership; Steffens, Munt, van Knippenberg, Platow, & Haslam, 2021). Even when, as per my observations above, researchers using experimental research may be confronted with the dominance of survey research in the publication process, experiments most likely are an easier method to adopt for leadership researchers than less common methods because of many of the processes outlined here (i.e., researcher, editor, and reviewer familiarity; implicit norms created by previous work; resource availability). That is, also when leadership research is not survey research, there are incentives unrelated to the importance of the research question studied to encourage adoption of more common over less common methods. This too works to sustain the limited diversity in research methods used in leadership research.

Why is Low Diversity of Methods Used A Problem?

To recap, from the perspective of the researcher, there are several reasons to prefer survey research. This feeds into the implicit norm in leadership research that survey research is the standard, creates a situation in which greater familiarity with survey research than with other methods is built and maintained through education, colleagues, and published research, and in which research using other methods is implicitly or explicitly judged in reference to the alternative of doing survey research. From the perspective of leadership as a field of research, this is a problem, because it limits most of our research efforts to what can be studied in survey research (survey research also comes with other limitations, but those are not my concern here; my focus here is on how any method is associated with constraints in research questions studied). What I believe we see as a result of the dominance of survey research in leadership, is not just that the majority of studies are survey studies and thus limited to what can be studied using surveys. We also see that the research questions we typically focus on are so shaped by survey research that studies using other methods often end up studying relationships that could also be studied in a survey. That is, even when we do not conduct survey research, we often develop our research questions from a “survey mindset” – the dominance of survey research leads us to use what can be studied in surveys as our frame of reference in the development of research such that using other methods we frequently end up studying what can also be studied in a survey.

These observations come with no claims whatsoever that I am able to comprehensively list the kind of research questions we are missing out on (or are not paying enough attention to). I too am limited in what I study by the methods I am familiar with. My line of sight is likewise limited, such that I do not consider many questions that would be valuable to address in leadership research but that would require less common or even new methods. With this caveat in mind, what I aim to do next is illustrate the problem of the dominance of survey research and the associated survey mindset through a few examples of the kind of issues that would be important to study, but seem to typically fall outside of our consideration set because of our self-imposed method constraints.

To that effect, I discuss two examples of methods we do use to some extent but that I argue we typically do not use to their full potential as a result of our survey mindset: lab experiments and longitudinal surveys (i.e., surveys with repeated measurement of the same variables, which has unique features compared with the common survey in which each variable is measured only once). These examples are intended to illustrate that even when we deviate from the common survey approach, we tend to use our research methods from the mindset of the common survey with one-time measurement of study variables.

Over the last two decades, we have seen a growing number of leadership studies that combine surveys and lab experiments to replicate hypothesis tests over studies. This clearly has added value in that results that are replicated over methods cannot be attributed to characteristic specific to one of the methods (this multi-method focus seems in important part to have been inspired by editor and reviewer biases against lab experiments in leadership research; when results replicate in surveys, the belief that findings from lab experiments do not generalize is refuted, van Knippenberg & van Knippenberg, 2005). From the perspective of the concern I highlight here, the problem is that this approach of replication over methods means that the experimental method is in effect used to study something that can be studied in a survey. In my reading of the literature, this survey mindset has become the norm to such extent that most lab experiments in leadership test hypotheses that could also be tested in a survey even when a survey is not included in the study.

This in many ways reflects missed opportunities, because the lab environment lends itself to measurement that cannot be conducted in survey research. Bringing participants into the lab for instance allows for behavioral observation. This not only has the advantage that behavior can be assessed through observation by trained assessors and measurement does not rely on the retrospective subjective measurement that surveys by necessity rely on. It also has the advantage that we can study behavior that is less easily “verbalized” in survey ratings by untrained raters, such as the display of emotions, paralinguistic elements of speech such as volume, tone, and speed, body posture and other nonverbal behavior, etc. Another important element of such behavioral observation is that the temporal order of behavior can be determined such that we do not simply know how much certain behavior was displayed in aggregate (i.e., “averaged” over time) but in which order and following which behavior from others (e.g., followers), thus allowing for the study of more dynamic leadership processes. A lab environment also allows for physiological measurement to measure variables that may be highly relevant to leadership processes such as stress reactions, arousal, and neurocognitive processes. Clearly, the costs of such measures (e.g., for equipment, for the time investment of trained coders, etc.) are also an issue, but I believe we typically do not see those kinds of measures in lab experiments in important part because experiments are designed from a survey mindset.

One examples that illustrates the advantage of using the lab environment for measurement that surveys cannot adequately deliver on is found in the study of leadership and emotions. Evidence has been building over the years that emotions can play an important role in leadership effectiveness, as for instance documented in an integrative by van Knippenberg and Van Kleef (2016). One of the things that this review captures is that the study of leadership and emotions is heavily dominated by the distinction between positive (pleasant) and negative (unpleasant) moods and emotions. Research has established that valence is one of the key dimensions capturing similarities and differences between affective states (Russell, 1980) and attention to the role of affective valence is definitely warranted. That does not mean that the almost exclusive reliance in research on studying positive versus negative affect is warranted. This strong focus on affective valence can, however, be understood from the fact that in survey measurement differences between emotions tend to be reduced to positive and negative affect – and likely not because people are not able to make more fine-grained distinctions but because survey measurement is not able to capture such more fine-grained distinctions (Watson & Clark, 1997). Research has established that similarities and differences between emotions can be captured through not just the valence of the emotion but also the physiological arousal associated with the emotion; some emotions (e.g., anger) are associated with higher physiological arousal than others (e.g., sadness; Russell, 1980). The role of the arousal associated with emotions has by and large been ignored in leadership research (cf. Damen, van Knippenberg, & van Knippenberg, 2008). I believe this is in important part the result of our survey mindset that leads us to focus heavily on what can most easily be measured in surveys – affective valence.

Other domains of research suggest, however, that some effects of emotions that have traditionally been attributed to emotional valence, such as the notion that positive emotions lead to more creativity than negative emotions, should be attributed to the arousal associated with the emotion (Baas, De Dreu, & Nijstad, 2008). This raises the question to what extent current conclusions about emotional valence in leadership research reflect arousal effects and to what extent emotional arousal may be as influential in leadership effectiveness as emotional valence. Experimental settings are better suited to study such issues than surveys. In experiments, high versus low arousal emotions can for instance be induced, physiological measurement of arousal can be used to capture the arousal associated with emotions, and behavioral observation by trained observers can be used for finer-grained measurement where surveys fail to capture the arousal dimension underlying emotions.

In considering important issues we largely leave unaddressed, it is also important to realize that not all surveys are created equal. The typical survey in leadership research measures each study variable once (whether this is at one point in time or separated over time is not relevant here). This allows for the test of static models in which one-time measurement of independent variables predicts one-time measurement of dependent variables. Longitudinal surveys in which the same set of variables is measured multiple times, such as experience sampling methods (e.g., Liao, Lee, Johnson, Song, & Liu, 2021), allow researchers to break away from such static models. In my experience as editor and reviewer, however, researchers using such methods often end up testing what are essentially static models that could also be tested in cross-sectional surveys. For instance, the longitudinal set-up may allow researchers to show that within-person variation in an independent variable X predicts within-person variation in a dependent variable Y. This has clear parallels, however, with showing that between-person variation in X predicts between-person variation in Y, which can be done in the common one-time measurement of variables survey design. Between-person variation is not the same as within-person variation, but the theory to predict either typically is the same. That is, even when researchers use longitudinal survey methods rather than the common approach of one-time variable measurement, they often do not let go of the mindset that points to research questions that can also be studied in surveys with one-time measurement of study variables.

As Shamir (2011) outlined, time is an important variable in leadership. Leadership may change over time (McClean, Barnes, Courtright, & Johnson, 2019) and such change may have effects not captured by cross-sectional comparisons (i.e., in addition to the leadership per se, change in leadership can have effects in and of itself). The effects of leadership may also change over time both because it may take time for an effect to develop and because repeated exposure to the same leadership may change its effects. Follower responses to leadership may also inform subsequent leadership such that leadership and followership coevolve in a dynamic process; changes in leadership may not only be cause but also effect in the leadership process. Such dynamics may be captured in longitudinal surveys (when they unfold on the same time scale as the repeated survey measurement; otherwise other methods may be more appropriate, such as behavioral observation for faster unfolding dynamics), but not cross-sectional surveys. The fact that longitudinal research seems to often underutilize its potential to address such research questions, I contend, can be understood from the dominant cross-sectional survey mindset that guides research even when it is not a cross-sectional survey.

How Can We Increase Diversity of Methods Used?

My observation that leadership research suffers from a self-perpetuating limitation in research questions studied as a result of the dominance of cross-sectional surveys is not to argue against the value of survey research. It is to argue that it is important to recognize that the reason survey research is dominant is not that the most relevant, interesting, and important research questions in leadership can all be studied with survey research. The dominance of survey research is limiting to the development of the field, because it leaves many important questions unanswered and often even unasked. As a research community, leadership researchers should consider how this issue can be addressed. Our field would benefit from more diversity of methods used, as methods diversity provides the basis for greater diversity of research questions.

Addressing this issue, a first step would be to reflect on what cannot be studied with survey methods and what we are missing out on by giving these issues little or no attention. Our scholarly journals can stimulate the consideration of such issues by creating a publication platform for such considerations (e.g., reviews of less common research methods, arguments for specific sets of research questions and associated methods, etc.). Taking stock of phenomena that do not lend themselves well to survey research is a step in the process of diversifying the research methods used and research questions asked in leadership research, but it is guaranteed not to be enough. It is easy to bemoan the absence of research on certain topics or processes; it is far more difficult follow up in actions (as a case in point, this essay is much easier to write than the research that would address the issue is to conduct). As I have tried to outline in the previous, important part of this issue is that the incentives for researchers are clearly in favor of conducting more “mainstream” research. What would seem to be particularly useful, then, are incentives that explicitly reward research that relies on less established research methods.

Our scholarly journals are uniquely positioned to create such incentives. Journals may for instance launch special issues or ongoing series (i.e., a series of papers that runs across issues) for which the use of less established methodology is a requirement (combined with an argument that the issue studied cannot be studied with more traditional methods). Such special issues or series would create an incentive because the vast majority of studies that would typically be competitors for journal space would be excluded from competition for these publication opportunities. They would also remove concerns associated with the use of less established methods, because the unusual method is not a liability when unusual methods are what is required. Such special issues or series would be unusual in that they are defined around methods and not around the phenomenon studied; they would allow for integrative conclusions even so in capturing the value-added of unlocking new research questions through less established research methods.

All this is easier said than done and in this respect I share two concluding remarks. First, it may help to realize that the start of this process of diversifying our research methods portfolio is the most difficult. As we build greater diversity in research methods and questions through such “interventions” as special issues or running series, we create role models for others’ use of these less established methods both in terms of the methods used and in terms of the issues studied. Second, it is important to recognize that our journals are uniquely positioned to provide the incentives for the use of less common methods. Researchers are unlikely to receive recognition for the use of less common methods on the job market or in tenure and promotion decisions, for instance, until the use of such methods is instrumental in the publication process. Our scholarly journals – the editorial teams running them – are uniquely positioned to determine how well submissions with less common methods fare in the publication process.