The Critical Role of Methodological Pluralism for Policy-Relevant Empirical Marketing Research

Research can be characterized in terms of three domains (Brinberg and McGrath 1985): (1) the substantive (the real-world problem of focus in the research), (2) the conceptual (the theoretical representation of some aspect of reality), and (3) the methodological (the approach taken to investigate a real-world problem or test theory). In empirical research, all three domains are usually involved, but researchers may emphasize each to different degrees. A distinguishing feature of research in the Journal of Public Policy & Marketing (JPP&M) is that it usually starts with a real-world problem that has important consumer, marketing, and public policy implications (see Martin, Borah, and Scott 2021). Due to its substantive focus, JPP&M articles enjoy an eclectic use of conceptual foundations and methods to explore important real-world problems. In this commentary, we explore JPP&M's methodological domain by conducting an analysis of recent empirical research and providing insights based on our work.

The Nature of JPP&M’s Empirical Research (2017–2021)

We analyzed and classified all contributions JPP&M published in volumes 36 to 40 (for details, see the Web Appendix). Of the 167 contributions, approximately two-thirds were empirical. Analysis of these articles found that just as theories are eclectically used in JPP&M, so are methods. Such a mix of methods has a long tradition at JPP&M and goes back to its first editor, Thomas Kinnear (1982, p. 2), who stated that “all types of research procedures are valued by the journal, including surveys, both laboratory and field experiments, time series, and legal analysis.”

Although lab and field experiments and studies based on observational data (often surveys) have been used most and in equal measure (see Table 1), qualitative studies are also common. There are no pronounced time trends and, overall, the articles over the most recent five-year period show a good mix of diverse methodological approaches, which we consider a strength of JPP&M.

OPEN IN VIEWER

Table 1.

Methodological Approaches and Data Sources in JPP&M Articles (2017–2021).

	Percentage of Articles with Data a	Percentage of All Articles b
Methodological Approaches
Qualitative studies	28%	18%
Experiments	44%	28%
Studies based on observational data	45%	29%
Data Sources
MTurk and Prolific Academic	33%	22%
Research-relevant (sub)populations	24%	16%
Market research companies	19%	12%
Students	18%	11%
Informants in qualitative research	18%	11%
Secondary data sources	15%	10%
Other sources (e.g., ads, tweets)	13%	8%

^a

Percentages do not sum to 100% because some articles use multiple methodological approaches or have more than one data source.

^b

60 out of 167 articles (35.93%) did not include data.

Given the substantive and professed policy-relevant nature of JPP&M, it is somewhat surprising that relatively few secondary data sources are used and that the most prevalent data come from online panel survey firms. We explore these two issues in greater detail next.

Using Secondary Data to Provide Real-World Evidence for Policy Recommendations

The most prevalent data sources in JPP&M articles are Amazon Mechanical Turk (MTurk) and, to a much lesser extent, Prolific Academic. In many instances, these data sources are only a portion of the data reported in an article. Nonetheless, for a journal that aims to provide relevant public policy recommendations, the number of studies based on MTurk or Prolific Academic data seems high, especially because the data collected via these means often involved hypothetical scenario studies. For those relying on such data to address policy issues, we recommend augmenting them with secondary data sources.

A recent JPP&M example of this recommended approach is provided by Ghotbi, Dhar, and Weinberg (2021), who studied whether consumers who choose a diet (vs. regular) soft drink might consume more calories during a meal. Theory suggests that a “healthy” choice may license a consumer to indulge by adding an unhealthy product. The authors used a large secondary data set on food-away-from-home meal consumption, supplemented by additional data on caloric content of food, to study this issue and found that, contrary to licensing theory, those who ordered a diet (vs. regular) drink actually consumed fewer calories within a meal.

There are many secondary data sets used by policy researchers in JPP&M, with examples including stock market data, annual reports, purchase data from retailers and research firms, census data, health insurance data from the federal government, and many more. Particularly promising opportunities exist when researchers merge multiple publicly available secondary data sets to study a research question (e.g., Liu, Gauri, and Jindal 2021). Moreover, the increased availability of social media and other online data has made it possible to extract valuable information from social media posts as well as online reviews. Recent examples of this approach have studied issues such as children's online privacy (Fox and Hoy 2019), far-right opposition to multicultural marketing (Ulver and Laurell 2020), and gambling advertising (Rossi et al. 2021). For authors interested in building strong policy implications for substantive problems, we encourage the creative use of secondary data.

Finding Participants for Policy-Relevant Subpopulations with Low Incidence

A critical decision that any researcher must make is selecting a sampling frame, and this is especially important with vulnerable (and rare) populations, who are often relevant to policy research. Reliance on online data platforms can raise sampling challenges for policy researchers. At any given time, the estimated number of active workers on MTurk ranges from 2,500 to 7,500. ¹ While one could use such panels to study high-incidence-rate subpopulations (e.g., subprime borrowers, 34.8% of Americans), it would be insufficient with lower incidence rates. For example, Rayburn, Mason, and Volkers (2020) wanted to study parents whose children were born in a neonatal intensive care unit. An optimistic calculation suggests that only 11 such participants exist on MTurk, ² unless there is misrepresentation or imbalance (see Sharpe Wessling, Huber, and Netzer 2017). Some have suggested to slowly grow panel subpopulations over time, yet attrition may outpace additions depending on incidence rates (e.g., the half-life on MTurk is approximately 400 days; see Difallah, Filatova, and Ipeirotis 2018). For Rayburn, Mason, and Volkers (2020), the use of panels would have been futile or very expensive.

To address this situation, scholars must intentionally bias their sampling frames to seek data with a theoretically relevant sample. Although one could turn to large commercial survey panels, such panels can be expensive, and it may be possible to partner with organizations that can offer their audiences as participants. For example, Rayburn, Mason, and Volkers (2020) ended up collaborating with a neonatal product company to reach 348 consumers. Another example is Cornwell et al. (2021), who recruited parent–child dyads from preschools to study food choices. Others turned to theoretically justified intercepts in other countries. For example, Arli and Cadeaux (2017) recruited participants both in a red-light district and a university district in Indonesia. Hasan, Lowe, and Petrovici (2019) identified 351 subsistence customers in Bangladesh at tea stalls and kiosks offering money transfers. In all these studies, researchers embraced purposeful selection in high-incidence subpopulations, seeking out partnerships and experimentation opportunities with companies and groups that cater to subpopulations of policy interest—a model that could be more widely employed by JPP&M contributors.

Conclusion

From its inception, JPP&M has embraced various methods to study marketing and public policy. Our analysis of the most recent five years of publications shows this to be the case; JPP&M continues to live up to the original vision for the journal. Given the unique substantive nature of policy-relevant topics, we encourage the field to consider embracing and using sources of data beyond online panels such as MTurk and to be creative in identifying the unique subpopulations that are most often the focus of policy makers.

Supplemental Material