Abstract
Archival proxies have long played a central role within strategic management research, but the degree to which archival proxies are construct valid measures of theoretical constructs remains a source of concern. In some cases, there does not appear to be a close association between an archival proxy and the construct that the proxy is meant to capture. In this brief commentary, we discuss the use of three prominent archival proxies (research and development intensity, patent counts, and patent citations) within recent articles in three leading journals. Each of these measures has been used to represent a wide variety of constructs, which creates challenges when interpreting findings. We then offer three suggestions for improving the use of archival proxies. Implementation of these suggestions would enhance knowledge development within the strategic management field.
Keywords
And so castles made of sand fall in the sea, eventually.
Across the past several decades, archival proxies have played a significant role in the development of the strategic management field. Understanding why this is so is easy given that most strategic management studies adopt the firm or some agglomeration of firms such as the strategic group or the industry as their primary level of analysis (Nag, Hambrick, & Chen, 2007). Meanwhile, a wide variety of quantitative measures are recorded at the firm level of analysis and then made available to the public. These measures are readily accessible; many can be obtained for free via electronic downloading. Relying on archival proxies to gather data in this manner is less complicated compared to the process associated with gaining access to sites and parties for the purpose of collecting primary data dealing with a phenomenon that interests the researcher.
Archival proxies have proven to be valuable assets to strategic management researchers, but their use has also been a source of concern. A key issue is the construct validity of an archival proxy. Construct validity is defined as how close of an association actually exists between a measure and the theoretical construct that the measure is meant to represent or appropriately capture (cf. Campbell & Fiske, 1959). In considering construct validity, a researcher asks the question, “How confident am I that I am measuring what I claim to be measuring?” Some empirical methods that are popular within organizational studies can offer a basis for strong confidence. Lab studies, for example, can be designed to isolate the role of a particular phenomenon in shaping an outcome. When using a questionnaire, a researcher may have the luxury of drawing on scales that have a body of evidence tying them to underlying constructs. If not, a series of steps can be performed to develop a new scale that taps directly into a construct of interest. Despite these advantages, Boyd, Haynes, Hitt, Bergh, and Ketchen (2012) found that strategy researchers’ use of surveys and laboratory studies declined from the 1980s to the 2000s whereas their use of archival data increased. This shift toward greater reliance on archival data highlights the need for high-quality practices when using proxies.
The use of archival proxies introduces unique challenges vis-à-vis construct validity because by definition a proxy is somewhat removed from the construct of interest. This places an extra burden on the researcher to demonstrate that a measure is solid. As shown in Figure 1, good proxies offer a very high degree of conceptual overlap with their corresponding constructs (i.e., strong construct validity). Unfortunately, some archival proxies that are relied on within strategic management studies resemble the lower portion of Figure 1 in that they appear to be poor approximations of underlying constructs (Boyd, Gove, & Hitt, 2005; Bromiley & Johnson, 2005). When a measure’s construct validity is low, the credibility of empirical findings involving the measure is also low. As a result, findings that rely on one or more poor archival proxies are much like the castles made of sand that music legend Jimi Hendrix sang about in 1967—they may look impressive initially, but they rest on a faulty foundation.
A visual representation of proxies with strong and weak construct validity.
The purpose of this brief commentary is to “set the stage” for better knowledge development within strategic management studies that rely on archival proxies. We encourage more careful use of archival proxies by discussing the use of three prominent archival proxies within recent articles in three leading journals. We also provide suggestions for how to validate that an archival proxy is indeed a reasonable approximation of a theoretical construct.
The Diverse Use of Three Archival Proxies
In a recent editorial, Bettis (2012) expressed concern about “the fact that multiple (perhaps numerous) researchers with similar interests will be using the same databases at any time and also across time” (p. 109). This situation is problematic, according to Bettis, because the nature of hypothesis testing within strategic management virtually ensures that relationships are being falsely identified as significant when they in fact are not significant (i.e., Type I errors are being made). We build on Bettis’s concern by pointing out that across the studies using various prominent databases, it is possible—and in some cases likely—that different researchers are using the same measure to represent different constructs. Within such a scenario, confusion about construct validity arises alongside the potential for Type I error. The result is lower confidence in findings.
As an illustration of how archival proxies are used to assess various constructs, we gathered all uses of three measures (research and development intensity, patent counts, and patent citations) within empirical articles in the Strategic Management Journal published from January 2002 to May 2012 as well as within strategic management articles published during the same period in the Academy of Management Journal and the Journal of Management. We focused on these three measures because of their prominence within the strategy literature (e.g., Bromiley & Johnson, 2005; Ketchen, Boyd, & Bergh, 2008). Although our approach to gathering examples was systematic, we did not attempt to complete a formal content analysis. In addition, using a different set of journals and/or a different set of proxies to complete this type of task would have the potential to yield different results.
As shown in Table 1, research and development intensity has been used to measure a broad array of constructs. In some cases, researchers appear to have achieved a tight coupling between the measure and its corresponding construct. In other cases, however, the degree of match is arguable. For example, some studies appear to equate activities such as investment levels with outcomes such as innovations. This calls to mind a quote from Apple’s cofounder and legendary entrepreneur Steve Jobs in the November 9, 1998, issue of Fortune magazine:
Innovation has nothing to do with how many dollars you have. When Apple came up with the Mac, IBM was spending at least 100 times more on R&D. It’s not about money. It’s about the people you have, how you’re led, and how much you get it.
The Use of R&D Intensity as an Archival Proxy.
Note: R&D = research and development; AMJ = Academy of Management Journal; JOM = Journal of Management; SMJ = Strategic Management Journal.
Tables 2 and 3 offer similar displays related to patent counts and patent citations, respectively. Several studies have relied on patent counts to represent the quantity of a firm’s innovations or inventions (Joshi & Nerkar, 2011; Kotha, Zheng, & George, 2011; Makri, Hitt, & Lane, 2010). This approach seems face valid in part because new inventions that are meaningful departures from past innovations are usually patented. Also, overlap between the relevant construct and its measure is enhanced in these studies by the fact that both center on the notion of quantity. Less convincing is the use of patent counts to measure technological capabilities. One troublesome issue is that applying for and/or receiving a patent does not demonstrate that a firm has the capability to develop the technology or put it to commercial use. In slightly different words, receiving a patent may capture an invention but may fail to capture the set of innovation-based abilities needed to commercialize that invention.
The Use of Patent Counts as an Archival Proxy.
Note: R&D = research and development; M&A = mergers and acquisitions; PC = personal computers; AMJ = Academy of Management Journal; JOM = Journal of Management; SMJ = Strategic Management Journal.
The Use of Patent Citations as an Archival Proxy.
Note: R&D = research and development; M&A = mergers and acquisitions; AMJ = Academy of Management Journal; JOM = Journal of Management; SMJ = Strategic Management Journal.
Past patents are cited in new patent applications in an effort to show the intellectual lineage of the new patent. It therefore seems reasonable to assert that the more a patent is cited, the greater the impact on subsequent developments the patent has had and the more the knowledge that is embedded in the patent has been utilized. We therefore view the use of patent citations to measure technological or invention impact (Kotha et al., 2011; Miller, Fern, & Cardinal, 2007) and knowledge utilization (Vasudeva & Anand, 2011) as offering high face validity.
Adopting a broader perspective, however, suggests that even if a proxy appears to be reasonable within the context of an individual study, its use among different studies to represent different constructs creates serious problems. Suppose, for example, an archival measure is used in five different studies to measure five different constructs. It would be difficult to believe that all five uses can be correct, even if each of them appears face valid when considered in isolation. Taking this issue one step further, the interpretation of findings involving such a measure is quite difficult. Consider a situation wherein a scholar is (a) interested in the relationship between strategic investments and market share and (b) has used research and development intensity as a proxy for the construct of strategic investments. If an empirical test shows that research and development intensity is indeed related to market share, there appears to be no inherent basis to be confident that this effect reflects the role of strategic investments rather than those of the various alternatives listed in Table 1, such as technological capabilities, managerial discretion, asset specificity, and absorptive capacity.
Toward Stronger Proxies
Before examining how to nudge the strategic management field away from relying on questionable proxies, it is useful to first consider why questionable proxies have been tolerated in the past. One possible reason is that knowledge development within the social sciences is a socially constructed process wherein certain practices become institutionalized (Mizruchi & Fein, 1999).
In particular, it seems that perhaps the strategic management field has engaged in a process that Popper (1959) referred to as conventionalism. As Ferguson and Ketchen (1999) note, conventionalism is
the tendency of scientists to agree on issues rather than debate them and then act as if what they have agreed upon is scientific fact. The actual foundation for the use of 0.05 [as the cutoff for statistical significance] is not logic, but simply an ad populum argument: because we all believe, it must be true. (p. 387)
It seems likely that the use of certain accounting measures—research and development intensity, advertising intensity, selling, marketing, and administrative expenses, among others—as archival proxies has simply been accepted as reasonable over time. Any scholar wishing to rely on these measures can draw from and cite a large array of previous studies that also used the measures. In addition, the acceptance of these measures by scholars within other business disciplines (e.g., accounting and finance) appears to further support the use of such measures.
A second possible reason why questionable proxies have been tolerated in the past has its roots in the time-tested mantra that “all research is flawed.” Because all research designs have limitations, editors and reviewers have been encouraged to consider a study’s weaknesses within the context of the quality of the entire research design. With the notion of trade-offs in mind, it seems likely that the presence of one or more questionable proxies might be forgiven by gatekeepers to the extent that a study examines important questions, builds valuable theory, tests interesting hypotheses, relies on a suitable sample, and includes other measures that are robust.
These two possible explanations lead us to offer three actionable suggestions for improving the use of archival proxies within strategic management research. First, scholars using an archival measure as a proxy for a construct should identify whether the measure has been used in the past as a proxy for another construct and, if so, demonstrate which usage is correct. Second, rather than authors and gatekeepers assuming that the use of a particular proxy is reasonable (i.e., conventionalism), gatekeepers should actively break free of institutionalized norms (Mizruchi & Fein, 1999) by requiring authors to provide sound logic for and empirical validation of their archival proxies. Providing sound logic to support a proxy involves explaining why the author believes a very significant conceptual overlap exists between the proxy and the construct that it is purported to assess. Proxies that require difficult-to-defend assumptions about causality are unlikely to pass this test. In terms of empirical validation, we recommend that scholars draw on insights from industry experts such as other scholars, executives, investment officers, and stock analysts. Such experts could serve as a helpful validity check. A good exemplar is Hambrick and Abrahamson (1995). These authors compared archival measures of managerial discretion, the views of a panel of academics on discretion, and the views of a panel of security analysts on discretion. Although the three assessments were significantly correlated, some archival measures fared better than others in this triangulation process. Indeed, two archival measures of discretion—regulation and demand instability—were unrelated to panelists’ assessments, possibly reducing the confidence that could be placed in findings that rely on these measures as proxies for managerial discretion.
Third, rather than viewing a dubious proxy as an excusable limitation within an otherwise reasonable research design, gatekeepers should treat poor proxies as potentially fatal flaws. We believe that gatekeepers should view measures’ role within research studies as akin to what stilts are to a beach house. If any one of the stilts supporting a beach house is faulty, a section of the house will inevitably fall into the sea, regardless of the quality of the other stilts and the craftsmanship embedded in other elements of the house. Similarly, if a measure does not effectively capture the construct that it is purported to capture, any statistical tests involving the measure cannot be considered valid. Authors should therefore be required to replace a faulty proxy with a defensible one or to remove hypothesis tests that involve the faulty proxy from their study. If one of these two remedies cannot be applied, the study (unfortunately) should be considered fatally flawed.
The actionable suggestions we offer here have a stronger likelihood of being implemented initially by those serving as gatekeepers for the more highly valued, established, and respected journals. Commonly known as “Class A” outlets or “top-tier” journals, these publishing outlets have the requisite level of influence with scholars to request or even demand appropriate adjustments to scholarly work that is eventually accepted for publication. In this manner, the social desirability of the suggestions we offer here has the potential to be rapidly diffused through a cascading approach with other journals publishing strategic management scholarship.
Conclusion
The difficulties surrounding archival proxies with poor construct validity are not limited to strategic management research. More than 50 years ago, Valavanis (1959) noted that
Econometric theory is like an exquisitely balanced French recipe, spelling out precisely with how many turns to mix the sauce, how many carats of spice to add, and for how many milliseconds to bake the mixture at exactly 474 degrees of temperature. But when the statistical cook turns to raw materials, he finds that hearts of cactus fruit are unavailable, so he substitutes chunks of cantaloupe; where the recipe calls for vermicelli he uses shredded wheat; and he substitutes green garment dye for curry, ping-pong balls for turtle’s eggs, and, for Chalifougnac vintage 1883, a can of turpentine. (p. 83)
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.