Abstract
Reporting biases refer to a truncated pool of published studies with the resulting suppression or omission of some empirical findings. Such biases can occur in positive research paradigms that try to uncover correlations and causal relationships in the social world by using the empirical methods of (natural) science. Furthermore, reporting biases can come about because of authors who do not write papers that report unfavorable results despite strong efforts made to find previously accepted evidence and because of a higher rejection rate of studies documenting contradictory evidence. Reporting biases are a serious concern because the conclusions of systematic reviews and meta-analyses can be misleading. The authors show that published evidence in win-win corporate social responsibility (CSR) research tends to overestimate efficiency. The research field expects to find a positive association between corporate social performance (CSP) and corporate financial performance (CFP), and findings meet that expectation. The authors explain how this pattern may reflect reporting bias. The empirical results show strong tentative evidence for a positive reporting bias in the CSP–CFP literature but only weak tentative evidence for CSP efficiency. The study also examines which factors, such as time trends, publication outlet, and study characteristics, are associated with higher reporting biases within this literature.
In physics, there was a long-held belief that light waves needed a “luminiferous ether” as a medium. In one of the greatest “failed experiments” of all times (1887), Albert Michelson and Edward Morley could not find this ether. It took Albert Einstein’s special theory of relativity (1905) to show that there was never a need for luminiferous ether. 1 Popper (1963) suggested that all scientific theories are by nature conjectures and inherently fallible. Should any theory survive a number of critical tests, it may be closer to some truth.
Win-win corporate social responsibility (CSR) arguably may be such a widely held belief. 2 “Empirical research has largely focused on establishing a positive connection between corporate social performance (CSP) and corporate financial performance (CFP)” (Margolis & Walsh, 2003, p. 273). The aim is to substantiate the kind of claims that UN Secretary-General Kofi Annan made to U.S. corporations in 2001, namely, that there is “a happy convergence between what your shareholders want and what is best for millions of people the world over” (Margolis & Walsh, 2003, p. 273). A large number of empirical studies do confirm that a positive connection exists between CSP and CFP. The results of meta-analyses confirm that the standardized CSP–CFP effect is positive and significant (Allouche & Laroche, 2005; Margolis, Elfenbein, & Walsh, 2007; Margolis & Walsh, 2001, 2003; Orlitzky, 2011; Orlitzky, Schmidt, & Rynes, 2003; Orlitzky & Swanson, 2008). The now 40 years of the search 3 for an association between CSP and CFP (see Wood, 2010) reflect the enduring quest to find a persuasive business case for social initiatives.
However, when it comes to belief in a strong, positive relationship between CSP and CFP, other scholars are more skeptical (McWilliams & Siegel, 2000, 2001). Some of these scholars are skeptical about positive empirical evidence, arguing that social responsibility that goes above just complying with the laws detracts from a firm’s financial performance (Friedman, 1970; Jensen, 2002). CSP raises a firm’s costs, thereby putting it at an economic disadvantage in a competitive market. Others take a more nuanced stand and argue that CSP may not only raise but also decrease a firm’s performance level (Benabou & Tirole, 2010), has diminishing returns for imitating firms (Brammer & Millington, 2008; Kopel, 2009), and can only provide insurance in the context of negative business events (Godfrey, Merrill, & Hansen, 2009). Finally, some scholars argue that the literature suffers from methodological problems and misspecifications (McWilliams & Siegel, 2000). These scholars also argue that, from a theoretical point of view, a neutral CSP–CFP link should be expected (McWilliams & Siegel, 2001).
This study posits that scientific knowledge is irreducibly conjectural. By relying on the conjectures and refutations of win-win CSR, the authors first argue that a genuine effect between CSP and CFP can only be expected under very restrictive conditions. Nevertheless, the CSP–CFP literature generally does find positive effects. Second, the authors explain how this pattern may reflect reporting bias in the CSP–CFP literature. Such selection bias in the literature might be attributed to authors not writing papers on unfavorable results, in spite of strong efforts being made to find accepted evidence. One can also attribute such selection bias to the higher rejection rate of studies documenting contradictory evidence. As demonstrated by recent research, the underlying reason may be that “social-scientific research is embedded in a variety of genealogies of inquiry that may shape and frame conclusions” (Orlitzky, 2011, p. 410). The underlying reason implies that “an institutional logic represents a much broader concept than researcher ‘bias,’ which presumes some undesirable deviations from a known, objective fact” (Orlitzky, 2011, p. 410). Thus, reporting or selection biases in the literature, as understood in all that follows, refer to unintended, collective consequences of individual actions which—from each researcher’s point of view—are desirable. 4 Third, the authors identify circumstances reinforcing reporting bias. We analyze whether, in CSP–CFP research, positive, favorable results are associated with factors such as publication time, publication in higher impact journals, or weak theory or method. The hypotheses are tested with a sample of 162 studies documenting 2,263 sub-effects on the CSP–CFP link. Sub-effects are estimations on the CSP–CFP link, which rely on the same study sample but rely on alternative measurements, time intervals, or regression methods used within a paper. The authors use meta-regression analysis—in particular, the funnel asymmetry test (FAT)—to quantify reporting biases within the literature.
Our research contributes to the literature by pointing out that empirical results—in particular, the results of meta-analyses—can be undermined by reporting biases (Dalton, Aguinis, Dalton, Bosco, & Pierce, 2012; Rosenberg, 2005; Scargle, 2000). Rosenthal (1979) is the seminal reference. Reporting biases are the result of scientific consensus and can be very powerful in leading a largely one-sided debate (Popper, 1934/1959). Scientists who question the paradigm may be shunned in a scientific community: Their articles may not be published, and their research grants may be revoked or denied. This article argues that reporting biases are an important but underinvestigated issue in the social sciences. Although the problems are well known in research, these claims have barely been systematically analyzed or explained in management research. Even though previous meta-analyses on the CSP–CFP link address what is termed the “file-drawer problem”—that is, the problem of missing studies—they do so only parenthetically. These analyses deal with this issue by computing the number of additional unlocated studies needed to cause the correlation to decrease to a minimal critical level or 0 (Margolis et al., 2007; Orlitzky, 2011; Orlitzky et al., 2003). Because large numbers of studies are arguably needed, it is reasoned that reporting bias is absent. Such a reporting-bias test remains rather silent on the size and significance of the reporting bias within the analyzed literature. Moreover, nothing is said about the size and significance of the remaining effect if reporting bias is absent. Furthermore, even though the quantity of empirical research is expanding rapidly, management research to date has shown little concern over the existence of reporting biases and the implication for the research field.
It should be noted that this article only refers to positive research paradigms in management in CSR: “A paradigm that tries to uncover correlations and causal relationships in the social world by using the empirical methods of (natural) science” (Scherer & Palazzo, 2007, p. 1096). This positive paradigm leads to a merely instrumental interpretation of CSR. This study excludes normative research paradigms in management or CSR: a paradigm based on the humanities “that does not look for observable causal relationships in the social world” (p. 1097) but rather is “centered on moral evaluation, judgment, and prescription of human action” (Swanson, 1999, p. 570, cited by Scherer & Palazzo, 2007, p. 1097).
The remainder of the article is as follows: In the next section “Hypotheses Development,” the authors will first bring forward two interrelated theoretical arguments against a general case for a clear and positive CSP–CFP link. While such a view implies that a clear, positive CSP–CFP link cannot be expected, it contradicts the view of empirical research in Social Issues in Management, Business Ethics, or Business & Society. The section second will argue that the published research within a research field can be confronted with a biased representation of the total population of research results, suggesting that the positive effect between CSP and CFP, as documented in most studies and meta-analyses, may be caused by selection bias in the literature. The third part of this section will analyze factors which may reinforce reporting bias. The section “Research Method” introduces the method, the sample, and the measurements to analyze the research question and hypotheses. In the “Results” section, the authors will introduce and summarize their findings. Our main result suggests that the positive CSP–CFP evidence is caused by reporting bias in the literature originating from collective cognitive structures within a research discipline. In the final section, the authors will discuss the implications with regard to future research and with regard to scholar education.
Hypothesis Development
The Absent CSP–CFP Link: Theoretical Arguments
Without attempting to be comprehensive, there are also two interrelated theoretical arguments against the general case for a clear and positive CSP–CFP link. This statement implies that the authors do not argue against a specific case for this positive link. Nevertheless, these theoretical arguments indicate that a clear, positive CSP–CFP link cannot be expected because there remain significant theoretical refutations.
First, in the economic research paradigm (see Orlitzky, 2011), firms envision very different roles for CSP. Benabou and Tirole (2010) distinguished three visions: win-win (doing well by doing good), delegated CSR (the firm as a channel for the expression of citizen values), and insider-initiated CSR. Although the first vision may show a positive correlation between CSP and profits, the reverse should hold true for the third vision. An application of the first vision could go hand-in-hand with product differentiation or innovation (e.g., ecological products), which matches a greater willingness of particular customer segments to pay (see also McWilliams & Siegel, 2001). Yet, the effects of such CSP investments, when they are not perfectly specific to the CSP leader firm (i.e., they spill over to follower firms investing later), can lead the follower firms to have a second-mover advantage (Kopel, 2009). There is then a CFP disadvantage for the CSP first mover.
The implementation of the second vision relies on a willingness by stakeholders to sacrifice money to bolster firm image, and so by definition can at best achieve a much more indirect effect than the first vision. However, even an assumed positive relationship between CSP and CFP for both visions could follow a non-linear pattern—that is, be subject to positive returns first, and then diminishing and eventually decreasing returns (Brammer & Millington, 2008). Following this line of argument further, one could also assume that CSP only has a positive insurance value in the context of negative business events. One might assume that firms very active in CSP would benefit from smaller declines in stock prices in the face of such events (Godfrey et al., 2009).
The third vision reflects a more managerial concept of CSP. Firms donating to institutions such as opera houses or museums can directly benefit senior management, as, for example, when chief executive officers (CEOs) and chief financial officers (CFOs) are offered board memberships there. CFP usually decreases with that sort of CSP.
Of course, in practice, CSP is likely to involve a mix of all three visions across the corporate sample. It is often unclear which specific channel is being tested. Let us assume that a partial value of CFP can be interpreted as a function of all the CSP investments made simultaneously by a firm because of the three different visions. We know that, by definition, these CSP investments follow different incentives, presumably with different results. However, when we do not know the weights of the CSP activities that follow the aforementioned visions or the functional forms of their particular CFP consequences (see Brammer & Millington, 2008, for very different forms), we cannot even determine ex-ante, whether CFP increases or decreases because of these CSP activities. For the ex-post analysis, it is almost impossible to solve the problem of reverse engineering and disentangling the CFP result to reveal its different CSP causes.
It becomes clear that, in the economic research paradigm, and only under certain circumstances, can a clear positive relationship between CSP and performance be expected (Vogel, 2005). McWilliams and Siegel (2001) using an arbitrage argument found that firms, whether they provided CSP or not, had the same rate of profit. From this finding, they predicted that there would be generally a neutral relationship between CSP activity and firm performance.
This inconclusiveness stems partly from one general problem that holds true for the underlying research field: using organizational performance as a dependent variable (March & Sutton, 1997). The primary focus of CSP–CFP studies is the idea that organizational performance can be predicted, understood, and shaped. Consequently, researchers take organizational performance as a dependent variable and seek to identify variables that produce variations in profits, sales, market share, productivity, debt ratios, and stock prices:
Researchers who study organizational performance in this way typically devote little attention to the complications of using such a formulation to characterize the causal structure of performance phenomena. These complications include the ways in which performance advantage is competitively unstable, the causal complexity surrounding performance, and the limitations of using data base on retrospective recall of informants. (March & Sutton, 1997, p. 698)
The findings of March and Sutton serve as an illustration for empirical and methodological problems when analyzing the CSP–CFP link. 5 Consequently, even if the firm’s CSP activities only follow the first vision, it is not certain that research will identify this positive CSP–CFP relationship. On an aggregated level, such noise results in a zero finding, as former empirical and theoretical research verifies (McWilliams & Siegel, 2000, 2001).
The Positive CSP–CFP Link: Reporting Bias
The considerations above contradict the view of empirical research in Social Issues in Management, Business Ethics, or Business & Society fields (Orlitzky, 2011). Building on instrumental stakeholder theory (Jones, 1995), it can be argued that socially responsible firms increase their legitimacy and develop a positive reputation (Orlitzky, 2011). This view seems to be supported by the findings of previous meta-analyses documenting an overall positive CSP–CFP relationship. A selection of these meta-analyses includes Allouche and Laroche (2005), Margolis et al. (2007), Margolis and Walsh (2001, 2003), and Orlitzky et al. (2003). However, meta-analyses can be confronted with a biased representation of the total population of research results (Rosenthal, 1979), or with organizational theories and research assumptions that become self-fulfilling prophecies (Orlitzky, 2011). Such reporting bias arises when the selection of studies for publication is made either on the basis of the statistical significance of results or on the basis of whether the results satisfy preconceived theoretical expectations (C. Doucouliagos, 2005; Stanley, 2005a). This condition leads to a truncated pool of published studies with the consequent suppression of some empirical findings. Various types of reporting biases exist, including publication bias, time-lag bias, location bias, citation bias, language bias, or outcome reporting bias (McGauran et al., 2010). Reporting bias is not a trivial issue. If it exists, it distorts the scientific record, hides the “truth,” influences decision making, misleads policy makers, and causes harm to affected persons (McWilliams & Siegel, 1997; Stanley & Doucouliagos, 2010).
Medical researchers have long acknowledged the importance and seriousness of publication selection (Begg & Berlin, 1988; Egger, Smith, Schneider, & Minder, 1997; Sterling, 1959). The Paxil and Vioxx scandals put reporting bias on the research agenda (Krakovsky, 2004). Paxil has the unfortunate side effect of increased teen suicide; Vioxx’s side effects include the increased risk of heart disease. The life-threatening side effects were well known from clinical trials, but sponsors of the clinical trials suppressed the reporting of these side effects. A review of five meta-analyses on clinical trials shows that 73% of all positive clinical trials are published, but only 41% of all negative clinical trials are published (Hopewell, Loudon, Clarke, Oxman, & Dickersin, 2009).
Reporting biases are a problem not only in medical research. A study analyzing more than 4,600 papers published in all disciplines shows that the overall frequency of positive support has grown by more than 22% between 1990 and 2007 (Fanelli, 2012). The increase was stronger in the social disciplines (Fanelli, 2010b) and in publish-or-perish environments such as the United States (Fanelli, 2010a). There is also a growing interest in reporting bias in the field of economics. Researchers found evidence for reporting biases in areas such as aid effectiveness on growth (H. Doucouliagos & Paldam, 2008), minimum-wage studies (Card & Krueger, 1995; H. Doucouliagos & Stanley, 2009), economic freedom and economic growth literature (C. Doucouliagos, 2005; H. Doucouliagos, Laroche, & Stanley, 2005), the estimates of returns from education (Ashenfelter, Harmon, & Oosterbeek, 1999), the productivity effects of multinationals (Görg & Strobl, 2001), and the price elasticity of water (Stanley, 2005a). Reporting bias is a widespread phenomenon that might be expected to occur in the business literature as well (McWilliams & Siegel, 1997).
H. Doucouliagos et al. (2005) hypothesized that reporting bias can be found in particular areas of research where mainstream theory supports a specific effect. Instrumental stakeholder theory in CSP–CFP research, the mainstream theory for researchers in Social Issues in Management (Orlitzky, 2011), promotes a social justice view of the world that strives to align with shareholder wealth maximization. Thus, CSP is viewed as a source of competitive advantage resulting in positive performance effects. This positive view stands in contrast to the economic view and general management view on the empirical CSP–CFP relationship: Both research paradigms expect a neutral relationship (Orlitzky, 2011). Hence, it is likely that reporting bias may affect the CSP–CFP literature: While two research paradigms expect no specific CSP–CFP relationship, one research paradigm expects a positive CSP–CFP relationship. The lack of a research paradigm which expects a clear negative CSP–CFP relationship can lead to an overdominance of positive research findings.
There are two particular sources that cause reporting bias (Card & Krueger, 1995; Hopewell et al., 2009). First, the dominant institutional logic in which authors are embedded shapes their cognitions and behaviors and frames their conclusions (Orlitzky, 2011). This logic may result in the fact that researchers, and in particular those closely connected to instrumental stakeholder theory, do not write and submit papers on negative results contradicting the conventionally expected result. This condition would be consistent with the results from other research disciplines: Author surveys have shown that the most common reasons for not submitting papers are negative results and a lack of interest, time, or other resources (Callaham, Wears, Weber, Barton, & Young, 1998; Weber, Callaham, Wears, Barton, & Young, 1998). Second, reviewers and editors, and in particular those connected to instrumental stakeholder theory, may be predisposed to accept papers consistent with a conventional view: “Social networking effects and interpersonal connections sometimes explain publication decisions much better than the objective characteristics of submissions” (Orlitzky, 2011, p. 412). As shown in prior empirical research, the main reason for the non-publication of unconventional results seems to be the non-submission of manuscripts and not the rejection of manuscripts by journals (Olson et al., 2002).
In the context of CSP–CFP, reporting bias can take several forms. First, authors, referees, and editors may disproportionately select significant positive results believing that they are more informative. This selection bias could partly be a consequence of collective cognitive structures, for example, of the professional business school environment where many researchers are employed (March & Sutton, 1997). The customers of these schools—such as aspiring managers—expect to be trained to create success stories in organizations. In addition, researchers secure funding and establish legitimacy as consultants to organizations, as lecturers to organizational audiences, or as authors of books providing suggestions for improving organizational performance. These audiences encourage researchers to create success stories (Clarkson, Li, Richardson, & Vasvari, 2008). Positive findings reflect the majority view in the field and have a higher probability of being read and cited. In addition, in the CSP–CFP literature, there is an overwhelming professional consensus of a social justice view of the world. 6 Negative findings contradict this view, and concerns may be raised about the possibility of misspecification, data problems, or estimation errors.
Second, reporting bias can result in authors finding it difficult to publish results where the CSP–CFP link is positive but not significant, even though some research paradigms expect such a non-specific relationship. In general, null findings are perceived as less interesting and most scholars will not even write a working paper on such results. Furthermore, referees may argue that CSP should show a statistically significant impact on CSP. Authors may be encouraged to locate this effect by, for example, changing the sample size, changing the empirical methodology, altering the set of control variables, or using a different estimation technique. This condition may, for example, explain why so many event studies of CSR have quite long and seemingly randomly chosen “window” lengths (McWilliams & Siegel, 1997). 7
Third, researchers who find a negative effect on CFP may choose not to submit these results to journals or other publication outlets because they believe the results to be incorrect, they expect that they will have a low probability of being accepted, or their preferred literature does not offer accepted theories to explain negative findings.
Summing up, the foregoing arguments suggest that the conventional view within CSP–CFP studies, that CSR is likely to pay off, may be the result of an overrepresentation of positive, significant findings, whereas insignificant or negative empirical findings remain suppressed. This analysis leads to the first hypothesis, suggesting that the positive effect between CSP and CFP, as documented in most studies and meta-analyses, may be caused by selection bias in the literature or by collective cognitive structures within this research discipline.
Characteristics of Studies: Moderators of Reporting Biases
This section analyzes factors reinforcing reporting bias. As shown by Orlitzky (2011), the journals in sub-disciplines explain differences in the empirical CSP–CFP cross-study variability. Furthermore, medical research demonstrates that studies with positive or favorable results are associated with various other factors, such as publication in journals with higher impact factors, a greater number of publications (including covert duplicate publications), more frequent citation, and more likely publication in English (Easterbrook, Berlin, Gopalan, & Matthews, 1991; Egger, Zellweger-Zahner, et al., 1997; Gotzsche, 1987; Kjaergard & Gluud, 2002; Tramer, Reynolds, Moore, & McQuay, 1997). Also, time to publication is 2 years less for positive results than for negative results (Ioannidis, 1998; Stern & Simes, 1997). This study will, therefore, test whether positive or favorable results in CSP–CFP research are associated with factors such as publication time, publication in journals with higher impact factors, or weak theory or method. The authors will develop conditional expectation for the case that Hypothesis 1 can be confirmed by the results. Therefore, further hypotheses will be presented as H0 hypotheses, typically corresponding to a general or default position.
The aforementioned arguments concerning research incentives lead us to expect a certain life cycle of publication bias in the empirical literature on CSR–CSP. The authors assume that the first publications, defined as studies that were published and thus made publically available in the earlier years of CSR–CSP research, will start with a bias, which could show a positive (or even a negative) CSR–CFP link. In each following period, new publication opportunities in the field arise that demonstrate a tendency toward the starting bias. This tendency can be viewed as informational cascades in the research field that lead “follower” researchers to imitate “first movers” in their research design. The cascades will happen when authors find it optimal to follow the pattern established by prior published work, even if that pattern of results contradicts their own research results. All innovative ideas toward the bias can be imitated at low cost by all researchers after publication. Both sides of the “market for articles” (authors and editors/reviewers) will, depending on their field, start to invest specifically in the reproduction of the biased view. While the research field moves toward a dominant design, both sides of the market will find it more and more costly to produce and evaluate unconventional views, thereby devaluing their previous investments in the conventional view.
Inserting the incentives available for producing strong results into the logic of a product life cycle, we should expect the following development over time. In the first phase of empirical papers on the CSR–CFP relationship, innovative CSR researchers should write papers with strong positive correlations; but before the dominant design has been established, we will find a small publication bias. With this research field moving toward a dominant design, authors will find it more and more costly to produce unconventional views, both because the conventional view gets more and more empirical support, and because its theoretical underpinnings come to be more elaborated upon. Publication bias will also result over time with authors finding it difficult to publish results where the CSP–CFP link is positive but not significant. Referees will argue that CSP should show a positive and statistically significant impact on CFP. Authors may be encouraged to locate this effect.
The cost argument also holds for editors and reviewers who, over time, accumulate specific investments in the conventional view, which would be devalued by opposing views. Papers that are unable to produce this conventional view will be excluded from the market. In addition, researchers may use the presence of an expected result as a model-selection test (Card & Krueger, 1995). Both incentives can lead to higher rejection rates by journals or to lower submission rates by authors. They can also lead to more data adjustments by authors in the latter stage of the CSP–CFP innovation in an attempt to find evidence for the conventional view. Therefore, in general, we expect to have a strengthening publication bias over time. The associated H0 hypothesis that will be tested is as follows:
As shown in medical research, studies with positive or favorable results are more often published in journals with higher impact factors (Easterbrook et al., 1991). Positive and favorable results are read and cited to a greater extent, which is important to maintain and extend the competitive advantage with respect to the journal impact factor. Authors, editors, and reviewers may be aware of this fact, and significant positive evidence will be both submitted and accepted to a greater extent in high-impact journals.
Furthermore, in the case of editorial decision-making biases, one should expect that the results of accepted journal articles contain larger reporting bias compared with the findings of published working papers. Some working papers may be continuously rejected by journals because the findings are either negative or non-significant. 8 This outcome could be the consequence of cost considerations of reviewers. As the research field moves toward a dominant design, it becomes more demanding and time-consuming for them to evaluate positively unconventional views, thereby devaluing their former investments in the conventional view.
Large numbers of less ground-breaking research contributions also characterize CSP–CFP research. Often, researchers follow the conventional view because CSP–CFP research is perceived as “marketable.” In this case, the underlying research motivation may not be the investigation of a specific theory, but more the desire to participate in the new research trend. Typically, the research field in these papers is poorly understood. Authors may, therefore, decide to publish their results only if they support the conventional view. The demand for explanations that contradictory or non-significant empirical evidence poses for authors is quite high, especially given the widely established CSR–CFP link. Furthermore, faced with a theoretically weak paper, reviewers and editors may only accept research modeled on the conventional view in the field. The authors, therefore, expect that the reporting bias is highest for studies investigating no specific theory.
Most studies in CSP–CFP research deduce directed hypotheses for the CSP–CFP link. However, directed hypotheses are vulnerable because arguments both for and against the CSP–CFP link can be made. It may, therefore, be advisable to formulate a H0 hypothesis, to test whether no link between CSP and CFP can be determined. Testing the H0 hypothesis allows more freedom when reporting the actual outcome of the study, whereas testing a directed hypothesis puts some pressure on the researcher not to reject the hypothesis. We, therefore, expect higher reporting biases for studies testing a directed hypothesis. Authors may only write papers on supportive evidence. Furthermore, they may also be engaged more often in some type of data adjustment to find at least some evidence for their hypothesis.
Some researchers within the field support a clear positive CSP–CFP link by outlining the competitive advantages for organizations. Other researchers are more critical by outlining the strengths and weaknesses of CSP with respect to CFP. Reporting bias should more often occur when the discourse in professional networks does not lend itself to conjecture and refutation but, instead, uncritically supports the conventional view. For example, authors more inclined to be uncritical may not write papers on negative or non-significant findings or may adjust their data to find positive and significant evidence.
CSP–CFP research is characterized by methodological problems such as performance instability, retrospective recall, or simple models of complex worlds (March & Sutton, 1997; Nicolai & Kieser, 2002). In particular, studies that face many of these problems will be modeled after the conventional view. The demand for explanations that contradictory empirical evidence poses for both sides of the market for publications is quite high, especially given the widely established CSR–CFP link. By definition, the presentation of contradictory evidence requires a deep theoretical and methodological understanding of the investigated research field. Therefore, authors write papers on methodologically weak studies only if the findings clearly confirm the conventional view within the field, thereby increasing the credibility of their research design. Furthermore, as explained above, editors and reviewers may accept these studies only if the results confirm the conventional views, thereby sustaining their past investments (see McWilliams, Siegel, & Teoh, 1999, for a discussion of the methodological problems with event studies of CSP).
Reporting bias results from authors finding it difficult to publish results where the CSP–CFP link is positive but not significant. Authors may be encouraged to locate this effect by, for example, changing the sample size, changing the empirical methodology, altering the set of control variables, or using a different estimation technique. The incentive to apply “permitted” manipulations is particularly strong for authors using multivariate empirical methods, such as regression analysis. They can alter the set of control variables or use a different estimation technique to obtain the desired result. In the case of univariate empirical methods, such as mean comparison or bivariate correlations, such gray areas of data adjustments exist to only a lesser extent. Therefore, the authors expect a stronger reporting bias for multivariate empirical methods.
Research Method
Sample
To investigate reporting bias in the CSP–CFP literature, the authors rely on a meta-analysis (Hunter & Schmidt, 2004). Meta-analysis has the advantage of being able to quantify the diverse results of prior research. To collect the study sample, the authors consulted prior meta-analyses or literature reviews on the CSP–CFP relationship. 9 We further conducted our own research using the Web of Science, Google, and other bibliographical databases by using several keywords (such as CSR and financial performance) and the forward and backward citations of collected studies. To ensure convergence in the study sample, two independent persons were involved. The search steps of the two persons together provided a list of 282 studies. Many studies were identified in both search steps and by both persons; however, especially newer studies were only identified by our own research and not by former meta-analyses. Furthermore, the keyword search was expanded and altered up to a point where the inclusion of new keywords did not lead to the inclusion of new studies. The authors subsequently excluded 120 studies because they did not document statistical results (being literature reviews or qualitative studies). The final sample includes 162 empirical studies on the CSP–CFP link (see Online Appendix C), and is, therefore, larger than prior samples.
To qualify for inclusion in the sample, a study had to document statistical results on the CSP–CFP link in the following form: (a) correlation or standardized coefficient and sample size; (b) unstandardized coefficient, standard deviation, and sample size; (c) t value and sample size; (d) mean, standard deviation, and group sample size for sub-samples; and (e) the difference in mean and t value for sub-samples. From each study, the authors included all documented effects on the CSP–CFP link. However, for hierarchical regression analyses that documented several estimation steps using the same measures and methods, we only included the main effect estimations of the CSP–CFP link and not the results of models that included additional moderating effects between CSP and other measurements. The rationale is that the main effects in interaction-effect models are meaningless if moderating effects are not, additionally, considered. If the authors changed methods (ordinary least squares regression vs. a fixed-effect regression) or used CSP/CFP measurements, more than one effect was included. Furthermore, for event studies, we excluded (cumulative) abnormal returns in the period day −∞ up to day −2, but included estimates that either started on day −1 or ended on day ≥1. The process identified 2,263 sub-effects 10 documented in the 162 studies. Half of the study sample was coded independently by two persons and subsequently compared to ensure reliability. The consistency between both coders was rather high. In the case of ambiguous coding, we discussed the reasons and agreed on rules for similar cases. The rest of the sample coding has been finished by only one coder.
Measurements
Size and direction of the CSP–CFP link
The authors’ dependent variable is the size and direction of the CSP–CFP link as measured in a study. We standardized the diverse effect sizes of all studies by computing Fisher’s z, the associated standard error, and z value. The Fisher transformation z is mainly associated with the Pearson sample correlation coefficient r. For correlations r smaller than |.5|, z is identical. Compared with Fisher’s z, the variance of r grows smaller as the population correlation coefficient |ρ| approaches 1. We apply the variance-stabilizing transformation of Fisher’s z instead of Pearson’s r to simplify considerations in graphical exploratory data analysis, namely in funnel plots, and to allow the application of simple regression-based techniques. We did, however, run robustness checks by using Pearson’s r. The results are comparable as most correlations are lower than |.5|. Standardized effects were computed by using the program Comprehensive Meta-Analysis that allows for different data entries (Borenstein, 2000). As a dependent variable, we use the z values of the effects (for more details, see the section on “Reporting Bias Analysis”). z values are calculated on the sub-effect level and on the study-effect level. A study-effect approach combines the sub-effects of each study within one effect. Meta-analysis recommends the use of study effects instead of sub-effects to reduce the non-independence error (Hunter & Schmidt, 2004). For robustness analysis, the authors calculated two dummy variables measuring whether sub-effects or study effects are positive and statistically significant. As a critical value, we chose z ≥ 1.96, indicating that an effect is positive and significant at the 5% level.
Publication time and outlet
At the study level, the authors measured publication year, journal impact factor, and working papers. About 50% of all studies were published after 1995. We constructed a dummy variable indicating whether the study was published before or after 1995. The authors did not include the metric information for publication year because we, additionally, controlled for the time period analyzed through a study. 11 Including the metric information of publication year causes multicollinearity problems. 12 We collected data on the impact factor of the journals in which a study was published. The journal impact factor indicates how many times an average article within a journal is cited by other journal articles within the first 2 years after publication. For all journals included, the impact factor was available either from Thomson Reuters or from RePEc: Research Papers in Economics (both use the same formula). For working papers, the authors assigned an impact factor of 0. We also controlled for whether a paper had already been accepted by a journal or published as a working paper.
Theoretical aspects
At the study level, the authors collected data on the underlying theory, on the way of discussing the CSP–CFP link, and the formulated CSP–CFP hypothesis. The underlying theory of a study was first coded into sub-categories that we later summarized in three general groups. The first general group covers social science theories, including (a) stakeholder theory, (b) resource-based view, (c) principal-agent theory, (d) legitimacy, (e) leadership, (f) organizational slack, (g) signaling theory, (h) studies contrasting the shareholder view with the stakeholder view, and (i) studies contrasting resource-based view with principal-agent theory. The second general group covers economic or financial theories, including (a) ethical investments and investor behavior, (b) efficient market view, (c) shareholder view, and (d) financial accounting. Finally, the third group covers studies that made no use of any particular theory. The authors further coded whether a study discussed the arguments for and against a positive CSP–CFP link. We constructed a dummy variable that indicates whether a paper contained a balanced discussion of the pros and cons or contained only a pro or no discussion. The underlying CSP–CFP hypothesis of a study was coded as directed if the study assumed a positive, negative, u-shaped, or inversely u-shaped CSP–CFP link and as H0 if the study explicitly started from the assumption that no CSP–CFP link exists.
Methodological aspects
On the sub-effect level, we documented whether authors made use of industry-fixed effects (or controlled for industry effects), firm-fixed effects, and time-lagged effects. Studies that control for industry-fixed effects, firm-fixed effects, or time-lagged effects have less methodological problems because they exclude some confounding factors of the CSP–CFP link. The research also measured which kind of analysis was used to compute sub-effects: multivariate methods, such as regression analysis, or univariate methods, such as bivariate correlation analysis, or the t test, or mean comparison.
Control variables
In line with prior meta-analyses (Margolis et al., 2007; Orlitzky et al., 2003) on the sub-effect level, the authors coded which indices were used to measure CSP and CFP. We first documented all measurements in detail and later developed useful categories. According to our analysis, studies measure CSP in six different ways: (a) subjective CSP rankings (KLD or Fortune 500 survey); (b) comparison of socially responsible investing (SRI) with non-SRI portfolios, funds, or indices (Dow Jones Sustainability Group Global Index vs. conventional index); (c) announcement effects of crime, product recalls, incidents, or withdrawal (announcement of unethical events; the effects were recoded, so that positive values indicated no violation); (d) cash giving and contributions (sum of foundation disbursements and direct cash contributions); (e) social disclosure (environmental disclosure); and (f) effects of CSP regulations and principles (Resource Conservation and Recovery Act). CFP was measured in three different ways: (a) accounting performance (Return On Assets (ROA), Return On Equity (ROE), sales growth), (b) market performance (Jensen’s alpha, excess returns), and (c) event-based performance ([cumulative] abnormal returns). At the sub-effect level, we further coded the time period analyzed by a study by five time intervals (1960-1969, 1970-1979, 1980-1989, 1990-1999, 2000-2009). The authors measure whether a sub-effect covers years within a time interval or not. We further included the number of years investigated by a sub-effect.
Table A1 in Appendix A reports the descriptive statistics and bivariate correlations for the variables in our sample.
Reporting Bias Analysis
In contrast to medical research, where studies must be registered from the beginning, a reporting bias analysis in social science research is more indirect. In social sciences, we do not know how many studies have been conducted, but subsequently not published. To overcome this problem, two simple tests, the funnel-plot test and the FAT test, have been performed (Moreno, Sutton, Ades, et al., 2009). The underlying assumption of both tests has been validated in medical research (Stanley & Doucouliagos, 2011). Validations compare the results of the indirect tests (working without the knowledge of the number or type of studies not published) with the results of direct tests (providing information on the findings of all registered studies and the findings of the sub-population of published studies). These real-life studies show that the findings of the indirect tests are reliable and precise (Moreno, Sutton, Ades, et al., 2009; Turner, Matthews, Linardatos, Tell, & Rosenthal, 2008).
Counter-enhanced funnel plots
The first graphical test performed for detecting reporting biases is funnel-plot test (C. Doucouliagos, 2005; Stanley, 2005a). The procedure plots a study’s effective size (x axis) against its accuracy, that is, its estimated standard errors (y axis; Egger, Smith, & Phillips, 1997). To measure a study’s effective size, we will use Fisher’s z for a study’s effective size (x axis) and its associated standard error for a study’s accuracy (y axis; for more details, see section “Size and direction of the CSP–CFP link”). The plot shows symmetrical, funnel-like patterns when there is no reporting bias in the literature (Borenstein, 2000). Studies with a lower sampling error (typically large studies) appear toward the top of the graph and cluster near the mean-effect size. Studies with a large sampling error (typically smaller studies) appear toward the bottom of the graph and are dispersed across a range of values around the mean-effect size because they contain more random variation. In the absence of reporting bias, the studies are distributed symmetrically around the mean-effect size (Borenstein, 2000). In the presence of bias, the bottom of the plot tends to show a higher concentration of studies on one side of the mean as opposed to the other, for example, because less precise studies are more likely to be published if they demonstrate larger-than-average effects (making them more likely to meet the criterion for statistical significance). Such errors may be caused by the high rejection rate of small studies documenting contradictory evidence or by the strong efforts to find accepted evidence (Card & Krueger, 1995).
However, asymmetry in the plot does not necessarily imply that reporting bias exists, as alternative explanations for the asymmetry may be present. For example, confounding factors, small-study effects, true heterogeneity, structural breaks, non-linearity, non-normal residuals or, in the case of few studies, chance may also distort the appearance of the plot (Callot & Paldam, 2011; Sterne et al., 2011). To disentangle genuine reporting biases from other causes of funnel asymmetry, the literature recommends contour-enhanced funnel plots (Moreno, Sutton, Turner, et al., 2009). Based on the standard Wald test, these plots include contours that partition them into areas of statistical significance and non-significance to identify the level of statistical significance of study effects. Reporting biases are related to statistical significance: “If studies seem to be missing in areas of statistical non-significance, then this adds credence to the notion that the asymmetry is due to publication biases. In such cases an attempt should be made to adjust for such biases” (Moreno, Sutton, Turner, et al., 2009, pp. 2-3).
FAT test
To explore reporting bias more rigorously, the literature recommends using meta-regression analysis (H. Doucouliagos & Stanley, 2009; Stanley, 2005b; Stanley & Jarrell, 1998, 2005). A meta-regression analysis checks for serious reporting bias and detects the size of the true effect between two variables by considering the influence of other drivers of effect size, such as different measurements, methods, time periods, and so on. Regression-based alternatives offer several advantages as compared with funnel plots. First, in contrast to funnel plots, regression-based methods not only detect publication bias but also adjust for it. Second, simulation studies (Moreno, Sutton, Ades, et al., 2009), as well as real-life studies (Moreno, Sutton, Turner, et al., 2009), show that regression-based alternatives, in contrast to funnel plots, have a lower tendency for misleading adjustments and poor coverage probabilities, especially when between-study variance is present (Moreno, Sutton, Turner, et al., 2009). Third, regression-based methods offer the opportunity to control for omitted variables, which may be the cause for funnel asymmetry (Callot & Paldam, 2011). We will use the FAT to check for serious reporting bias and to detect the size of the true effect between CSP and CFP (Stanley, 2005a). 13 FAT adjusts the effect between two variables by regressing the inverse of standard error (1 / SEi) on the value of the z statistic (zi; see Equation 1). The slope β1 in this equation offers information on the existence and size of the “real” effect between two variables, whereas the constant β0 indicates the presence or absence of reporting bias (C. Doucouliagos, 2005).
The logic behind this equation builds on the same idea as the funnel plot. If reporting bias is absent, then the constant β0 in the equation equals 0 and is not significant. First, studies with high standard errors (smaller studies) are counted as close to 0, as one expects to see such “noisy” studies associated with a mean standardized effect of 0. Some studies will find small positive or negative effects, although others will find huge positive or negative effects. Second, studies with small standard errors (large studies), therefore, get a higher weight as one expects precise studies to be less noisy and to converge at the “real” standardized effect. Because, on average, less precise studies detect a standardized effect of 0, and precise studies converge at the real standardized effect, this pattern creates a regression line whose intercept approaches the origin, that is, 0. The slope of this regression line shows the size of the real effect by detecting how effect size evolves if the precision of a study increases.
However, if there is a reporting bias, then the constant β0 in the equation does not equal 0 and is significant. This measure indicates that there is a selection bias in the literature because studies with high standard errors report large positive or large negative standardized effects disproportionately often. This effect creates a regression line with an intercept different from 0: a regression line that already starts with a large positive standardized effect even though the studies have a precision close to 0. Again, the slope of this line indicates the real effect between two variables without selection bias.
Even though in the social sciences, the FAT test is the best tool in meta-analysis to detect reporting biases and to adjust for them, it should be noted that there are still a range of problems that may generate natural asymmetries in funnel plots, such as data dependencies, estimation faults, or omitted variables, which are unknown to the researcher (Callot & Paldam, 2011). While the present study controls for several of these reasons, by replicating the results for an independent-study sample, by including different estimation methods, or by including omitted variables, natural asymmetry cannot be ruled out completely.
Analytical Method
Hypothesis 1 will be tested by the funnel-plot test and the FAT test. If we find indications of a significant reporting bias, we will investigate which study characteristics reinforce or reduce this bias. This possibility can be tested by analyzing whether certain study characteristics significantly affect the size and direction of the z value as documented in a study, and whether this effect, in turn, significantly affects whether a study documents a significant, positive effect. All tests and models will be analyzed for the sub-effects within a study, as well as for overall study effects. Study-effect models are clustered at the study level to get unbiased t values. Sub-effect models are weighted by the precision of a study. Meta-analyses either use a fixed-effect model or a random-effect model to assign weights to studies (Borenstein, Hedges, Higgins, & Rothstein, 2010). These two model types lead to different mechanisms for assigning weights that, in turn, causes different assumptions about the nature of the studies. Under the fixed-effect model, the assumption is that there is one true effect size shared by all included studies. In contrast, under the random-effect model, the studies are assumed to be a random sample of the relevant distribution of effects; here, the true effect is permitted to vary from study to study. 14 Both models will be applied. On one hand, for our sample, the random-effect model seems more realistic; for example, crime may cause different effects on performance than a top position in rankings. On the other hand, if funnel plots are asymmetrical, random-effect estimates will be pulled more toward findings from smaller studies than the fixed-effect estimates will be. Random-effect models are thus not always conservative (Sterne et al., 2011). For both models, weights were computed at the study-effect level, meaning that all sub-effects of one study sample enter the regression with the same weight.
Robustness Test
Appendix B documents the results of two prior meta-analyses—namely, Orlitzky et al. (2003) and Margolis et al. (2007)—on the CSP–CFP link. Both meta-analyses report significant positive effects on the CSP–CFP link but control for reporting biases only parenthetically. They deal with the problem by computing the number of additional unlocated studies needed to cause the correlation to decrease to a minimal critical level or 0. As mentioned in the introduction, such a test is incomplete and does not adequately include recent knowledge about reporting biases. To test the robustness of our results, we, therefore, repeat the funnel-plot test and FAT test for both samples. Orlitzky et al.’s sample covers data on 52 studies that report 388 effects between CSP and CFP. In Appendix B, the authors document Orlitzky et al.’s average effect per study, allowing us to collect a one-study one-effect sample. Our coding led to a sample consisting of 51 studies. 15 Margolis et al.’s sample covers data on 148 studies that report 192 sub-effects between CSP and CFP. Also in Appendix B, we document Margolis et al.’s sub-effects, which allowed us to collect sub-effects per study as well. 16 The advantage of sub-effect samples first lies in the fact that one can work with the original data as documented in the studies and does not have to rely on combined measurements which are corrected by study reliability or sample size. Second, sub-effect samples capture more variance with respect to different measurements or model specifications. In both samples, we control for study characteristics (as documented by both Orlitzky et al., 2003, and Margolis et al., 2007), which are different CSP and CFP measures, sample sizes, publication years, control variables, and the use of time-lagged effects.
Appendix B further documents a funnel-plot analysis for a non-independent-study sample of our meta-analysis. We combined into one source the studies which rely on similar samples, such as the rankings of Fortune Magazine, KLD, and The Wall Street Journal. These samples may not be strictly independent of each other, because they share similar sources of measurement errors.
Results
Evidence for Reporting Biases
Funnel plot
Figures 1a and 1b show the counter-enhanced funnel plots for our sample by relying on study effects and on all sub-effects published within the studies. In the figures, the standardized effect size (Fisher’s z) is measured on the horizontal axis and its accuracy (the standard error of Fisher’s z) is measured on the vertical axis. The figures show that the funnel plots are not symmetrical, which may indicate reporting bias in the CSP–CFP literature. Studies tend to overestimate the effects of the CSP–CFP link because the plots show a higher concentration of studies on the right side of the mean. Furthermore, for positive results on the CSP–CFP link, at the right side of the plot, there is a high concentration of studies which just meet the criteria of statistical significance or report non-significant results. Non-significant results for studies which just meet the criteria of statistical significance are, however, largely missing for negative results on the CSP–CFP link, at the left side of the plot.
Contour-enhanced funnel plot of the meta-sample.
The authors run a simple Egger test—a FAT test without control variables—as a first statistical test to test for the presence of serious reporting bias (Egger, Smith, & Phillips, 1997). For the sub-effect level as well as the study-level sample, the intercept of the Egger test is significant, indicating reporting biases which lead to an overestimation of the true CSP–CFP effect (see the notes of Figure 1). For the sub-effect-level sample, the observed CSP–CFP link is 0.075, but when adjusted for reporting bias, the effect decreases to 0.037. For the study-level sample, the observed CSP–CFP link is 0.090, whereas the adjusted effect decreases to 0.033.
Figures B1 and B2 in Appendix B document the counter-enhanced funnel-plot results of Orlitzky et al.’s (2003) and Margolis et al.’s (2007) meta-analyses. Because the findings are similar, there seems to be evidence for a strong and significant reporting bias in the CSP–CFP literature. Again, the figures indicate that the funnel plots are not symmetrical. The plots show a higher concentration of studies on the right side of the mean. Furthermore, for positive results on the CSP–CFP link, there is a high concentration of studies which just meet the criteria of statistical significance or report non-significant results. In Orlitzky et al.’s sample, the observed effect between CSP and CFP amounts to 0.235, whereas the adjusted effect decreases to 0.124 (see notes of Figure B1). In Margolis et al.’s sample, the observed effect between CSP and CFP amounts to 0.133, whereas the adjusted effect decreases to 0.051 for the sub-effect sample and to 0.037 for the study sample (see notes of Figure B2).
Figure B3 in Appendix B also documents the funnel-plot results and the Egger test for the non-independent-effect sample of our study. The results are comparable: The reporting bias is positive and significant. The observed effect between CSP and CFP amounts to 0.020, whereas the adjusted effect decreases to 0.008 (see notes of Figure B3).
FAT test
Tables 1 and 2 document the results of the FAT test using meta-regression analysis. The dependent variable is the z value of the sub-effects, that is, the study effects (see section on “Size and direction of the CSP–CFP link”). Column 1 reports the result for the random-effect sub-effect model, that is, the true effect is permitted to vary from study to study (see section on “Analytical Method”). Column 2 reports the result for the fixed-effect sub-effect model: The assumption is that there is one true effect size shared by all included studies (see section on “Analytical Method”). Column 3 makes no use of study weights but instead clusters the sub-effects at the study level to get unbiased t values (see section on “Analytical Method”). Table 1 shows the results without including study characteristics, whereas Table 2 shows the results by including study characteristics.
FAT Test: Determinants of the Size and Direction of Estimated CSP–CFP Effects (Excluding Study Characteristics).
Note. FAT = funnel asymmetry test; CSP = corporate social performance; CFP = corporate financial performance.
Unstandardized regression coefficient.
p < .05. **p < .01. ***p < .001.
FAT Test: Determinants of the Size and Direction of Estimated CSP–CFP Effects (Including Study Characteristics).
Note. FAT = funnel asymmetry test; CSP = corporate social performance; CFP = corporate financial performance; SRI = socially responsible investing.
Unstandardized regression coefficient.
p < .05. **p < .01. ***p < .001.
Independent of the estimation method, the results in Tables 1 and 2 first confirm a positive and highly significant reporting bias, indicating that the published literature tends to overestimate the CSP–CFP link. According to the results, this overestimation is far from negligible: Studies tend to over report the z value by an amount varying from 0.553 up to 3.802. The FAT test was repeated for Orlitzky et al.’s (2003) and Margolis et al.’s (2007) meta-analyses (see Appendix B, Tables B1 and B2). The analyses show comparable findings. In both samples, the reporting bias is positive and highly significant. According to the results, this overestimation is far from negligible: Studies tend to over report the z value by an amount varying from 1.165 up to 4.856. The results hold for both the simple regression model and the full model that controls for study diversity.
The results in Tables 1 and 2 also show that, after correcting for reporting bias, the CSP–CFP effect is mostly absent; in accordance with our theoretical explanation, the effect seems to be neutral. Only in two of six models is the corrected effect significant. However, the size of this effect is negligible; its standardized size ranges from −0.006 up to 0.029. The analyses were repeated for Orlitzky et al.’s (2003) and Margolis et al.’s (2007) meta-analyses (see Appendix B, Tables B1 and B2). The analyses show comparable findings. The unbiased CSP–CFP effect is only significant in three of nine models. Again, the size of the unbiased CSP–CFP effect seems negligible; its standardized size ranges from −0.003 up to 0.132. The results hold for both the simple regression model and the full model that controls for study diversity.
The results of the funnel-plot test and the FAT test provide prima facie evidence in support of Hypothesis 1, showing that the significant and positive performance evidence for CSP, as documented in the empirical literature, may be caused by reporting bias and not because CSP has a strong business case as is often anticipated in many CSP–CFP studies.
Moderators of Reporting Biases
This section analyzes those factors reinforcing reporting bias, looking at the circumstances under which the published literature tends to overestimate the CSP–CFP link. Table 3 documents descriptive results. Table 2 documents the meta-regression results with the z value as the dependent variable. Table 4 documents the meta-regression results with significant positive results as the dependent variable.
Descriptive Statistics of the Sample.
Note. CSP = corporate social performance; CFP = corporate financial performance.
Robustness Test: Determinants of Significant Positive CSP–CFP Findings.
Note. CSP = corporate social performance; CFP = corporate financial performance; SRI = socially responsible investing.
Unstandardized regression coefficient.
p < .05. **p < .01. ***p < .001.
Publication time and outlet
Hypothesis H0-2 assumes that the supportive CSP–CFP evidence is not biased toward an increase in publication time. The descriptive findings in Table 3 show that 59% of studies published after 1995 report significantly positive findings (on the sub-effect level: 22%). In studies published before 1996, the percentage of studies reporting significantly positive findings was 43% (on the sub-effect level: 16%). On the sub-effect level for later studies, higher z values can also be found. The regression findings in Table 2 support the finding that the z values are significantly higher and more positive for publications after 1995 compared with publications before 1995. The findings in Table 4 confirm that publications after 1995 report significant positive effects on the CSP–CFP link significantly more often than publications before 1995 do. Thus, the null hypothesis H0-2 is rejected, assuming that the supportive CSP–CFP evidence is not biased toward an increase in publication time.
Hypothesis H0-3 suggests that the supportive CSP–CFP evidence is not biased toward an increase of the journal impact factor. According to Table 3, 72% of the studies published in high-impact journals report significant positive findings (on the sub-effect level: 27%), whereas studies published in low-impact journals report 30% significant positive findings (on the sub-effect level: 11%). In addition, the z values reported in high-impact journals are higher than those reported in low-impact journals: 6.353 (on the sub-effect level: 1.171) compared with 1.242 (on the sub-effect level: 0.297). The regression findings in Table 2 support the random- and the fixed-effects models, and they suggest that the z values are significantly higher and more positive for high-impact journal publications than for low-impact journal publications. The findings in Table 4 confirm this result by showing that high-impact journal publications report significant positive effects on the CSP–CFP link significantly more often than do low-impact journal publications. Thus, the null Hypothesis H0-3 is rejected, assuming that the supportive CSP–CFP evidence is not biased toward an increase in the journal impact factor.
Hypothesis H0-4 further argues that the supportive CSP–CFP evidence is not biased toward journal articles as compared with published working papers. However, the descriptive findings in Table 3 indeed support the suggestion that the z values are significantly lower and less positive for working papers than for journal articles: For working papers, the z value is −2.149 (on the sub-effect level: 0.278), while for journal articles the value is 4.477 (on the sub-effect level: 0.789). In all, 27% of the working papers report significant positive findings (on the sub-effect level: 13%), whereas journal articles report 54% (on the sub-effect level: 20%). However, when controls for other factors such as the journal impact factor are applied, the findings in Table 2 do not confirm that z values are significantly higher and more positive for journal publications compared with working paper publications. The findings in Table 4 only confirm that on the study-effect level, working papers report significant positive effects on the CSP–CFP link significantly less often than journal publications. Thus, there is only weak evidence for potential editorial decision-making biases. Our data thus fail to reject the null hypothesis H0-4, suggesting that the supportive CSP–CFP evidence is not biased toward journal articles as compared with published working papers.
Theoretical aspects
Hypothesis H0-5 suggests that the evidence supporting the positive CSP–CFP link is not biased toward studies testing no theory than for studies testing theory from social science or economics. Table 3 indicates that the z value for studies relying on no special theory is 8.442 (on the sub-effect level: 1.341), which is higher than for studies relying on a finance/economic theory (1.109/0.281) or on a social science theory (3.863/1.109). However, the results with respect to the amount of positive significant support are not so clear. The regression results in Table 2 support the supposition that for the random-effect model and the study-effect model, the z values are significantly higher and more positive for studies relying on no theory than for studies relying on a social science or finance/economic theory. Table 3 confirms this result for study sub-effects: Publications without a theory report significant positive sub-study effects on the CSP–CFP link significantly more often than publications with a theory. We do, therefore, find some, albeit not comprehensive, evidence for rejecting the null hypothesis H0-5.
According to Hypothesis H0-6, the supportive CSP–CFP evidence is not biased toward studies not testing the H0 hypothesis for the CSP–CFP link. The descriptive findings in Table 3 show that studies testing the H0 hypothesis report lower z values (0.724/0.330) than studies testing a directed hypothesis (4.429/0.821). There are, however, only weak differences when determining whether a study finds a positive significant effect. Table 2 shows that on the sub-effect level, the z values for H0 studies are significantly lower than for studies testing directed hypotheses. However, the z values on the study-effect level do not differ significantly between the two study types. According to Table 4, the significant positive evidence between H0 studies and studies testing directed hypotheses also does not differ significantly. We therefore find only very weak support rejecting the null hypothesis H0-5, suggesting that the supportive CSP–CFP evidence is not biased toward studies not testing the H0 hypothesis for the CSP–CFP link.
Hypothesis H0-7 claims that the supportive CSP–CFP evidence is not biased toward studies not discussing the pros and cons of a positive CSP–CFP link. The descriptive findings in Table 3 support the supposition that the z values are significantly higher for studies containing no balanced discussion (9.69/1.525) compared with studies containing a balanced discussion (1.952/0.465). In all, 64% of the studies without discussion report significant positive evidence for a positive CSP–CFP connection (on the sub-effect level: 33%), and 47% of studies with discussion report positive evidence (on the sub-effect level: 14%). The findings in Table 2 support the hypothesis that z values are significantly lower for studies containing a balanced discussion of the pros and cons of a positive CSP–CFP link. Table 3 confirms this result for less significant positive effects on the CSP–CFP link on the sub-effect level. Thus, the null hypothesis H0-7 is rejected, assuming that the supportive CSP–CFP evidence is not biased toward studies not discussing the pros and cons of a positive CSP–CFP link. There is evidence for a bias—most likely caused by beliefs of the authors of these studies.
Methodological aspects
According to Hypothesis H0-8, evidence supporting a positive CSP–CFP link is not biased toward studies with greater methodological problems. The descriptive findings in Table 3 show that for studies that do not control for industry-fixed effects, firm-fixed effects, or time-lagged effects, the z values are 3 times higher and the amount of significant positive evidence is twice as high as in those studies that do control for these effects. The findings in Table 2 support the finding that methodologically weak studies report significantly higher z values. The findings in Table 3 confirm that studies including industry-fixed effects, firm-fixed effects, and time-lagged effects report significant positive effects on the CSP–CFP link less often. Thus, the null hypothesis H0-8 is rejected, assuming that the supportive CSP–CFP evidence is not biased toward studies with greater methodological problems.
Finally, Hypothesis H0-9 suggests that the supportive evidence for a positive CSP–CFP connection is not biased toward studies using multivariate empirical methods any more than for studies using univariate empirical methods. Table 3 illustrates that, on a descriptive level, we find no such evidence. As one would expect, the z values of multivariate results are much lower than the z values of univariate results. However, the form of analysis correlates with other study aspects such as the kind of CSP or CFP measure, or the underlying theory. Table 2 reports the meta-regression results by holding these aspects constant. There is support for the contention that multivariate analyses report higher z values for sub-effects compared with univariate analyses. However, we do not find evidence for study effects. Table 4 shows that studies reporting bivariate correlations or mean comparisons document significant evidence on positive CSP–CFP links to a much lesser extent on the sub-effect and study-effect levels. Thus, the null hypothesis H0-9 is rejected, assuming that the supportive CSP–CFP evidence is not more biased toward studies using multivariate empirical methods than toward studies using univariate empirical methods.
Summary of the Empirical Findings
This study finds only weak evidence for a true effect between CSP and CFP. However, the authors do find evidence for a positive reporting bias in the CSP–CFP literature (Hypothesis 1). The reporting bias in the CSP–CFP literature is associated with later publications (Hypothesis H0-2 rejected), with publications in high-impact journals (Hypothesis H0-3 rejected), with studies testing no theory (Hypothesis H0-5 rejected), with studies arguing dogmatically for a positive CSP–CFP link (Hypothesis H0-7 rejected), with studies applying weak methods (Hypothesis H0-8 rejected), and with studies using multivariate methods (Hypothesis H0-9 rejected). The findings show only weak evidence for the assumption that working papers are associated with fewer reporting biases (Hypothesis H0-4 failed to reject) and for the assumption that formulating a H0 hypothesis prevents reporting biases (Hypothesis H0-6 failed to reject). Table 5 summarizes these findings.
Summary of the Meta-Regression Findings.
Note. CSP = corporate social performance; CFP = corporate financial performance.
+sig.: Significant positive.
−sig.: Significant negative.
//: Non-significant effect.
Discussion and Conclusion
Reporting or selection biases occur in the management literature. For the positivist research paradigm in CSR research, the authors’ findings tentatively, and for current research, show that published evidence tends to overestimate efficiency. The CSP–CFP literature generally tends to find positive effects. These results suggest that this positive evidence may be affected by reporting bias in the literature. Such reporting biases arguably originate from collective cognitive structures within a research discipline.
Even though there remain other reasons that may generate natural asymmetries in funnel plots, such as data dependencies, estimation faults, or omitted variables, which are unknown to the researcher (Callot & Paldam, 2011), reporting biases seem to be very likely explanations for asymmetry in CSP–CFP findings. Collective cognitive structures within a research discipline may be the reason for reporting biases in the empirical literature. However, the extent of misestimation is often unknown. In the management literature in particular, almost no studies quantify the overestimation or underestimation of effects due to reporting biases. As a result, the conclusions of systematic reviews and meta-analyses based on published evidence alone, without controlling for reporting bias, may be misleading. This distortion is a serious concern in cases where these documents are being used to support decision making in the business context and to justify research questions and results. A logical implication is that reporting bias may result in inappropriate decisions by policy makers, managers, or scholars, and could thereby harm employees, shareholders, and other stakeholders, and lead to wasted resources and misdirected future research.
With respect to the conditions reinforcing reporting biases in a research discipline, this study’s findings indicate that editorial decision-making biases seem not to be the main cause. The remaining evidence can be explained by authors not writing papers on unfavorable results. Furthermore, methodologically weak studies are only published if the result confirms the mainstream view. Finally, we also find some evidence which may best be explained by manipulation. Multivariate analyses more often confirm the conventional view. In contrast to univariate analyses, multivariate analyses can legitimately achieve confirmation of the conventional view by altering the set of control variables or by using different types of regression analyses.
Future research has the following options for mitigating reporting bias. First, management researchers could start systematically to investigate reporting bias within their research field. There are many prominent effects which could be affected by reporting biases as well: for example, the pay-for-performance link for CEOs, the outsider-performance link within corporate governance research, or the management entrenchment evidence. Such investigations would possibly show the significance of reporting biases in business research. It would further help understand the circumstances under which reporting biases develop.
Second, business schools teach their management students how important scientific testing is for business decisions. Unexpected results are very important in everyday management because they help avoid costly negative present-value investments. In that respect, a stronger refocus on courses in theory and philosophy of science will also help scholars recognize the value and the importance of empirical results to falsify scientific theories. Today, most graduate schools educate their management students, especially PhD students, in statistics and the science of top-tier publications]at the expense of courses such as the theory of science that would help them cope with findings that contradict the collective cognitive structure within a field. It is not surprising that some papers find that reporting biases have increased in recent years (Fanelli, 2012), particularly in the social sciences (Fanelli, 2010b).
The authors invite other researchers to test the above assumptions. For the positivistic research paradigm in management research, it is important to identify and quantify reporting biases and to offer solutions.
This study has limitations that should be considered when interpreting our empirical findings, and in further research. First, other reasons that may generate natural asymmetries in funnel plots remain, such as data dependencies, estimation faults, or omitted variables, unknown to the researcher (Callot & Paldam, 2011). Second, we used statistical methods, with an assumption of normal distribution, to identify publication bias. A direct test would be more credible. Future research could try to collect a sample of published and unpublished studies to see whether our results can be replicated with such a sample. Third, the authors can only speculate which reasons cause the overrepresentation of positive research findings in CSP–CFP research. Our assumptions are that the collective cognitive structures within the field lead authors to write papers on unfavorable results less often, and that reviewers have a preference for positive results. These arguments are preliminary and should be tested by further research, by conducting author and editor surveys or by using experimental designs. Fourth, in the moderator analysis we only showed which studies are associated with more positive findings on the CSP–CFP link. While for most of our moderators, for example, the journal impact factor, the overdominance of positive findings should be caused by publication bias, there may be also plausible reasons for differences in CSP efficiency. We tried to validate our findings by running the FAT for subgroups and by analyzing the size of the corrected effect and of the publication bias. These results are vulnerable because we split the sample, and thus do not rely on the distribution of all research findings, and also because we do not control for other determinants. Future research may come up with better methods that allow differentiating between publication bias and corrected effect within subgroups. Finally, the study sample was collected until the end of the year 2009. The collection thus excluded newer publications on the CSP–CFP link. Newer studies could show less publication bias, because, for example, collective cognitive structures have changed. The authors can give no evidence on this issue, for which future research is needed.
Footnotes
Appendix A
Descriptive Statistics and Bivariate Correlations.
| ID | Variable | M | SD | Minimum | Maximum | Dummy | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | t value (sub-effect) | 0.74 | 2.02 | −8.78 | 17.77 | No | ||||||||||||||||||||||||||||
| 2 | t value (study effect) | 3.83 | 7.29 | −8.09 | 24.36 | No | .53 | |||||||||||||||||||||||||||
| 3 | Positive significant (sub-effect) | 0.19 | 0.39 | 0.00 | 1.00 | No | .70 | .47 | ||||||||||||||||||||||||||
| 4 | Positive significant (study effect) | 0.51 | 0.50 | 0.00 | 1.00 | No | .41 | .69 | .36 | |||||||||||||||||||||||||
| 5 | Publication after 1995 | 0.55 | — | 0.00 | 1.00 | Yes | .06 | −.01 | .07 | .16 | ||||||||||||||||||||||||
| 6 | Journal impact factor | 1.47 | 1.73 | 0.00 | 6.48 | No | .18 | .32 | .17 | .29 | −.28 | |||||||||||||||||||||||
| 7 | Working paper | 0.10 | — | 0.00 | 1.00 | Yes | −.07 | −.27 | −.05 | −.15 | .30 | −.28 | ||||||||||||||||||||||
| 8 | No theory | 0.31 | — | 0.00 | 1.00 | Yes | .20 | .43 | .18 | .21 | .13 | .05 | −.14 | |||||||||||||||||||||
| 9 | Finance/economic theory | 0.53 | — | 0.00 | 1.00 | Yes | −.24 | −.40 | −.21 | −.35 | −.18 | −.27 | .22 | −.71 | ||||||||||||||||||||
| 10 | CSP–CFP pros and cons discussion | 0.74 | — | 0.00 | 1.00 | Yes | −.23 | −.43 | −.21 | −.16 | .10 | −.41 | .15 | −.36 | .41 | |||||||||||||||||||
| 11 | H0 hypothesis | 0.16 | — | 0.00 | 1.00 | Yes | −.09 | −.19 | −.07 | .00 | .18 | −.10 | .43 | −.21 | .32 | .25 | ||||||||||||||||||
| 12 | Industry-fixed effects | 0.38 | — | 0.00 | 1.00 | Yes | −.11 | −.23 | −.06 | −.19 | −.05 | −.12 | .31 | −.04 | .13 | .15 | .17 | |||||||||||||||||
| 13 | Firm-fixed effects | 0.68 | — | 0.00 | 1.00 | Yes | −.20 | −.28 | −.17 | −.34 | −.14 | −.19 | .02 | −.35 | .43 | .08 | .08 | .14 | ||||||||||||||||
| 14 | Time-lagged effects | 0.79 | — | 0.00 | 1.00 | Yes | −.11 | −.16 | −.07 | −.31 | −.08 | −.11 | .03 | −.26 | .30 | −.03 | .01 | .16 | .70 | |||||||||||||||
| 15 | Correlation | 0.12 | — | 0.00 | 1.00 | Yes | .04 | .17 | .04 | .23 | −.05 | .19 | −.12 | .31 | −.29 | −.07 | −.07 | −.25 | −.51 | −.50 | ||||||||||||||
| 16 | t test, mean comparison | 0.62 | — | 0.00 | 1.00 | Yes | −.05 | .03 | −.11 | −.14 | −.22 | −.06 | −.30 | −.09 | .18 | −.07 | −.30 | −.13 | .41 | .34 | −.47 | |||||||||||||
| 17 | SRI vs. non-SRI portfolio/funds | 0.19 | — | 0.00 | 1.00 | Yes | −.11 | −.28 | −.09 | −.18 | .31 | −.33 | .55 | −.09 | .23 | .24 | .47 | .44 | .21 | .15 | −.18 | −.22 | ||||||||||||
| 18 | Crime/recall/incidents | 0.30 | — | 0.00 | 1.00 | Yes | −.04 | .02 | −.03 | −.15 | −.24 | −.05 | −.21 | −.03 | .04 | −.13 | −.29 | −.17 | .43 | .33 | −.24 | .41 | −.32 | |||||||||||
| 19 | Cash giving/contributions | 0.07 | — | 0.00 | 1.00 | Yes | −.01 | −.04 | −.07 | .19 | .21 | −.10 | −.08 | .23 | −.27 | .06 | −.09 | −.15 | −.40 | −.43 | .45 | −.28 | −.14 | −.18 | ||||||||||
| 20 | Social disclosure | 0.10 | — | 0.00 | 1.00 | Yes | −.02 | .00 | −.01 | .04 | .01 | .18 | −.11 | −.06 | .15 | −.02 | .22 | −.22 | .07 | .06 | −.01 | .05 | −.17 | −.23 | −.10 | |||||||||
| 21 | CSP regulations/principles | 0.02 | — | 0.00 | 1.00 | Yes | .02 | −.02 | .05 | .00 | −.09 | .00 | −.05 | −.10 | .00 | .03 | −.07 | −.02 | .02 | −.01 | −.06 | .01 | −.08 | −.10 | −.04 | −.05 | ||||||||
| 22 | Event based | 0.48 | — | 0.00 | 1.00 | Yes | −.11 | −.04 | −.11 | −.21 | −.35 | .00 | −.31 | −.20 | .23 | −.10 | −.28 | −.15 | .64 | .50 | −.35 | .64 | −.43 | .64 | −.27 | .14 | .04 | |||||||
| 23 | Market based | 0.25 | — | 0.00 | 1.00 | Yes | −.08 | −.21 | −.05 | −.08 | .22 | −.21 | .42 | −.07 | .19 | .20 | .43 | .36 | .02 | .05 | −.07 | −.23 | .73 | −.35 | −.10 | −.14 | −.05 | −.56 | ||||||
| 24 | 1960-1969 | 0.14 | — | 0.00 | 1.00 | Yes | −.08 | −.05 | −.08 | −.14 | −.11 | −.09 | −.12 | .01 | .01 | .04 | −.14 | .09 | .11 | .09 | −.06 | .10 | −.08 | .02 | −.10 | −.14 | .12 | .20 | −.10 | |||||
| 25 | 1970-1979 | 0.17 | — | 0.00 | 1.00 | Yes | .11 | .26 | .10 | .13 | −.39 | .34 | −.15 | .12 | −.11 | −.39 | −.10 | −.06 | .02 | −.01 | .07 | .12 | −.22 | .20 | −.06 | .17 | .20 | .18 | −.15 | −.06 | ||||
| 26 | 1980-1989 | 0.52 | — | 0.00 | 1.00 | Yes | .03 | .14 | .02 | −.04 | −.50 | .16 | −.25 | −.08 | .06 | .01 | −.23 | .03 | .19 | .09 | −.11 | .35 | −.27 | .37 | −.25 | −.10 | .01 | .43 | −.21 | .11 | .03 | |||
| 27 | 1990-1999 | 0.68 | — | 0.00 | 1.00 | Yes | −.05 | −.15 | −.05 | −.14 | .31 | −.29 | .04 | −.11 | .15 | .23 | .10 | −.08 | .30 | .20 | −.14 | .11 | .17 | .06 | .04 | −.11 | −.12 | .14 | .04 | .02 | −.55 | .02 | ||
| 28 | 2000-2009 | 0.27 | — | 0.00 | 1.00 | Yes | −.03 | −.16 | .00 | −.20 | .55 | −.26 | .49 | .09 | −.05 | −.03 | .12 | .16 | −.05 | .09 | −.19 | −.19 | .48 | −.17 | −.06 | −.21 | −.10 | −.35 | .35 | .00 | −.27 | −.54 | .04 | |
| 29 | Number of years analyzed | 6.69 | 4.98 | 1.00 | 29.00 | No | .04 | .05 | .03 | −.12 | .00 | .00 | .09 | −.01 | .02 | −.14 | −.16 | −.09 | .39 | .36 | −.31 | .28 | .10 | .53 | −.27 | −.09 | −.06 | .32 | .03 | −.10 | .24 | .31 | .20 | .11 |
Note. CSP = corporate social performance; CFP = corporate financial performance; SRI = socially responsible investing.
Appendix B
FAT Test on the CSP–CFP Link, Meta-Analysis of Margolis, Elfenbein, and Walsh (2007).
| Dependent variable: z Value | Sub-effects (weighted by random effects) |
Sub-effects (weighted by fixed effects) |
Study effects (clustered by study) |
|||||
|---|---|---|---|---|---|---|---|---|
| B a | t | B a | T | B a | t | B a | t | |
| Std. CSP–CFP effect (1 / SE) | 0.013 | 0.64 | 0.051 | 1.90 | −0.003 | −0.19 | 0.057* | 2.09 |
| Reporting bias (constant) | 1.165*** | 4.70 | 2.778*** | 3.60 | 4.856*** | 5.54 | 2.373** | 2.92 |
| CSP measure: Observer perceptions | ||||||||
| Charitable contributions | −0.096 | −0.11 | 0.425 | 0.45 | −0.153 | −0.18 | ||
| Corporate policies | −2.674** | −2.84 | −2.852** | −2.87 | −2.476** | −2.89 | ||
| Environmental performance | −1.381* | −2.44 | −1.021 | −1.52 | −1.277* | −2.22 | ||
| Revealed misdeeds | −1.229 | −1.42 | −1.544 | −1.23 | −0.991 | −1.23 | ||
| Screened mutual funds | −2.087** | −2.72 | −2.712** | −3.38 | −1.905** | −2.55 | ||
| Third party audit | −2.140*** | −3.87 | −2.799*** | −4.03 | −1.859** | −3.23 | ||
| Transparency | −1.945 | −1.96 | −1.729 | −1.68 | −1.919* | −2.23 | ||
| CFP measure: Account based | ||||||||
| Market based | −0.447 | −1.08 | −0.588 | −1.42 | −0.409 | −1.08 | ||
| CFP as time-lagged dependent variable | −0.742 | −1.31 | −0.970 | −1.61 | −0.661 | −1.19 | ||
| CFP/CSP simultaneously measured | −0.469 | −0.93 | −1.147 | −1.91 | −0.312 | −0.66 | ||
| Control for industry | −0.067 | −0.18 | −1.005* | −2.33 | 0.085 | 0.23 | ||
| Control for ownership control | 0.481 | 0.51 | −1.389 | −1.10 | 0.715 | 0.88 | ||
| Control for firm size | −0.289 | −0.70 | −0.230 | −0.54 | −0.292 | −0.71 | ||
| Control for risk | 0.195 | 0.43 | 0.493 | 0.98 | 0.159 | 0.36 | ||
| Number of observations | 205 | 205 | 205 | 148 (205) | ||||
| F test | 0.41*** | 2.19** | 4.17*** | 1.90* | ||||
| R 2 | .0016 | .1429 | .2911 | .1335 | ||||
Note. FAT = funnel asymmetry test; CSP = corporate social performance; CFP = corporate financial performance.
Unstandardized regression coefficient.
p < .05. **p < .01. ***p < .001.
The article was accepted during the editorship of Duane Windsor
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
Supplemental Material
The online appendix C is available for this article online.
Notes
Author Biographies
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
