Reply to “A Replication of ‘Political Influence Behind the Veil of Peer Review’ ( Journal of Law and Economics 2009)”

In Hannum’s initial attempt to replicate Hegde (2009), as well as Hegde and Mowery (2008a; 2008b), he undertook what he called a series of robustness checks, did not find anything to “break” our originally published findings, and proceeded to alter the data in ad hoc ways. In particular, Hannum converted pooled cross-sectional data into panel data by filling zeroes for organizations that did not receive National Institutes of Health (NIH) grants. Hannum now seems to have done away with this alteration to the data.

Instead, Hannum (this issue) has now made three changes to the original data and specifications that appear to produce estimates of political influence that differ from those my coauthor and I reported in our papers. First, he selectively applies name standardization to grant recipients in the state of New York alone but not to recipients in other states, and we had used the data set available from the NIH as is, based on the assumption that not undertaking name standardization would at best result in nonsystematic error. Second, he selectively includes a dummy in the regressions to absorb the effects specific to the state of California during a particular time period without providing an economic basis for the inclusion. Third, he removes the dummy we had included in our original regressions for an outlier to pick up the “Alfonso D’Amato effect.” These ad hoc changes, taken together (if not individually), appear to reduce the estimate of the House Appropriation Committee’s Labor, Health and Human Services, Education and Related Agencies subcommittee variable from 5.3 percent (p < .01) reported in the original paper to 3.3 percent (p < .15 approximately).

Based on this reduction in the estimate of our coefficient, Hannum concludes that the credibility of our original results are “undermined” and that “a reexamination of [the evidence presented in the original paper] does not support that conclusion.” I have several issues with the above analysis and conclusion.

First, Hannum appears to inject several errors to the data and analysis in his alacrity to poke holes in our original results. For example, he applies name standardization to grant recipients in New York but not to recipients in other states, he adds dummies to absorb the effects of grant receipts to California for certain time periods without economic basis, and he omits the control for the outlier used in our original estimations. It is no surprise then that these alterations produce a different coefficient (3.3 percent rather than 5.3 percent). However, without checking for whether this altered coefficient is significantly different from the ones we report in our papers (it appears that it is not—the confidence intervals of the originally reported estimate and the ones proposed by Hannum clearly overlap), he hastens to conclude that our original findings do not stand up to closer scrutiny.

Second, the flawed replication exercise is incomplete. Our originally reported findings go well beyond the baseline findings replicated by Hannum. In our paper, we go beyond the baseline regressions replicated by Hannum by using plausibly exogenous variation from the exits of Congressional subcommittee members to identify the effects of representation. In additional analyses, conducted at the researcher-field-year level, we also control for the quality of the fields of researchers, which may be systematically related to political representation and grant receipts. Hannum does not mention these important tests essential for credibly establishing the effect of political influence on grants and jumps to an erroneous conclusion without attempting to replicate these results.

Third, and most egregiously, in his abstract, Hannum downplays our several robustness checks that confirmed, or even strengthened, our original findings. Instead, the departure from the original finding is broadly generalized, overclaimed, and sensationalized (“A reexamination of their evidence does not support that conclusion”).

I encourage and appreciate replication efforts, which is why I immediately supplied the data and the code to Hannum on request. Such efforts are essential to keep in check the tendency of researchers to jump to conclusions that appear interesting—if not deliberately misreport results—without adequately checking their analysis for robustness. However, in his eagerness to show us wrong, Hannum appears to have committed exactly the wrong that replication exercises are meant to right. Hannum’s replication analysis is inaccurate, incomplete, and sensationalist.