A Different Certification Effect of the Small Business Innovation Research (SBIR) Program

Abstract

This study examines the role of public investments in inducing small firms to develop risky, early-stage technologies. It contributes to expanding our understanding of the consequences of high-technology policies by investigating in more depth the effect of the Small Business Innovation Research (SBIR) program on the innovation effort and ability to attract external capital of small business start-ups using a new sample and estimation approach. The authors found empirical evidence that the public cofinancing of private research and development has a positive effect on the innovation propensity of small high-tech start-ups. However, contrary to theoretical expectations, they did not find any significant “certification effect” of receiving an SBIR award on attracting follow-on investment. What the authors discovered is a different certification effect: SBIR recipient firms are more likely to attract external patents. This finding confirms that enterprises need to orchestrate a portfolio of internal and external knowledge assets to produce innovations with unique competitive advantage.

Keywords

policy evaluation research and development propensity score matching Small Business Innovation Research innovation policy

Technology programs, although not uncontroversial, are ubiquitous.¹ With the rise of evidence-based policy and performance measurement movements in the public sector (Cozzens & Melkers, 1997; Heinrich, 2007; Shapira & Kuhlmann, 2003), studies examining their effectiveness also abound.² (See e.g., Aerts & Schmidt, 2008; Berube & Mohnen, 2009; González & Pazó, 2008; Koga, 2005; Lee & Cin, 2010; Özçelik & Taymaz, 2008, which examined research and development [R&D] programs in Germany, Canada, Spain, Japan, Korea, and Turkey, respectively.) In the United States, one such program is the Small Business Innovation Research (SBIR) program, which underwrites small-firm R&D. The SBIR is also well studied. Perhaps with the exception of Wallsten (2000), SBIR evaluation studies have found a positive effect of the federal technology program on employment, sales, entrepreneurship, research commercialization, and social welfare.³

This study reexamines the impact of the SBIR using a new sample of small business start-ups that started operations in 2004. Only a few evaluation studies have focused on the role of public financing on small business start-ups. My focus in this study is early-stage technology development by small business start-ups. From a Schumpeterian perspective, small business start-ups are agents of technical change because of their propensity to introduce new products and processes in emerging or less-crowded technological fields (Acs & Audretsch, 1990; Almeida & Kogut, 1997; Breitzman & Hicks, 2008). These new technologies can potentially supersede current technologies and, in the process, redefine new market opportunities that can sustain the innovating firm’s and the nation’s technological leadership and global competitiveness. The unique integrated data set of small business start-ups that I constructed allows for examination of the effect of SBIR financing on risky, early-stage technology development.

I integrated the Kauffman Firm Survey (KFS) from the Ewing Marion Kauffman Foundation with the SBIR recipient data set from the U.S. Small Business Administration (SBA), and used advances in statistical matching to improve unit homogeneity between treated and comparison groups of small business start-ups. The integrated KFS–SBA data set—which contains both recipient and nonrecipient small firms—along with statistical matching allowed me to empirically construct the counterfactual outcomes of SBIR recipients.

Although this study estimates the effect of the SBIR program on the innovation effort, the ability to attract external capital, and other metrics of postentry performance (e.g., sales, employment size) of small business start-ups, its main contribution to the literature is the uncovering of what I call a different certification effect of the SBIR program. This benefit, which may be defined as the strengthening of the recipient’s capacity to orchestrate internal and external knowledge assets, has implications to national innovation strategy and the way oversight agencies evaluate public programs that cofinance private R&D.

The next section of the study discusses the intellectual support for the SBIR program. The third section presents the data and method used in estimating the treatment effect of SBIR on small business start-ups, while the fourth section provides descriptive statistics and the results of the treatment effects analyses. The final section concludes and derives implications for both theory and practice of technology-based economic growth and development.

Basic Economics of SBIR

The main objective of the SBIR program is to stimulate technological innovation among small firms, or as Toole and Czarnitzki (2007) put it, the program is a source of financing to develop unproven (and thus risky) but promising technologies. SBIR provides R&D subsidy at the early stage of technology development. It thus could be construed as a government venture capital initiative to underwrite technologies that are not close to commercialization (Branscomb & Keller, 1999; Lerner, 1999). Because the outputs and outcomes of public financing are inherently uncertain, technology programs like the SBIR could also be regarded as bets on the country’s technological future (Borrus & Stowsky, 1999).

The more often-used theoretical rationale for the public cofinancing of private R&D is the market failure argument that dates back to Arrow’s (1962) seminal article on the economics of inventive ideas. It posits that technology R&D requires large fixed costs to procure technical manpower, equipment, and support services; that the potentially commercially useful knowledge resulting from this R&D is partially nonexcludable; and that most innovative projects face technical, market, and competitive uncertainties. First, firms may have to commit at least a “critical minimum level of innovation effort” (Metcalfe, 1995, p. 424) before their R&D programs can produce desired innovation outputs. Second, they may be unable to prevent competitors from reverse engineering or copying their new product or production technique. Even in the presence of a strong patent system, small and new enterprises lack both legal resources to protect their innovation from imitation and the market power to extract monopoly rents from the same (Cooper, 2003). Third, the new product prototype may not work on a commercial scale, limiting market potential. Or, if the same product is mass produced, it may not enjoy market demand at a volume sufficient to recoup the cost of R&D. If a competing firm develops a similar or more superior technology, the returns to R&D are limited or, worse, the first-mover innovating firm drives out of the market. All these lead to what is termed in the literature as the “problem of appropriability,” which weakens the ability of innovators to realize a reasonable rate of return from their innovation (Geroski, 1995).

The SBIR program is expected to alleviate the problem of appropriability, more specifically in the production of risky, early-stage technology by small- and medium-sized enterprises. The SBIR research grant helps small firms satisfy the required minimum scale of R&D (e.g., minimum size and competence of the R&D team); it allows, for example, the hiring of university scientists or engineers to spearhead or support its R&D effort. The infusion of external financing from the SBIR grant enables the recipient small firm to intensify its R&D effort and, by extension, use its R&D resources more efficiently. Second, the public cofinancing of R&D (through the SBIR program) shifts the recipient firm’s marginal cost to the right (David, Hall, & Toole, 2000; Metcalfe, 1995), pushing its innovation effort theoretically up to a level that closes the gap between the private level and the “socially optimal” level of R&D. Third, the SBIR grant alleviates the risks and uncertainties of the outcomes of the innovation effort. By providing small firms the opportunity to undertake more systematic R&D, the SBIR program enables recipients to have a better estimate of the rewards and risks of developing their intended new product or process. The prospect of having federal agencies procure the proposed technology to pursue their mandates and missions (e.g., national defense and health care) also lowers market uncertainties for some SBIR-financed research projects.

The increase in the innovation effort of recipient firms as a result of public financing alleviating the problem of appropriability is the so-called additionality effect of technology policies and programs (Clarysse, Wright, & Mustar, 2009; Georghiou, 2002).

The risks and uncertainties in innovation also have an intermediate effect on the firm’s ability to attract external capital. Banks and external investors (e.g., venture capitalists and angel investors) are reluctant to extend credit without a credible market signal of the quality of the firm and the prospect of its innovation project. For innovation projects that are perceived—rightly or wrongly—as risky, capital providers usually require an extra premium to extend credit, pushing up the cost of external capital (Hubbard, 1998). Information asymmetries in the financing of innovation are more problematic for R&D-performing small firms and are compounded when we factor in the stage of the technology development cycle. More specifically, Cooper (2003) found that small businesses lacked sufficient funding at the early stage of technology development, implying that capital providers are risk averse in extending credit to innovation projects that are not “near-market” (Lerner, 1999). Such risk aversion leads to a substantial “financing gap” (Auerswald & Branscomb, 2003) that deprives firms that are willing to assume a portion of the risks and resources necessary to develop early-stage technologies. Some economically viable R&D projects may not take off, generating a social welfare loss. This is where the SBIR program can have a significant contribution. The research grant can have a “halo” or “certification” effect in the application for external capital (Lerner, 1999; Link & Scott, 2010). Recipient small firms can leverage their SBIR awards to signal the “viability of the project and the company” (Siegel, Wessner, Binks, & Lockett, 2003, p. 124). Because the innovation project is subjected to a vetting process by SBIR participating agencies, SBIR financing can thus certify that it is both “scientifically sound” and “commercially promising” (Feldman & Kogler, 2008, p. 442), providing the extra push to financiers to extend additional capital. Lerner (1999) showed that small firms that received SBIR grants were three times more likely than nonrecipients to attract venture capital, a finding validated by a more recent study by Toole and Turvey (2009), who documented that SBIR Phase I grants had a positive effect on receiving follow-on external private investment.

Data and Method

Data

The Ewing Marion Kauffman Foundation granted access to the confidential microdata of the KFS, an inflow sample of 4,928 businesses founded in 2004 and tracked ever since.⁴ I also requested from the SBA a data set of SBIR recipients from 2004 to 2008.⁵ I subsequently requested the Kauffman Foundation to integrate the KFS and SBIR recipient data sets using the Data Universal Numbering System, a unique numeric identifier assigned to a single business identity. The integrated KFS-SBIR data set identified 25 small business start-ups that received SBIR financing to develop new technologies between 2007 and 2008.

In the empirical analysis, each of these 25 recipient small business start-ups is matched with an observationally similar nonrecipient small business start-up. An advantage of this data set is that both recipient and nonrecipient small firms came from the same probability sample of firms that started operations in 2004, unlike prior SBIR studies that manually combined the two groups of small enterprises.⁶ Statistically, the two subsamples are identically distributed.

Prior to matching and estimation, I also restricted the sample of potential controls to small businesses by dropping all start-ups that had more than 500 employees prior to the 2007-2008 SBIR treatment period.

The Treatment Sample and Potential Controls Before Matching

Table 1 shows that nonrecipient small business start-ups significantly differ from the 25 start-ups that received SBIR financing in 2007-2008.

Table 1.

Baseline Characteristics by Treatment Status.

Baseline characteristics (2004)	Potential controls (n = 4,000+)	Recipient small business start-ups (n = 25)	Difference	p
Firm size
Number of employees	1.94	1.68	0.26	.840
Human capital
Postgraduate education	0.20	0.80	−0.60	.000
Industry experience	0.55	0.72	−0.17	.095
Technological capacity
Prior R&D performance	0.21	0.68	−0.47	.000
Number of patents	0.15	3.24	−3.09	.000
Positive sales	0.91	0.65	0.26	.000
High-tech industry
Pharmaceutical	0.01	0.08	−0.07	.000
Chemicals	0.02	0.08	−0.06	.014
Machinery	0.04	0.08	−0.04	.350
Electronics	0.04	0.24	−0.20	.000
Electrical equipment	0.01	0.04	−0.03	.204
Medical/surgical equipment	0.002	0.12	−0.118	.020
R&D services	0.20	0.28	−0.08	.346
Geographical location
Location in R&D intensive States (e.g., CA, MA)	0.84	0.80	0.04	.594

The treated sample is four times more likely than the untreated sample to have owners with a postgraduate education or training (p < .001). Moreover, the owner–entrepreneurs of SBIR recipients have longer experience in the industry where the start-up operates (p < .10). The entrepreneur’s graduate training (whether it be a research degree in science and engineering or a professional degree like an MBA) and prior industry experience are knowledge assets that are critical in searching for and recognizing new business opportunities that are commercially promising (Shane, 2000).

Recipient small business start-ups also have a significant initial advantage in technological capacity. SBIR-financed start-ups are more than three times more likely to conduct R&D than the untreated group (p < .001). They are also more productive in generating patents (p < .001). On average, treated start-ups had more than three patents at the end of 2004, whereas a majority of the potential controls did not possess any patent.

A significantly larger proportion of SBIR-backed start-ups are in the fields of pharmaceuticals (p < .001), chemicals (p < .05), electronics (p < .001), and medical/surgical equipment (p < .05). A larger percentage of treated start-ups are also operating in other high-tech areas like machinery, electrical equipment, and R&D and engineering services, but the differences in proportions between recipient and nonrecipient start-ups are not significantly different from zero.

Untreated start-ups, in contrast, have an advantage over SBIR recipients in employment size, sales performance, and location in R&D intensive states. Ninety-one percent of the potential controls sold goods and/or services in 2005 compared with only 65% of SBIR-financed small business start-ups. The 25 percentage point advantage of untreated start-ups over their treated counterparts is statistically significant (p < .001). However, the same cannot be said of firm size and location advantages of nonrecipient start-ups. On average, untreated and treated start-ups had 1.9 and 1.7 employees, respectively, in their initial year of operation, but this difference is both substantively and statistically negligible (p < .85). The potential controls are 4 percentage points more likely to locate their operations in top R&D performing states like California and Massachusetts than did SBIR recipients, but this difference is also not statistically significant (p < .60). The possibility that such difference across treatment status is due to random chance cannot be ruled out.

The systematic differences between the treated and untreated samples justify the construction of a comparison group that is similar in observable characteristics to the treated small business start-ups.

Estimation Approach

In this study, I seek to identify the causal impact of SBIR on small firm-level outcomes. I follow the counterfactual approach to causal analysis, which requires, before causality can be attributed from X to Y, that these two conditions are met: (a) Y increased as a result of X and (b) Y did not increase in the absence of X (Lewis, 1973). To establish that X causes Y, it is not enough that X and Y occurred together; the second condition, which is the counterfactual condition, must also be true.

The counterfactual condition is not observable empirically. If a small firm is funded by SBIR, we can observe its outcome with the SBIR but not its counterfactual outcome; that is, its outcome had it not received the SBIR subsidy. In the same manner, if a small firm is not subsidized by SBIR, we can observe its outcome without SBIR but not its counterfactual outcome—its outcome had it received the SBIR research grant. Thus, the causal inference problem is a “missing data” problem (Holland, 1986).

To solve the missing data problem, we need to look for observations that can serve as empirical surrogates for treated observations had they not participated in the SBIR program. An attractive option is to consider the observed outcomes of nonrecipient small firms as a proxy for the unobserved counterfactual outcomes of SBIT recipients. The problem with this strategy is that nonrecipients may differ in observable and unobservable characteristics. These differences, if systematic, will confound the relationship between the SBIR program and relevant firm-level outcomes. In short, it will bias the treatment effect estimates of SBIR.

In this study, I use matching as a method to control for observable characteristics. My aim is to reorganize the original sample (Gelman & Hill, 2007) or, more aptly, to create a synthetic sample (Cameron & Trivedi 2005) or a strategic subsample (Morgan & Winship, 2007) that includes a comparison group similar in observational attributes to those of the treated sample. All nonrecipients that are not observationally similar with any of the SBIR recipients are dropped from this synthetic sample. In this sense, matching can be construed as an attempt to mimic the random assignment process in experimental studies (Khandker, Koolwal, & Samad, 2010) or to create a “quasi-experimental contrast” (Morgan & Winship, 2007) by balancing observed covariates across treatment status.⁷

When implemented manually, matching is tedious; the procedure becomes more cumbersome the larger the set of covariates or observable characteristics to match. This dimensionality problem can be significantly reduced by matching on the propensity score; that is, the conditional probability of treatment or program participation, formally expressed as follows:

P r o p e n s i t y s c o r e = P (T = 1 / X)

Propensity score matching (PSM), which originated from Rosenbaum and Rubin (1983), is a statistical method to match recipients and nonrecipients on the basis of the propensity score, which is a scalar variable, instead of manually matching on a vector of variables. On average, observations with the same distribution of propensity scores will have the same distribution of observed covariates X (Pearl, 2009). Thus, if the strong ignorability of assignment assumption holds, the use of a comparison group (selected through PSM) to reconstruct the unobserved counterfactual outcomes of treated cases is sufficient to remove selection bias (Heckman, Ichimura, & Todd, 1998; Rosenbaum & Rubin, 1983).

Several studies support the use of nonexperimental evaluation methods. For example, Glazerman, Levy, and Myers (2003) found that the average bias of the estimates in nonexperimental studies was close to zero and thus not systematically positive or negative; Greenberg, Michalopoulos, and Robins (2006) found that experimental and nonexperimental evaluations provide the same mean effect; and Dehejia and Wahba (1999, 2002) found that PSM estimates could reasonably replicate experimental affect estimates.

Furthermore, following Ho, Imai, King, and Stuart (2007), I also estimated the treatment effect of SBIR by running regression analyses after balancing the data. I fitted least squares regressions on the homogeneous sample of recipient and nonrecipient small business start-ups. Gelman and Hill (2007) have shown that controlling for confounding covariates through ordinary least squares (OLS) regression is cleanest “if the units receiving the treatment are comparable to those receiving the control” (p. 199).

Propensity Score Model

In the propensity score model, I controlled for the following covariates that affect both the selection of small business start-ups to the SBIR program and their postentry performance: firm size, human capital, technological capacity, industrial classification, and geographical effects (Figure 1).⁸ Firm size is measured by the number of employees of the start-up at the beginning of its operation in 2004; human capital by the level of education and length of experience of the owner–entrepreneur; technological capacity by prior R&D performance, possession of patents, and sales prior to the receipt of the SBIR funding; industrial classification by seven categorical variables—pharmaceuticals, chemicals, machinery, electronics, electrical equipment, medical and surgical equipment, and R&D and engineering services—with other sectors as the reference category; and geographical effects by the start-up’s location in R&D intensive states. Appendices A and B provide the definitions of the variables used in the analysis.

Figure 1.

SBIR propensity score model.

After PSM, I compare the posttreatment outcomes of SBIR recipients and comparable nonrecipient start-ups in terms of R&D performance, innovation propensity, patent size, licensing-out of own patents, licensing-in of external patents, ability to attract external capital, sales, and employment size. Mathematically, where Y is the outcome of interest, the average treatment effect on the treated (ATT) is calculated as follows:

A T T E s t i m a t o r_{M A T C H I N G} = E_{X i / T = 1} [E (Y_{i} / T_{i} = 1, X_{i}) - E (Y_{i} / T_{i} = 0, X_{i})

The outer expectation in Equation (2) is taken over the distribution of X_i/T = 1; that is, the distribution of observed X in the treatment group. The overlap condition for identifying ATT requires that the support of X for the treated sample be a subset of the support of X for the untreated sample (Sekhon, 2008). This implies that untreated observations whose covariate values are outside of common support will be dropped in the estimation of ATT. Only treated cases and matched untreated cases are retained in the analysis. Dropping observations outside of common support will improve unit homogeneity between treated and untreated cases, making policy and program evaluation more meaningful (Guo & Fraser, 2010). In addition, Rosenbaum (2005) has shown that improving unit homogeneity not only reduces variability of the estimates of treatment effects but also their sensitivity to unobserved bias.⁹

Empirical Results

This section discusses the impact of SBIR on postentry performance of small business start-ups, focusing on its additionality and certification effects.

Treatment and Comparison Sample After Propensity Score Matching

More than 4,000 start-ups that did not receive SBIR funding are matched with the treated sample.¹⁰ Consistent with the propensity score theorem (Pearl, 2009), units with identical or nearly identical propensity scores have, on average, the same distribution of covariates, which in this case are antecedent variables that confound the relationship between receiving SBIR subsidy and firm-level outcomes. Table 2 presents the test of differences in means and proportions of these explanatory variables. The null hypotheses cannot be rejected at the 5% level, indicating that the distributions of human capital, technological capacity, geographical location, and industrial classification are not significantly different across treatment status.

Table 2.

Difference in Baseline Characteristics After Propensity Score Matching.

Baseline characteristics (2004)	Matched comparison group	Recipient small business start-ups	Difference	p
Firm size
Number of employees	1.25	1.17	0.08	0.891
Human capital
Postgraduate education	0.68	0.83	−0.15	0.219
Industry experience	0.82	0.72	0.10	0.394
Technological capacity
Prior R&D performance	0.53	0.61	−0.08	0.529
Number of patents	1.96	1.94	0.02	0.995
Positive sales	0.65	0.72	−0.07	0.566
High-tech industry
Pharmaceutical	0.11	0.06	0.05	0.527
Chemicals	0.07	0.11	−0.04	0.577
Machinery	0.07	0.06	0.01	0.828
Electronics	0.21	0.22	−0.01	0.916
Electrical equipment	0.11	0.06	0.05	0.527
Medical/surgical equipment	0.05	0.11	−0.06	0.386
R&D services	0.19	0.28	0.09	0.444
Geographical location
Location in R&D intensive States (e.g., CA, MA)	0.74	0.72	0.02	0.903

Treatment Effect Estimates

Tables 3 and 4 present the results of the treatment effects analyses. I focus on two estimates: (a) estimates from PSM, which is the difference in group means between SBIR recipients and their well-matched nonrecipient counterparts (column 7 of the results tables); and (b) estimates from OLS regression within common support; that is, the estimate from fitting a least squares regression using only data from the homogenous sample of recipient start-ups and their observationally similar nonrecipient counterparts (column 8 of the results tables).¹¹ For comparison purposes, I also included in Tables 3 and 4 estimates from the naïve estimator, derived as the difference in group means of SBIR recipient and all potential controls.

Table 3.

Average Treatment Effect on the Treated Estimates: R&D and Innovation.

Models	Outcome variable	Treated subsample	Matched untreated subsample	Total sample size	Treatment effect estimate
Models	Outcome variable	Treated subsample	Matched untreated subsample	Total sample size	Naïve estimator	PSM estimator	Regression within common support
Model I	R&D performance in 2008	18	57	75	0.73***	0.56***	0.49***
Model I	R&D performance in 2008	18	57	75	(8.57)	(4.80)	(5.15)
Model II	R&D expenditure in 2008	18	57	75	672,092***	477,900*	442,412*
Model II	R&D expenditure in 2008	18	57	75	(10.31)	(2.00)	(1.97)
Model III	Log R&D expenditure in 2008	12	35	47	3.56***	2.55***	2.34**
Model III	Log R&D expenditure in 2008	12	35	47	(6.73)	(3.74)	(3.36)
Model IV	Innovation propensity in 2009	18	49	67	0.47***	0.33*	0.39*
Model IV	Innovation propensity in 2009	18	49	67	(5.51)	(2.18)	(2.36)
Model V	Licensing-out of patents	13	26	39	0.09	0.19^†	0.16
Model V	Licensing-out of patents	13	26	39	(0.098)	(1.46)	(1.25)
Model VI	Licensing-in of patents	18	54	72	0.25***	0.22*	0.16*
Model VI	Licensing-in of patents	18	54	72	(8.36)	(2.20)	(1.72)
Model VII	Patent size	11	17	28	3.98**	1.34	0.83
Model VII	Patent size	11	17	28	(3.38)	(0.75)	(0.40)
Model VIII	R&D performance in 2009	17	52	69	0.73***	0.43***	0.41***
Model VIII	R&D performance in 2009	17	52	69	(8.38)	(3.96)	(3.55)
Model IX	R&D expenditure in 2009	17	55	72	609,597**	270,388	446,644^†
Model IX	R&D expenditure in 2009	17	55	72	(2.47)	(0.48)	(1.33)

Note. PSM = propensity score matching. Significant at ***0.1%, **1%, *5%, and ^†10%; numbers in parentheses are t statistics.

Table 4.

Average Treatment Effect on the Treated Estimates: External Capital and Other Outcome Variables.

Models	Outcome variable	Treated subsample	Matched untreated subsample	Total sample size	Treatment effect estimate
Models	Outcome variable	Treated subsample	Matched untreated subsample	Total sample size	Naïve estimator	PSM estimator	Regression within Common Support
Model X	Employment size in 2009	18	53	71	5.36*	7.28**	6.09***
Model X	Employment size in 2009	18	53	71	(1.94)	(3.04)	(3.69)
Model XI	External capital from banks and nonbank financial institutions	17	53	70	0.10*	0.12	0.08
Model XI		17	53	70	(1.72)	(1.27)	(1.02)
Model XII	External capital from all sources	11	34	45	0.05	−0.05	−0.12
Model XII	External capital from all sources	11	34	45	(0.55)	(−0.43)	(−1.04)
Model XIII	Sales in 2009	18	54	72	−0.01	0.14	0.09
Model XIII	Sales in 2009	18	54	72	(−0.13)	(1.23)	(1.24)
Model XIV	International sales in 2009	14	45	59	0.36**	0.11	0.06
Model XIV	International sales in 2009	14	45	59	(3.88)	(0.67)	(0.40)
Model XV	Profit in 2009	17	52	69	−0.08	0.01	0.01
Model XV	Profit in 2009	17	52	69	(−0.67)	(0.10)	(0.05)

Note. PSM = propensity score matching. Significant at ***0.1%, **1%, *5%, and ^†10%; numbers in parentheses are t statistics.

As expected, SBIR recipients are more likely to perform R&D in 2008 than observationally similar start-ups that did not obtain an SBIR R&D subsidy. Focusing on the PSM estimator, 19 out of 57 matched nonrecipients (or 33%) performed R&D in 2008 compared with almost 89% of SBIR recipients. This 56 percentage point difference in the probability of small business start-ups to conduct R&D in 2008 is statistically significant at the 0.1% level. The odds of an SBIR recipient performing R&D in 2008 is 16 times as high as the odds of a nonrecipient (p < .001), holding constant human capital, technological capacity, geographical location, and industrial classification. The OLS estimate of the same probability difference is close at 49 percentage points (p < .001).

How much is the actual R&D expenditure advantage of treated start-ups? Without PSM, the estimate of the advantage is $672,092 (p < .001). After balancing the covariates, the treatment effect estimate is reduced to $477,900, but it remains statistically significant at the 5% level. On average, SBIR recipient start-ups spent $663,379, whereas their observationally similar nonrecipient counterparts spent only $185,479. The OLS estimate of the R&D expenditure advantage of SBIR recipients over nonrecipients is slightly smaller at $442,412 (p < .05). In Model III, in which the outcome variable is the natural logarithm of the total R&D expenditure in 2008, SBIR recipient start-ups spent on average 234% more in R&D than their observationally similar nonrecipient counterparts.

SBIR recipients also have a decisive advantage over their observationally similar nonrecipient counterparts in the introduction of product and process innovations in 2009. PSM and OLS estimates indicate that SBIR-financed start-ups are 33 and 39 percentage points, respectively, more likely to introduce innovation in 2009 than start-ups not supported by the R&D subsidy program for small businesses (p < .01). The odds of the treated subsample in introducing innovation are about four times as high as the odds of the matched untreated subsample.

I also examined the propensity of small business start-ups to license-in external patents and to license-out their own patents in 2009. The treatment effect estimate without matching is a 9 percentage point advantage of treated small business start-ups in licensing out their own patents to other firms, but it is not statistically significant (p < .50). After balancing the confounders, the estimated ATT is substantially higher at 19 percentage points and now marginally statistically significant at the 10% level.

A very interesting finding is that SBIR recipients are more likely to license-in external patents. After balancing the data, the treatment effect of SBIR financing on the probability of licensing-out own patents is 22 percentage points, which is statistically significant at the 5% level. The point estimate of OLS is lower at 16 percentage points but still significant at the 5% level because of lower estimated standard error than what was obtained from the difference in group means after PSM.

The naïve estimator put the posttreatment employment size advantage of SBIR recipients at 5.4 employees (p < .05). However, when observable characteristics were balanced through PSM, the firm size advantage of SBIR-backed start-ups grew to 7.3 employees (p < .01). On average, the treated subsample had 9.4 employees in 2009, whereas nonrecipients had only 2.2 employees. Least squares regression analysis within common support estimates the size advantage of SBIR recipients at 6.1 employees (p < .01), which is very close to the PSM estimate. In Table 2, the treated and the matched untreated start-ups, by force of statistical matching, started on an equal footing in employment size. Both started at about one employee in 2004 (p < .50). But after 5 years, SBIR recipients grew to about nine employees, or more than an eightfold increase. On the other hand, their observationally similar counterparts only managed to grow from about one employee in 2004 to about two employees in 2009, or only a twofold increase.

Contrary to expectations, the treatment effect estimates of SBIR financing on attracting capital are not statistically significant. SBIR-financed small business start-ups are about 12 percentage points more likely to obtain additional capital from banks, government agencies, and other nonbank financial institutions, but such an advantage is not statistically significant (p < .25). Moreover, the sample data show that SBIR recipients are even slightly less likely than observationally similar nonrecipient start-ups to obtain capital from all external sources including family, friends, and other individuals (p < .80).¹² When all sources of external capital are taken into account (i.e., loans from family, friends, other individuals, government agencies, banks, and nonbank financial institutions), there is almost no difference in the ability of treated and untreated small business start-ups to attract external capital.

Discussion and Conclusion

The objective of this study is to contribute to expanding the knowledge base on the consequences of research, technology, and innovation policies. Although I agree that public programs like the SBIR can be construed as technological bets on our collective economic future (Borrus & Stowsky, 1999), the payoffs from these bets should at least be nonnegative. They must address actual technology market failures (Tassey, 2007). This research could be seen as part of the larger effort to identify more systematically which policies purport to encourage innovation work and which do not, specifically by examining in more depth the SBIR program.

Using the realized outcomes of observationally similar nonrecipient start-ups as the counterfactual outcomes of SBIR recipients had they not received SBIR funds, I found empirical evidence of the input additionality effect of the SBIR program.¹³ The expectation is that SBIR recipients are undertaking more risky but higher-return innovation projects with the R&D subsidy. I also found significant output additionality of SBIR. Recipient start-ups are more likely to introduce process and/or product innovations 1 to 2 years after receiving R&D subsidy.¹⁴ Third, SBIR recipients grew significantly faster than nonrecipients in terms of employment size at least 1 year after receiving the R&D subsidy. This result may indicate that SBIR program funds can help in augmenting technical personnel like scientists and engineers and, possibly, in hiring complementary human resources like market researchers. This result could be construed as the employment effect of the SBIR.

Contrary to the findings of Lerner (1999) and Toole and Turvey (2009) that the SBIR award positively affects follow-on venture capital financing, I did not observe any significant “halo” or “certification” effect of receiving an SBIR award on attracting external capital, regardless of the source of this external capital. The SBIR subsidy may obviate the need for external private capital; that is, as internal resources for R&D are released, SBIR recipients can redeploy these resources for marketing, production, and operations. Alternatively, the new firm may not have grown and expanded enough to warrant external capital infusion. These hypotheses can be further examined in future studies on the medium- and long-run certification effects of the SBIR.

Although I did not find a significant halo effect of SBIR on attracting external capital, I discovered a different certification effect of the program.¹⁵ Start-ups that received SBIR grants were more likely to attract external patents. This finding can indicate that the SBIR award certifies the quality of the company and the innovation project that the recipient is undertaking through the SBIR subsidy. As such, individuals, government laboratories, and companies that own patents may be more willing to license their knowledge assets to this group of small firms that they believe will be more successful in using their patents to produce innovations. The expectation that more successful use of these external patents by “certified” small firms translates to a steadier revenue stream of royalties by these patent holders makes SBIR grantees more attractive buyers of external patents. Another plausible explanation is that the SBIR has another value added to recipients: It expands the network of SBIR grantees. SBIR program administrators may introduce SBIR recipients to other innovators, making their collaboration more likely. This collaboration can include sharing knowledge assets.

That SBIR grantees are more likely to outsource complementary assets is thus an empirical evidence of the orchestration activities for innovation of SBIR recipients and the role played by SBIR to facilitate this process. This result is important for at least two reasons. First, it indicates that innovation requires a combination of internal and external knowledge assets. Competitive advantage may not lie in knowledge assets produced by private R&D investment and/or public R&D subsidy. It is now increasingly defined by the firm’s ability to orchestrate an internal–external portfolio of knowledge assets (Chesbrough, 2003; Teece, 1986, 2009). For a small business start-up that is specifically operating in the high-technology sector, assembling these internal and external knowledge assets may take precedence over acquiring or attracting external capital. In fact, it can be a precondition to external capital infusion. SBIR financing and networking allow recipient small firms to combine internal and external patents and produce innovation, making external capital infusion more likely in the future. But the first step is still orchestration for innovation, which SBIR apparently facilitates. Attracting external capital becomes necessary only after this orchestration activity.

Second, at the SBIR program level, it is not enough to measure R&D inputs and outputs. It is now very critical to look deeper into the orchestration activities of program grantees; that is, how they combine previous knowledge assets with new knowledge assets generated through the SBIR funding, and how they integrate both old and new internal assets with external knowledge assets to produce innovation. Future evaluations of the SBIR should take into account the complexity of the innovation process. New product prototypes that result from the public cofinancing of private R&D may not be enough to produce innovations that have high customer value added. The recipient’s ability to procure knowledge externally may be a positive effect of the public financing of commercial R&D; the public program may strengthen the grantee’s absorptive capacity to use external technologies. These long-run effects will benefit the local and national innovation economy in the long run, at least from an evolutionary economic perspective (Nelson & Winter, 1982).

Footnotes

Appendix

Appendix B

Variable Definitions: Outcomes.

Variables	Definition
R&D and innovation
R&D performance in 2008	Coded 1 if the start-up performed R&D in 2008, 0 otherwise
R&D expenditure in 2008	Amount of total R&D expenditure in 2008 (in US$)
Innovation propensity in 2009	Coded 1 if the start-up introduced new product or process in 2009, 0 otherwise
Licensing-out of patents in 2009	Coded 1 if the start-up licensed out own patent in 2009, 0 otherwise
Licensing-in of patents in 2009	Coded 1 if the start-up purchased a license to use external patent in 2009, 0 otherwise
Patent size in 2009	Number of patents that start-up had in 2009
R&D performance in 2009	Coded 1 if the start-up performed R&D in 2009, 0 otherwise
R&D expenditure in 2009	Amount of total R&D expenditure in 2009 (in US$)
External capital infusion
External capital—banks and nonbank in 2009	Coded 1 if the start-up obtained capital from a bank or nonbank financial institution in 2009, 0 otherwise
External capital—all sources in 2009	Coded 1 if the start-up obtained capital from a bank or nonbank financial institution, government agencies, family, friends, and other individuals in 2009, 0 otherwise
Employment, sales, and profit
Employment size 2009	Number of employees the start-up had in 2009
Positive sales in 2009	Coded 1 if the start-up sold goods and services in 2009, 0 otherwise
International sales in 2009	Coded 1 if the start-up sold goods and services in the global market in 2009, 0 otherwise
Profit in 2009	Coded 1 if the start-up had a profit in 2009, 0 otherwise

Acknowledgements

The author is grateful to the Ewing Marion Kauffman Foundation for granting him access to the Kauffman Firm Survey data.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

Author Biography

Reynold V. Galope is an assistant professor in the College of Community Studies and Public Affairs at Metropolitan State University, St. Paul, Minnesota. He earned his PhD in public policy from Georgia State University and the Georgia Institute of Technology as a Fulbright Fellow/Scholar.

References

Acs

Z. J.

Audretsch

D. B.

(1990). Innovation and small firms. Cambridge: MIT Press.

Aerts

Schmidt

(2008). Two for the price of one? Additionality effects of R&D subsidies: A comparison between Flanders and Germany. Research Policy, 37, 806-822. doi:10.1016/j.respol.2008.01.011

Almeida

Kogut

(1997). The exploration of technological diversity and geographic localization in innovation: Start-up firms in the semiconductor industry. Small Business Economics, 9, 21-31. doi:10.1023/a:1007995512597

Arrow

K. J.

(1962). Economic welfare and the allocation of resources for invention. In Nelson

R. R.

(Ed.), The rate and direction of inventive activity (pp. 609-625). Princeton, NJ: Princeton University Press.

Audretsch

D. B.

Link

A. N.

Scott

J. T.

(2002). Public/private technology partnerships: Evaluating SBIR-supported research. Research Policy, 31, 145-158. doi:10.1016/S0048-7333(00)00158-X

Audretsch

D. B.

Weigand

(2002). The impact of the SBIR on creating entrepreneurial behavior. Economic Development Quarterly, 16, 32-38. doi:10.1177/089124240201600104

Auerswald

P. E.

Branscomb

L. M.

(2003). Valleys of death and Darwinian seas: Financing the invention to innovation transition in the United States. Journal of Technology Transfer, 28, 227-239. doi:10.1023/a:1024980525678

Berube

Mohnen

(2009). Are firms that receive R&D subsidies more innovative? Canadian Journal of Economics, 42, 206-225. doi:10.1111/j.1540-5982.2008.01505.x

Borrus

Stowsky

(1999). Technology policy and economic growth. In Branscomb

L. M.

Keller

J. H.

(Eds.), Investing in innovation: Creating a research and innovation policy that works (pp. 40-63). Cambridge: MIT Press.

10.

Branscomb

L. M.

Keller

(Eds.). (1999). Investing in innovation: Creating a research and innovation policy that works. Cambridge: MIT Press.

11.

Breitzman

Hicks

(2008). An analysis of small business patents by industry and firm size (Contract report to the Small Business Administration). Retrieved from http://www.imamidejo.si/resources/files/doc/analiza%20patentov%20v%20msp%20usa.pdf

12.

Cameron

A. C.

Trivedi

P. K.

(2005). Microeconometrics: Methods and applications. New York, NY: Cambridge University Press.

13.

Chesbrough

H. W.

(2003). Open innovation: The new imperative for creating and profiting from technology. Boston, MA: Harvard Business School Press.

14.

Clarysse

Wright

Mustar

(2009). Behavioural additionality of R&D subsidies: A learning perspective. Research Policy, 38, 1517-1533. doi:10.1016/j.respol.2009.09.003

15.

Collier

Brady

Seawright

(2004). Sources of leverage in causal inference: Toward an alternative view of methodology. In Brady

Collier

(Eds.), Rethinking social inquiry: Diverse tools, shared standards (pp. 229-266). Oxford, England: Rowman & Littlefield.

16.

Cooper

R. S.

(2003). Purpose and performance of the Small Business Innovation Research (SBIR) program. Small Business Economics, 20, 137-151. doi:10.1023/A:1022212015154

17.

Cozzens

S. E.

Melkers

J. E.

(1997). Use and usefulness of performance measurement in state science and technology programs. Policy Studies Journal, 25, 425-435. doi:10.1111/j.1541-0072.1997.tb00032.x

18.

Czarnitzki

Licht

(2006). Additionality of public R&D grants in a transition economy. Economics of Transition, 14, 101-131.

19.

David

P. A.

Hall

B. H.

Toole

A. A.

(2000). Is public R&D a complement or substitute for private R&D? A review of the econometric evidence. Research Policy, 29, 497-529.

20.

Dehejia

R. H.

Wahba

(1999). Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs. Journal of the American Statistical Association, 94, 1053-1062.

21.

Dehejia

R. H.

Wahba

(2002). Propensity score-matching methods for nonexperimental causal studies. Review of Economics and Statistics, 84, 151-161.

22.

Feldman

M. P.

Kogler

D. F.

(2008). The contribution of public entities to innovation and technological change. In Shane

(Ed.), Handbook of technology and innovation management (pp. 431-459). Chichester, England: John Wiley.

23.

Galope

R. V.

(2014). What types of start-ups receive funding from the Small Business Innovation Research (SBIR) program? Evidence from the Kauffman Firm Survey. Journal of Technology Management & Innovation, 9, 17-28. doi:10.4067/S0718-27242014000200002

24.

Gelman

Hill

(2007). Data analysis using regression and multilevel/hierarchical models. New York, NY: Cambridge University Press.

25.

Georghiou

(2002). Impact and additionality of innovation policy. In Boekholt

(Ed.), Innovation policy and sustainable development: Can public innovation incentives make a difference (pp. 57-66)? Brussels, Belgium: IWT Observatory.

26.

Geroski

(1995). Markets for technology: Knowledge, innovation and appropriability. In Stoneman

(Ed.), Handbook of the economics of innovation and technological change (pp. 90-132). Oxford, England: Blackwell.

27.

Glazerman

Levy

D. M.

Myers

(2003). Nonexperimental versus experimental estimates of earnings impacts. ANNALS of the American Academy of Political and Social Science, 589(1), 63-93. doi:10.1177/0002716203254879

28.

González

Pazó

(2008). Do public subsidies stimulate private R&D spending? Research Policy, 37, 371-389.

29.

Greenberg

D. H.

Michalopoulos

Robins

P. K.

(2006). Do experimental and nonexperimental evaluations give different answers about the effectiveness of government-funded training programs? Journal of Policy Analysis and Management, 25, 523-552. doi:10.1002/pam.20190

30.

Guo

Fraser

M. W.

(2010). Propensity score analysis: Statistical methods and applications. Thousand Oaks, CA: Sage.

31.

Heckman

J. J.

Ichimura

Todd

(1998). Matching as an econometric evaluation estimator. Review of Economic Studies, 65, 261-294.

32.

Heinrich

C. J.

(2007). Evidence-based policy and performance management: Challenges and prospects in two parallel movements. American Review of Public Administration, 37, 255-277. doi:10.1177/0275074007301957

33.

D. E.

Imai

King

Stuart

E. A.

(2007). Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Political Analysis, 15, 199-236.

34.

Holland

P. W.

(1986). Statistics and causal inference. Journal of the American Statistical Association, 81, 945-970.

35.

Hubbard

R. G.

(1998). Capital-market imperfections and investment. Journal of Economic Literature, 36, 193-225.

36.

Khandker

S. R.

Koolwal

G. B.

Samad

H. A.

(2010). Handbook on impact evaluation: Quantitative methods and practice. Washington, DC: World Bank.

37.

Klette

T. J.

Møen

Griliches

(2000). Do subsidies to commercial R&D reduce market failures? Microeconometric evaluation studies. Research Policy, 29, 471-495.

38.

Koga

(2005). R&D subsidy and self-financed R&D: The case of Japanese high-technology start-ups. Small Business Economics, 24, 53-62. doi:10.1007/s11187-005-3096-z

39.

Lee

E. Y.

Cin

B. C.

(2010). The effect of risk-sharing government subsidy on corporate R&D investment: Empirical evidence from Korea. Technological Forecasting and Social Change, 77, 881-890. doi:10.1016/j.techfore.2010.01.012

40.

Lerner

(1999). The government as venture capitalist: The long-run impact of the SBIR program. Journal of Business, 72, 285-318.

41.

Lewis

(1973). Causation. Journal of Philosophy, 70, 556-567.

42.

Link

A. N.

Scott

J. T.

(2000). Estimates of the social returns to Small Business Innovation Research projects. In Wessner

C. W.

(Ed.), The small business innovation research program. (pp. 275-290). Washington, DC: National Academy Press.

43.

Link

A. N.

Scott

J. T.

(2010). Government as entrepreneur: Evaluating the commercialization success of SBIR projects. Research Policy, 39, 589-601. doi:10.1016/j.respol.2010.02.006

44.

Metcalfe

(1995). The economic foundations of technology policy: Equilibrium and evolutionary perspectives. In Stoneman

(Ed.), Handbook of the economics of innovation and technological change (pp. 409-557). Oxford, England: Blackwell.

45.

Morgan

S. L.

Winship

(2007). Counterfactuals and causal inference: Methods and principles for social research. New York, NY: Cambridge University Press.

46.

Nelson

R. R.

Winter

S. G.

(1982). An evolutionary theory of economic change. Cambridge, MA: Harvard University Press.

47.

Özçelik

Taymaz

(2008). R&D support programs in developing countries: The Turkish experience. Research Policy, 37, 258-275.

48.

Pearl

(2009). Causality: Models, reasoning and interference (2nd ed.). New York, NY: Cambridge University Press.

49.

Robb

Reedy

E. J.

Ballou

DesRoches

Potter

Zhao

(2010). An overview of the Kauffman Firm Survey: Results from the 2004-2008. Retrieved from http://www.kauffman.org/~/media/kauffman_org/research%20reports%20and%20covers/2010/05/kfs_2010_report.pdf

50.

Rosenbaum

P. R.

(2005). Heterogeneity and causality: Unit heterogeneity and design sensitivity in observational studies. The American Statistician, 59, 147-152.

51.

Rosenbaum

P. R.

Rubin

D. B.

(1983). The central role of the propensity score in observational studies. Biometrika, 70, 41-55.

52.

Rubin

D. B.

(1974). Estimating causal effects of treatments in randomized and nonrandomized Studies. Journal of Educational Psychology, 66, 688-701.

53.

Sekhon

J. S.

(2008). The Neyman-Rubin model of causal inference and estimation via matching methods. In Box-Steffensmeier

J. M.

Brady

H. E.

Collier

(Eds.), The Oxford handbook of political methodology (pp. 271-299). Oxford, England: Oxford University Press.

54.

Shane

(2000). Prior knowledge and the discovery of entrepreneurial opportunities. Organization Science, 11, 448-469. doi:10.1287/orsc.11.4.448.14602

55.

Shapira

Kuhlmann

(2003). Learning from science and technology policy evaluation: Experiences from the United States and Europe. Cheltenham, Gloucestershire, England: Edward Elgar.

56.

Siegel

D. S.

Wessner

Binks

Lockett

(2003). Policies promoting innovation in small firms: Evidence from the US and UK. Small Business Economics, 20, 121-127.

57.

Tassey

(2007). The technology imperative. Cheltenham, Gloucestershire, England: Edward Elgar.

58.

Teece

D. J.

(1986). Profiting from technological innovation: Implications for integration, collaboration, licensing and public policy. Research Policy, 15, 285-305. doi:10.1016/0048-7333(86)90027-2

59.

Teece

D. J.

(2009). Dynamic capabilities and strategic management: Organizing for innovation and growth. New York, NY: Oxford University Press.

60.

Toole

A. A.

Czarnitzki

(2007). Biomedical academic entrepreneurship through the SBIR program. Journal of Economic Behavior & Organization, 63, 716-738.

61.

Toole

A. A.

Turvey

(2009). How does initial public financing influence private incentives for follow-on investment in early-stage technologies? Journal of Technology Transfer, 34, 43-58. doi:10.1007/s10961-007-9074-7

62.

Wallsten

S. J.

(2000). The effects of government-industry R&D programs on private R&D: The case of the Small Business Innovation Research program. RAND Journal of Economics, 31, 82-100.