Building the Evidence Base for Population-Level Interventions

Abstract

Population-level interventions focused on policy, systems, and environmental change strategies are increasingly being used to affect and improve the health of populations. At the same time, emphasis on implementing evidence-based public health practices and programming is increasing, particularly at the federal level. Valuing strategies in the population health domain without the benefit of demonstrated efficacy through highly rigorous methods introduces an inherent tension between planning and acting on the best evidence available, waiting for more rigorous evidence to emerge, as well as exploring innovative ways to evaluate and model evidence-based strategies. This article describes the creation of a resource that helps public health practitioners use current evidence for strategic decision making while building the evidence base for population-level interventions. The resource addresses topics of current discussion in the field of evaluating population-level interventions, including the tension between internal and external validity, the need to include measures of health equity, and the balance between fidelity to the intervention and adaptation to the community context. The resource is intended to advance development of evidence in the field by providing practitioners, project managers, and evaluators with a practical guide for using, reviewing, and adding to the existing evidence base.

Keywords

community health evaluation health policy health disparities social determinants

Population-level interventions focused on policy, systems, and environmental change strategies (PSE) are increasingly being used to improve the health of populations. PSE strategies and population-level interventions implement positive changes at a community or state level, such as ensuring access to healthier living through tobacco-free work spaces, improving access to healthy and affordable food and beverages, and promoting the use of effective, well-enforced seat belt laws. These population-level interventions are generally undertaken by community partners responding to the unique needs and situations of a given community and address the ways in which environments can encourage healthful behaviors and discourage unhealthy ones (Centers for Disease Control and Prevention, 2009; National Association of County and City Health Officials, 2011). At the same time, emphasis on implementing evidence-based public health practices is increasing, particularly at the federal level. According to the 2012 Office of Management and Budget memo on promoting efficient spending,

Since taking office, the President has emphasized the need to use evidence and rigorous evaluation in budget, management, and policy decisions to make government work effectively. Where evidence is strong, we should act on it. Where evidence is suggestive, we should consider it. Where evidence is weak, we should build the knowledge to support better decisions in the future.

Public health practitioners are challenged to demonstrate what interventions work and why. However, population-level interventions focused on PSE strategies have unique features that can make it difficult to use traditional evaluation methods (Braveman, Egerter, Woolf, & Marks, 2011; Brownson, Royer, Ewing, & McBride, 2006; Lomas, 2005; Nutley, Powell, & Davis, 2012). Our goal in creating the resource is to help develop evidence in the field by guiding practitioners toward using, reviewing, and adding to the existing evidence base. This resource is a step among many toward creating pathways for practitioners to “build the evidence to support better decisions in the future” for population-level interventions.

While much work has been conducted to identify what works in population health, gaps remain in our understanding of how best to gather, evaluate, and disseminate evidence for population-level strategies. Extensive research has been conducted to examine which types of evidence are most appropriate for evidence-based public health, how to weigh the trade-off between more and less rigorous study designs, and how to evaluate the relative strengths of different research designs for different types of interventions (Black, 1996; Brownson, Fielding, & Maylahn, 2009; Carande-Kulis et al., 2000; Fielding & Briss, 2006; Glasgow & Emmons, 2007; Hawe, Shiell, & Riley, 2004; Livingood et al., 2011; Mays, Pope, & Popay, 2005; Mercer, DeVinney, Fine, Green, & Dougherty, 2007; Mindell et al., 2010; National Research Council of the National Academies, 2011; Petticrew, Chalabi, & Jones, 2012; Rychetnik, Frommer, Hawe, & Shiell, 2002; Schorr & Farrow, 2011; Secretary’s Advisory Committee on National Health Promotion and Disease Prevention Objectives for 2020, 2010; Silverman, 2009). Researchers rely on several methods and different types of evidence to evaluate population-level public health interventions, and many existing systematic reviews and evidence frameworks focus on intervention efficacy to rate the relative strength of interventions. This project revealed, however, that focusing on the efficacy of population-level interventions may not be sufficient to provide the evidence needed by practitioners in the field. Population-level interventions have some unique research questions specific to how they interact with elements of the complex community context in which they are implemented that require researchers to look for and evaluate different types of evidence. The development of clearly defined standards for evidence around population-level interventions, including the relative emphasis of internal and external validity in traditional randomized controlled trials (RCTs) and the need for contextual information, is a current area of research for several stakeholders in the evaluation and systematic review fields, including the Cochrane Review, the Centers for Disease Control and Prevention’s Community Preventive Services Task Force, and others. While this work is ongoing, this project sought to translate the current research in the field to help public health practitioners and decision makers use the current evidence for strategic decision making and simultaneously build the evidence base in a way that is more suited for population-level interventions.

Right now, the evidence for population-level interventions is growing as stakeholders work to address this gap, particularly for interventions focused on PSE strategies. The London-based Alliance for Useful Evidence describes the idea of an “evidence journey,” which promotes promising early findings as the basis for developing substantive evidence-based knowledge (Nutley et al., 2012). The framework presented by Spencer et al. (2013) moves that discussion forward by presenting criteria and processes for classifying and eventually valuing work that may be considered a “promising practice.” A need remains, however, to understand the strengths and challenges associated with implementing and evaluating population-level interventions. Public health practitioners need additional tools and approaches that best measure their success and challenges (Lieberman, Golden, & Earp, 2013). This project aimed to create a resource that examines how to identify and understand evidence specifically for population-level interventions. However, the implications are much wider, as this resource works toward helping communities make decisions in any area where the evidence is still emerging.

Method

This study involved an environmental scan of the literature, key informant discussions, expert panel input, and public comment. The environmental scan was developed as a reaction piece for the expert panel to respond to and was intended to give an overview of the current research on evidence and population-level interventions, discuss known gaps in the research, and suggest areas where a resource for the field might have the most impact. Peer-reviewed literature and other materials were identified using a list of key search terms in EBSCOhost (see the appendix for a list of key search terms). The study team examined the gray literature by searching federal websites and the publications of private stakeholders and research institutions such as the Robert Wood Johnson Foundation, the Centers for Disease Control and Prevention’s Community Preventive Services Task Force, the Institute of Medicine, and others. A search of works cited in relevant articles was conducted to identify additional resources. Concurrent with the environmental scan, discussions were conducted with 18 public health experts within the federal government, academic institutions, and private research institutions. Experts suggested additional resources and discussants, collecting information in a “snowball” sampling. More than 200 materials were reviewed, and 92 documents were included. From the discussions and the review of the materials, six general areas where research gaps existed were determined: the definitions of evidence for population-level strategies, the strengths and weaknesses associated with different research methodologies, the standards of evidence in existing systematic reviews and evidence frameworks, the context of population-level interventions, fidelity and adaptation, and translating research to practice. Materials were sorted by these six topics and then synthesized by the research team into a summary of the current research and the open questions in the field.

An expert panel of 13 members was recruited based on input from the U.S. Department of Health & Human Services, discussions, and prominent names in the literature. Panelists were chosen to represent a wide variety of areas of expertise and from federal, state, academic, and research institutions. Their areas of expertise included implementation science, program and policy evaluation, systematic review development, economics, epidemiology, and performance improvement, as well as experience in public health fields ranging from health disparities to early childhood development and chronic disease prevention. The panel reviewed the environmental scan and met three times to discuss research gaps and produce a resource useful to the intended audience. The draft resource was presented at the 2013 American Public Health Association conference and a website to gather public comment. In addition, 23 experts from the field were invited provide input. A complete list of references and panel members is available on the website for the Office of the Assistant Secretary for Planning and Evaluation.

Results

Findings From the Environmental Scan

The environmental scan found several gaps in the research on evidence-based public health and population-level interventions and documented ways in which community-based strategies differ in evidence needs. Criticism of the use of RCTs as the “gold standard” for evidence was noted by several authors for a multitude of reasons, including that the focus on RCTs prioritizes internal validity over external validity in a way that is less useful for population-level interventions (Black, 1996; Braveman et al., 2011; Durrheim & Reingold, 2010; Pawson, 2006; Sanson-Fisher, Bonevski, Green, & D’Este, 2007; Secretary’s Advisory Committee on National Health Promotion and Disease Prevention Objectives for 2020, 2010; Silverman, 2009). External validity has been discussed in the literature as a shortcoming of RCTs and systematic reviews for clinical interventions (Rothwell, 2005), and the environmental scan found that questions of external validity may be even more important for population-level interventions (Ahmad, Boutron, Dechartres, Durieux, & Ravaud, 2010; Black, 1996; Sanson-Fisher et al., 2007). Whether an intervention works depends not only on the program itself but also on community contextual factors that are often not reported or measured in existing systematic reviews and frameworks (Aarons, Hurlburt, & Horwitz, 2011; Braveman et al., 2011; Hawe et al., 2004; Institute of Medicine, 2012; Lomas, Culyer, McCutcheon, McAuley, & Law, 2005; Nutley et al., 2012). As a result, community-based stakeholders are left with insufficient information to determine whether a given intervention is likely to be effective, generalizable, or translatable to their own or other communities (Schorr & Farrow, 2011).

The environmental scan found that other research designs may be required when considering emerging fields, such as physical activity, in which causal links are less established (Brownson, Fielding, et al., 2009; Petticrew et al., 2012; Petticrew & Roberts, 2003; Schorr & Farrow, 2011). Research into practical evaluation designs intended to improve the translation of research to practice is a growing field, with the emergence of practice-based evidence and evaluation designs that are more applicable to the public health field (Ammerman, Smith, & Calancie, 2014). Additionally, we found that expensive and rigorous designs should be undertaken after less expensive and simpler designs have shown strong potential for an intervention or policy (Rychetnik et al., 2002).

Overall, the scan found that population-level interventions are highly context dependent, change behavior and health outcomes over long time frames, act in complex environments with other policies and factors at work simultaneously, and contend with real-world adaptations and circumstances (Brennan, Castro, Brownson, Claus, & Orleans, 2011; Brownson, Chriqui, & Stamatakis, 2009). Although these differences make evaluating these strategies challenging, it is important to expand the evidence base for these strategies so that practitioners and decision makers can make the most informed decisions possible to improve the health of their communities (Brownson, Fielding, et al., 2009).

Development of the Resource

Based on the results of the environmental scan and the expert panel, we found a need for a brief resource aimed at public health decision makers and practitioners outlining decision-making principles specific to population-level interventions focused on PSE strategies and approaches. The audience for this resource includes grantees of large federal or private grants looking for tools to decide whether and how to implement a population-level intervention, community and state public health practitioners looking for tools for working with communities on population-level interventions, and researchers and evaluators designing evaluations for population-level interventions. Funders, policy makers, and health administrators may also find the resource useful for communities undertaking these interventions, particularly health administrators working with their nonprofit hospitals to implement interventions as a result of their community health needs assessment. The resource answered two main questions:

How can one decide between possible population-level interventions based on the existing evidence and community contextual factors?

How can a population-level intervention be implemented and evaluated in a way that helps to build the evidence base for the intervention and related strategies?

The resource describes significant issues and considerations at each stage of the process, from reviewing the existing evidence base to implementation to evaluation for population-level interventions, noting where these interventions are unique at every step of the process. This simplified process is captured in Figure 1, which shows the key considerations at each stage of the process. The resource is intended to give decision makers and practitioners an overview of the three steps to ensure that evaluation, adaptation, and fidelity are considered concurrently, particularly for complex interventions (Hawe et al., 2004). While this resource does not address every step in implementing these interventions, it is intended to help practitioners and decision makers consider the specific implementation and evaluation needs for population-level interventions focused on PSE strategies and approaches.

Figure 1.

Population-level intervention decision diagram.

We found that decision makers and practitioners need guidance to decide among population-level interventions based on an assessment of their community’s needs and resources. The resource guides readers to the best available evidence in the form of systematic reviews and other evidence reviews and emphasizes looking for reviews that include contextual evidence, as well as a list of key contextual factors such as the following (Khavjou, Honeycutt, Hoerger, Trogdon, & Cash, 2014):

Individual community assets and resources

Organizational characteristics and partnerships

Community processes and resources

Community values and political will

Implementation costs and resources, including an understanding of cost and what factors affect the cost of an intervention in one community context versus another

Additionally, we found that our audience needed to consider implementation of population-level interventions within their community context, balancing the need for adaptation to their community’s resources with maintaining fidelity to the intervention. We also present resources for balancing fidelity with community adaptation, and stress the need to consider health equity and the how implementation affects different populations.

Finally, the ability to measure the outcomes of the interventions, taking into consideration the most appropriate evaluation methods and outcomes measures to ensure that new knowledge is translatable and useful, was found to be a large gap and of great importance to building the evidence base, particularly for interventions that are considered promising or emerging. The resource presents key evidence types and appropriate measures for population-level interventions and emphasizes the importance of comprehensive reporting to ensure that community contextual factors are explicitly included. While the literature is emerging to support practice-based evidence, there is still a gap in developing guidance, tools, and frameworks for evaluating and reporting on population-level interventions (Ammerman et al., 2014).

Discussion

The development of this resource and the input from both the environmental scan and the expert panel clearly show the need to develop the “evidence journey” for population-level interventions that includes rigorous methods needed for evidence-based public health (Nutley et al., 2012). However, there are methodological challenges to evaluating population-level interventions. Two major findings came from developing the resource. First, there is a lack of focus and intention around understanding and reporting on community context. Second, there is a need for more work around building health equity efforts into population-level interventions and understanding the positive and negative impacts of any population-level intervention on high-risk or vulnerable populations.

Defining, Measuring, and Reporting on Contextual Factors

The most important step in the evidence journey may be defining how to measure and report on the community context. Contextual factors are rarely reported, even in multilevel, community-wide interventions, and they may heavily affect the effectiveness and fidelity to population-level interventions. Given these issues, it is imperative for the public health field to work toward reporting and measuring contextual factors beyond demographics in a routine and systematic way (McDonald, 2013; Stange, Breslau, Dietrich, & Glasgow, 2012).

An open question is determining what contextual factors should be included in an evaluation of a community-based strategy (McDonald, 2013). The factors associated with program effects or impacts may vary between communities and between interventions, but there are broad categories of contextual factors that are useful for framing this discussion, as noted in the results above. Even as important community contextual factors are defined, there is also a need to incorporate this information without overreliance on nonexperimental designs. Community contextual factors need to be rigorously reported, measured, and evaluated.

Integrating Health Equity Into Evidence-Based Public Health

Another key consideration is the integration of health equity into the design of population-level interventions at all levels. Influencing health equity is important to both researchers and policy makers, but current evidence frameworks tend to not provide specific guidance on how to address equity concerns. Such concerns can be relevant when deciding among interventions, in part because of what Welch et al. describe as the “staircase effect,” in which the effectiveness of both clinical and public health interventions is reduced in disadvantaged populations due to low resources in the community and low adherence. Several effectiveness groups, including the United Kingdom’s National Institute for Health and Care Excellence, World Health Organization’s Commission on Social Determinants of Health, the Campbell Collaboration and Cochrane Collaboration, have proposed adding health equity considerations to their frameworks. Subgroup analyses are included in some systematic reviews, such as those conducted by the Community Preventive Services Task Force, where there is appropriate data available. However, Welch et al. (2012) found that few of the 224 systematic reviews include equity considerations in their work with subgroup analysis, assessing whether effects were different across groups. Given this major gap, further research should focus on the effects of public health interventions on improving health equity and reducing health disparities. Analyses of results should include subgroup analysis, or the use of counterfactual approaches for estimating causal effects as outlined by Glass, Goodman, Hernan, and Samet (2013).

There is need to develop measures and reporting mechanisms of community context and community potential for change in a systematic way to ensure that population-level interventions are promising for all populations, particularly those at high risk. Planning for population-level interventions should consider potential unintended outcomes on health disparities and health equity as well as be considered part of any evaluation of these interventions and strategies. Additionally, the integration of health equity considerations into intervention design and evaluation is dependent on advancing the study of contextual factors. Community-based participatory research has been documented as an appropriate method to address health equity and health disparities in program design and implementation, and the method can provide underserved or vulnerable populations an authentic influence in research and evaluation (Viswanathan et al., 2004; Wallerstein & Duran, 2010). Brownson, Fielding, et al. (2009) cite participatory approaches, which actively involve community members in the development, implementation, and evaluation of population-level strategies, as promising sources of evidence. At the same time, they note that these approaches, which emphasize community context, may present challenges in adhering to evidence-based practices (Brownson, Fielding, et al., 2009; Glasgow & Emmons, 2007). Studying the effectiveness of interventions is a cyclical process that involves translating interventions into new settings and exploring underlying theories and factors that had not been revealed in previous settings. For this reason, it is difficult to draw a distinct line between effectiveness research and translational research (Mercer et al., 2007). According to Wallerstein and Duran (2010), participatory approaches integrate “culturally based evidence, practice-based evidence, and indigenous research methodologies, which support community knowledge [that is] based on local explanatory models, and programs.” (p. S42).

Limitations

This study was exploratory in nature and has several limitations. First, the environmental scan conducted was not a true systematic review of the literature as it was intended to identify gaps for discussion of our expert panel. Second, the expert panelists recruited for this study brought strong expertise in multiple disciplines to the project, but the total number of experts was small. Additionally, while some practitioner-level feedback on the resource was gathered through roundtable discussions conducted at the 2014 National Network of Public Health Institutes meeting, the resource needs to be piloted by practitioners and decision makers to ensure its usefulness to the field. Last, the resource is meant to be a not final product but rather an exploration of factors that affect evidence-based public health and a continued effort for the federal government to engage with the field on complicated issues that can affect the success of population-level interventions.

Opportunities

This resource will be broadly disseminated by U.S. Department of Health & Human Services to the intended audience of state- and local-level public health practitioners and decision makers. Additionally, opportunities are being considered for piloting use of the resource to identify what population-level interventions to implement with appropriate evaluation planning and methods. Input from the field is necessary to take into account practical considerations for a framework intended to help review existing evidence through a community-based lens, make decisions about the inclusion of health equity and of adaptation and fidelity, and design practical and rigorous evaluations that illuminate the interactions between community context and the intervention. Our ultimate goal was to add to the dialogue about evidence-based public health and the need for alternate methods beyond the RCT to assess evidence for population-level interventions.

Footnotes

Appendix

Authors’ Note

The findings and conclusions of this article are those of the author(s) and do not necessarily represent the views of the Office of the Assistant Secretary for Planning and Evaluation or the U.S. Department of Health & Human Services.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Contribution was prepared by Altarum Institute under contract with the U.S. Department of Health & Human Services Office of the Assistant Secretary for Planning and Evaluation (HHSP23320095625WC).

Supplement Issue Note

This article is part of a Health Education & Behavior supplement, “The Evidence for Policy and Environmental Approaches to Promoting Health,” which was supported by a grant to the Society for Public Health Education (SOPHE) from the Robert Wood Johnson Foundation. The entire supplemental issue is open access at .

References

Aarons

Hurlburt

Horwitz

(2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health, 38(1), 4-23.

Ahmad

Boutron

Dechartres

Durieux

Ravaud

(2010). Applicability and generalisability of the results of systematic reviews to public health practice and policy: A systematic review. Trials, 1120-1128.

Ammerman

Smith

Calancie

(2014). Practice-based evidence in public health: Improving reach, relevance, and results. Annual Review of Public Health, 35, 47-63. doi:10.1146/annurev-publhealth-032013-182458

Black

(1996). Why we need observational studies to evaluate the effectiveness of health care. British Medical Journal, 312, 1215-1218.

Braveman

Egerter

Woolf

Marks

(2011). When do we know enough to recommend action on the social determinants of health? American Journal of Preventive Medicine, 40(1 Suppl. 1), S58-S66.

Brennan

Castro

Brownson

Claus

Orleans

(2011). Accelerating evidence reviews and broadening evidence standards to identify effective, promising, and emerging policy and environmental strategies for prevention of childhood obesity. Annual Review of Public Health, 32, 199-223.

Brownson

Chriqui

Stamatakis

(2009). Understanding evidence-based public health policy. American Journal of Public Health, 99, 1576-1583.

Brownson

Fielding

Maylahn

(2009). Evidence-based public health: A fundamental concept for public health practice. Annual Review of Public Health, 30, 175-201.

Brownson

R. C.

Royer

Ewing

McBride

T. D.

(2006). Researchers and policymakers: Travelers in parallel universes. American Journal of Preventive Medicine, 30, 164-172.

10.

Carande-Kulis

Maciosek

Briss

Teutsch

Zaza

Truman

. . .Fielding

(2000). Methods for systematic reviews of economic evaluations for the guide to community preventive services: Task Force on Community Preventive Services. American Journal of Preventive Medicine, 18(1 Suppl.), 75-91.

11.

Centers for Disease Control and Prevention. (2009). Chronic diseases: The power to prevent, the call to control: At a glance. Atlanta, GA: Author. Retrieved from http://www.cdc.gov/chronicdisease/resources/publications/aag/chronic.htm

12.

Durrheim

Reingold

(2010). Modifying the GRADE framework could benefit public health. Journal of Epidemiology and Community Health, 64, 387.

13.

Fielding

Briss

(2006). Promoting evidence-based public health policy: Can we have better evidence and more action? Health Affairs, 25, 969-978.

14.

Glasgow

R. E.

Emmons

K. M.

(2007). How can we increase translation of research into practice? Types of evidence needed. Annual Review of Public Health, 28, 413-433.

15.

Glass

T. A.

Goodman

S. N.

Hernán

M. A.

Samet

J. M.

(2013). Causal inference in public health. Annual Review of Public Health, 34, 61-75. doi:10.1146/annurev-publhealth-031811-124606

16.

Hawe

Shiell

Riley

(2004). Complex interventions: How “out of control” can a randomised controlled trial be? British Medical Journal, 328, 1561-1563.

17.

Institute of Medicine. (2012). An integrated framework for assessing the value of community-based prevention. Washington, DC: National Academies Press.

18.

Khavjou

Honeycutt

Hoerger

Trogdon

Cash

(2014). Collecting costs of community prevention programs: Communities putting prevention to work initiative. American Journal of Preventive Medicine, 47, 160-165. doi:10.1016/j.amepre.2014.02.014

19.

Lieberman

Golden

Earp

(2013). Structural approaches to health promotion: What do we need to know about policy and environmental change? Health Education & Behavior, 40, 520-525. doi:10.1177/1090198113503342

20.

Livingood

W. C.

Allegrante

J. P.

Airhihenbuwa

C. O.

Clark

N. M.

Windsor

R. C.

Zimmerman

M. A.

Green

L. W.

(2011). Applied social and behavioral science to address complex health problems. American Journal of Preventive Medicine, 41, 525-531. doi:10.1016/j.amepre.2011.07.021

21.

Lomas

(2005). Using research to inform healthcare managers’ and policy makers’ questions: From summative to interpretive synthesis. Healthcare Policy (Politiques De Santé), 1(1), 55-71.

22.

Lomas

Culyer

McCutcheon

McAuley

Law

(2005, May). Conceptualizing and combining evidence for health system guidance (Final report). Ottawa, Ontario, Canada: Canadian Health Services Research Foundation. Retrieved from http://www.cfhi-fcass.ca/migrated/pdf/insightAction/evidence_e.pdf

23.

Mays

Pope

Popay

(2005). Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field. Journal of Health Services Research & Policy, 10(Suppl. 1), 16-20.

24.

McDonald

(2013). Considering context in quality improvement interventions and implementation: Concepts, frameworks, and application. Academic Pediatrics, 13(6 Suppl.), S45-S53. doi:10.1016/j.acap.2013.04.013

25.

Mercer

DeVinney

Fine

Green

Dougherty

(2007). Study designs for effectiveness and translation research: Identifying trade-offs. American Journal of Preventive Medicine, 33, 139-154.

26.

Mindell

Biddulph

Taylor

Lock

Boaz

Joffe

Curtis

(2010). Improving the use of evidence in health impact assessment. Bulletin of the World Health Organization, 88, 543-550.

27.

National Association of County and City Health Officials. (2011). Healthy communities, healthy behaviors: Using policy, systems, and environmental change to combat chronic disease. Washington, DC: Author. Retrieved from http://www.naccho.org/topics/HPDP/mcah/upload/issuebrief_pse_webfinal.pdf

28.

National Research Council of the National Academies (2011). Improving health in the United States: The role of health impact assessment. Washington, DC: National Academies Press.

29.

Nutley

Powell

Davis

(2012). What counts as good evidence? Provocation paper for the alliance for useful evidence. London, England: Nestea. Retrieved from http://www.nesta.org.uk/library/documents/A4UEprovocationpaper2.pdf

30.

Office of Management and Budget. (2012). Memorandum to the heads of executive departments and agencies: Promoting efficient spending to support agency operations. Retrieved from http://www.whitehouse.gov/sites/default/files/omb/memoranda/2012/m-12-12.pdf

31.

Pawson

(2006). Evidence-based policy: A realist perspective. Thousand Oaks, CA: Sage.

32.

Petticrew

Chalabi

Jones

(2012). To RCT or not to RCT: Deciding when “more evidence is needed” for public health policy and practice. Journal of Epidemiology and Community Health, 66, 391-396.

33.

Petticrew

Roberts

(2003). Evidence, hierarchies, and typologies: Horses for courses. Journal of Epidemiology and Community Health, 57, 527-529.

34.

Rothwell

(2005). External validity of randomised controlled trials: To whom do the results of this trial apply? Lancet, 365, 82-93.

35.

Rychetnik

Frommer

Hawe

Shiell

(2002). Criteria for evaluating evidence on public health interventions. Journal of Epidemiology and Community Health, 56, 119-127.

36.

Sanson-Fisher

Bonevski

Green

D’Este

(2007). Limitations of the randomized controlled trial in evaluating population-based health interventions. American Journal of Preventive Medicine, 33, 155-161.

37.

Schorr

L. B.

Farrow

(2011). Expanding the evidence universe: Doing better by knowing more. For discussion at the Harold Richman Public Policy Symposium. New York, NY: Center for the Study of Social Policy. Retrieved from http://lisbethschorr.org/doc/ExpandingtheEvidenceUniverseRichmanSymposiumPaper.pdf

38.

Secretary’s Advisory Committee on National Health Promotion and Disease Prevention Objectives for 2020. (2010). Evidence-based clinical and public health: Generating and applying the evidence. Draft Report. Washington, DC: U.S. Department of Health & Human Services. Retrieved from http://www.elsevierbi.com/~/media/Images/Publications/Archive/The%20Gray%20Sheet/36/031/01360310010/080210_healthypeople_subcommitte_final_draft.pdf

39.

Silverman

S. L.

(2009). From randomized controlled trials to observational studies. American Journal of Medicine, 122, 114-120.

40.

Spencer

L. M.

Schooley

M. W.

Anderson

L. A.

Kochtitzky

C. S.

DeGroff

A. S.

Devlin

H. M.

Mercer

S. L.

(2013). Seeking best practices: A conceptual framework for planning and improving evidence-based practices. Preventing Chronic Disease, 10, 130186.

41.

Stange

Breslau

Dietrich

Glasgow

(2012). State-of-the-art and future directions in multilevel interventions across the cancer control continuum. Journal of the National Cancer Institute. Monographs, 2012(44), 20-31. doi:10.1093/jncimonographs/lgs006

42.

Viswanathan

Ammerman

Eng

Garlehner

Lohr

Griffith

. . .Whitener

(2004). Community-based participatory research: Assessing the evidence. Evidence Report/Technology Assessment (Summary), 99, 1-8.

43.

Welch

Petticrew

Ueffing

Jandu

Brand

Dhaliwal

. . . Tugwell

(2012). Does consideration and assessment of effects on health equity affect the conclusions of systematic reviews? A methodology study. PLOS ONE, 7(3), 1-6.

44.

Wallerstein

Duran

(2010). Community-based participatory research contributions to intervention research: the intersection of science and practice to improve health equity. American Journal of Public Health, 100(Suppl. 1), S40-S46.