Systems Thinking Tools as Applied to Community-Based Participatory Research

Abstract

Community-based participatory research (CBPR) is being used increasingly to address health disparities and complex health issues. The authors propose that CBPR can benefit from a systems science framework to represent the complex and dynamic characteristics of a community and identify intervention points and potential “tipping points.” Systems science refers to a field of study that posits a holistic framework that is focused on component parts of a system in the context of relationships with each other and with other systems. Systems thinking tools can assist in intervention planning by allowing all CBPR stakeholders to visualize how community factors are interrelated and by potentially identifying the most salient intervention points. To demonstrate the potential utility of systems science tools in CBPR, the authors show the use of causal loop diagrams by a community coalition engaged in CBPR activities regarding youth drinking reduction and prevention.

Keywords

alcohol and substance abuse coalitions community-based participatory research systems science

Community-based participatory research (CBPR) is a collaborative research approach that requires the participation of community stakeholders and academic partners to address health issues (Minkler, 2005; Viswanathan et al., 2004; Wallerstein & Duran, 2010). In CBPR, researchers and community stakeholders fully participate in all aspects of the research process. CBPR has emerged as an important public health strategy given the need to facilitate (a) knowledge transfer between the academy and the community and (b) changes in the underlying social determinants, structures, and institutions that inhibit optimal health and well-being (Viswanathan et al., 2004). CBPR is increasingly being used to address complex health problems and is a promising approach for addressing health disparities in medically underserved, vulnerable populations in which engaging community members is necessary to gaining an understanding of the root causes of health disparities (Meade, Menard, Luque, Martinez-Tyson, & Gwede, 2009). CBPR methods have been successful in mobilizing communities to address health problems (Minkler, 2005). Additionally, CBPR has become a driving force in translational research and intervention science (Lindamer et al., 2009). CBPR serves to improve the relevance and stability of intervention and translational research by “engaging community stakeholders in adaptation within complex systems of organizational and cultural context and knowledge” and by sustaining programs through integration with existing community structures (Wallerstein & Duran, 2010, p. S41).

We propose that CBPR can benefit from a systems science framework to (a) visualize and specify the complex and dynamic characteristics of problems faced by community residents and (b) identify intervention points and potential “tipping points,” or points at which a community can change from one phase (disproportionate burden of disease) to another (lower burden of disease; Meadows, 1999). Systems science refers to a field of study that posits a holistic framework focused on component parts of a system in the context of relationships with each other and with other systems. A system can be considered group of interacting and interdependent components (e.g., community members or community based organizations) that form a complex and unified whole (a community; Cabrera, Colosi, & Lobdell, 2008).

Systems science frameworks have been presented as an approach to improve the practice of public health (Livingood et al., 2011) and political and social science (Green, 2006; Hirsh, Levine, & Miller, 2007; Homer & Hirsch, 2006; Leischow et al., 2008; Mabry, Marcus, Clark, Leischow, & Méndez, 2010; Mabry, Olster, Morgan, & Abrams, 2008; Milstein, 2008; Resnicow & Page, 2008; Trochim, Cabrera, Milstein, Gallagher, & Leischow, 2006) as well as enhance the study of organization and neighborhood functioning (Dinno, 2008; McDaniel, 1999; Olney, 2005).

Systems science is a field of study comprising many methods and approaches. Specifically, we propose complex adaptive systems (CAS) as an organizing framework for CBPR.

A CAS is a dynamic network of many heterogeneous agents (e.g., people, families, organizations/businesses) that interact freely and in ways that are nonlinear (inputs not necessarily proportional to outputs) and not necessarily predictable (Anderson, Crabtree, McDaniel, & Steele, 2005; McDaniel, 1999). The overall behavior of the CAS is the result of continual interacting decisions of individual agents (Holland, 2006).

Within the field of systems science, two very different ideas explain how one should approach the modeling of a CAS. The top-down approach focuses on modeling a system by dividing it into its main components and modeling the interactions of the components. Bottom-up modeling focuses on individual agents that compose the system and their interactions, exemplified by grass roots and group-based efforts in which decisions and changes are based on the efforts of a group or multiple agents (Osgood, 2009; Rahmandad & Sterman, 2008).

We posit that CBPR is inherently confluent with a bottom-up approach through improving the quality of connections among diverse agents and by assisting community leaders to create positive self-organization from the bottom-up. Self-organization is a process of evolution in which the development of new, complex structures takes place.

While the bottom-down approach is confluent with CBPR and is useful for examining how interactions among heterogeneous agents produce emergent phenomena, top-down approaches can be used complementarily to examine behavioral phenomena identified using bottom-up approaches or to initially specify the main components of the system.

To illustrate the use of systems thinking in a CBPR framework, we present a case study of a community coalition that addresses problem drinking among adolescents and other community health issues.

The Study Setting

In July of 2006, the Pennsylvania Department of Health launched a State Health Improvement Plan (SHIP) with an overall mission to “increase the years and quality of life for all Pennsylvanians and to eliminate health disparities.” To achieve these goals, SHIP aims to empower communities to address health issues by creating partnerships across multiple state and local entities and to “shift the mode of community health planning from a prescriptive model to a shared responsibility” (PA Department of Health, 2006). One of these SHIP-affiliated partnerships is the Clinton County Healthy Communities Community Health Outreach Coalition (CCHC), which is a partnership between local county-level agencies and Clinton County Cooperative Extension, an affiliate of The Pennsylvania State University. The CCHC community is a semirural, mountainous woodlands region with a population of approximately 37,500. The CCHC coordinates community agencies to address issues related to substance abuse, teen pregnancy, and the social determinants of health. The group consists of a volunteer board that includes a president, vice president, secretary, and treasurer. Subcommittees also meet on a regular basis and report to the board. The subcommittees are the following: prenatal care, early care and education, tobacco and other drugs, diabetes, and childhood safety. CCHC meets on a monthly basis. The CCHC board uses Pennsylvania Department of Health data, national health data trends, Health People 2010/2020 objectives, and local and statewide needs assessments to determine priority areas.

How Can Systems Thinking Tools Be Applied to CBPR?

We suggest that systems thinking tools, which assist in the visualization and the understanding of how a CAS is structured and how it behaves over time, can complement the current CBPR methods toolkit. Systems thinking tools serve to describe CAS from a broad perspective by visualizing overall system structures, patterns, and feedback loops. Current CBPR tools, such as logic models, community mapping, surveys, and ethnography, are useful in identifying community resources, program planning, and partnership building (Minkler, 2005). Systems thinking tools can be added to the CBPR toolkit in order to assist in intervention and project planning by allowing all stakeholders to visualize how community factors are interrelated and by potentially identifying the most salient intervention points. They can graphically depict a group’s understanding of a system’s structure and behavior, aid communication with others about one’s understanding of the system components, and assist in the design of high-leverage interventions for problematic system behavior (Kim, 1999). These tools can also be implemented so that they align with the documented CBPR principles, including “promoting co-learning and capacity building among all partners . . . [involving] systems development through a cyclical and iterative process . . . [recognizing] community as a unit of identity . . . [and facilitating] collaborative, equitable partnerships in all phases of the research” (Israel et al., 2003, pp. 55-58). To illustrate the utility of systems thinking tools and CBPR, we describe in Figure 1 the use of a specific tool, a causal loop diagram (CLD), to address adolescent drinking problem in rural Pennsylvania.

Figure 1.

CHCC causal loop diagram exercise

Causal loop diagrams were developed within the industrial science movement of the 1960s when scientists began to argue for a shift in how organizational cause and effect relationships were perceived. Forrester (1968) recognized that within the system of an organization, treating decisions and their outcomes as simply linear relationships (referred to by Forrester as open-loop assumptions where variables are considered independent and feedback does not exist) is unreasonable. Although this may simplify analysis, in reality, the inputs for any process are in many cases affected through some sort of feedback mechanism in a closed-loop (Richardson, 1986).

Causal loop diagrams account for this possibility of feedback within a system by providing a visual representation of the most salient variables and how these variables are related to one another. Furthermore, these diagrams include descriptions of the recognized connections with regard to the hypothesized impact of changes from one variable on another. This tool may be used as a starting point for model conceptualization to help portray a system and identify specific relationships of interest, or the CLD may be used after many relationships have already been examined in-depth as a means of connecting variables and summarizing the results of numerous tests (Forrester, 1994).

Researchers are increasingly using CLDs in the public health realm and other related fields in scenarios, such as customs policies (Cavana & Mares, 2004), to examine the effects of neighborhood greenspace, programs on mental health (Dinno, 2008), evaluations of disability demonstration programs (Fredericks, Deegan, & Carman, 2008), education reform (Hirsch et al., 2007), emergency department usage (Lane, Monefeldt, & Rosenhead, 1998), and explanations of feedback in relation to public health policy (Sterman, 2006).

Causal Loop Diagram Example

The CLD in Figure 1 depicts (some of) the relevant factors related to underage drinking among high school students in a semirural community. As part of the coalition’s desire to improve program effectiveness and evaluation, CCHC academic and community partners seek new and different approaches to program planning. The CCHC chose to address problem drinking among high school students in its target area because making improvements and finding effective strategies to address problem drinking rates among high school students have been challenging. Information sources included primarily CCHC practice-based knowledge and public health data sources such as substance abuse committee member knowledge, knowledge from practicing CCHC alcohol abuse prevention program managers, knowledge of the effect of past CCHC programming related to problem drinking, CCHC member knowledge of the social determinants of health, and public health data specific to the target county. Prior to the exercise, academic and community CHCC partners established the following boundaries and discussion points: (a) drinking among high school students, (b) a timeframe since the inception of the coalition in 2006, and (c) both mutable and immutable social, behavioral, contextual, and programmatic issues. Otherwise, the CCHC had not been exposed previously to systems thinking tools. The CLD exercise was conducted during a regular CHCC monthly meeting.

The CLD was originally drawn by hand real-time on a large tablet in front of the participating CCHC members and then recreated using Vensim software for later documentation. Each connection (line with arrow) between two factors has a corresponding sign (“+” or “−”) that indicates the hypothesized causal link. Connections with a “+” sign indicate that whichever direction the input variable changes, increase/adds to or decrease/subtracts from, the output variable will change in the same direction. For example, as the level of denial of the underage drinking problem increases, the level of underage drinking increases. Connections labeled with a “−” sign indicate that whichever direction the input variable should change, the output variable will change in the opposite direction. These connections form feedback loops.

Feedback loops are labeled either R_x or B_x. These letters indicate whether the loop is a reinforcing loop or a balancing loop. A reinforcing feedback loop contains an even number of negative causal links (negative signs). For example, in R₁, denial of the drinking problem increases (same “+”), leading to further normative acceptance. Hence, the reinforcing loop reinforces and generates behaviors of growth. On the other hand, balancing loops are feedback loops that contain an odd number of negative causal links (negative signs) and tend to stabilize behaviors over time (Richardson, 1991).

The CLD created is flawed, and the hypothesized variables and links were not exhausted within the set bounds. For example, one could hypothesize additional connections, such as (a) underage drinking leading to additional delinquent behavior that encourages greater parental supervision or (b) alcohol-related injury or fatality that reverses community denial and eventually affects community norms that tolerate youth drinking. Finally, no balancing loops exist in the CLD as it is has been currently generated.

Additionally, more intermediary variables are necessary to explain the connection between “Parent Drinking” and “Access to College Events” and “Youth Access” and “Parent Drinking.” Other links between variables could also be hypothesized and created. These errors are likely due to several factors, including that participants had inadequate training to conduct such an exercise and that CLDs can represent a set of flawed mental models since participants’ mental models may be flawed.

CLDs can carry some inhere1nt problems if care is not taken in labeling loop polarity. A CLD does not distinguish between (Richardson, 1997) variables that accumulate/decrease in number (e.g., stocks), such as the number of children who have a drinking problem, versus variables that represent change in a rate (e.g., flows), such as rate of access to alcohol-based activities. Coalition participants debriefed and discussed the CLD process and experiences immediately after completing the exercise.

Prior to the CLD exercise, the public health coalition focused mainly on increasing the availability of dry activities. Now, the coalition will consider other mutable constructs. However, some factors presented in the CLD are not mutable. For example, the mountainous geography that encourages “hidden” drinking is not mutable. However, approaches that increase parental supervision or counteract lack of parental supervision may be employed, as it seems to be a factor that was not considered in past programming. For example, before the CLD exercise, the logic model would have been centered on the creation of alternative alcohol-free activities, whereas now discussions of adolescent drinking prevention activities would consider other factors such as parents to increase supervision. Therefore, the creation of a new logic model would guide the practical implementation of the program strategy.

Approximately 4 months after the CLD, additional semistructured interviews with coalition members indicated if any programmatic changes or decisions were related to the CLD exercise. We specifically focused our interviews on coalition members from three organizations that managed alcohol reduction programs pre- and post-CLD exercise. Participants unanimously indicated that they found value in the CLD exercise; however, the exercise would be most useful during program planning periods. Often, when coalition members choose to implement a preexisting/prepackaged community-based alcohol prevention program, difficulty centers on identifying ways to modify program components or create new components to address the newly identified target issue.

Discussion

Using systems thinking that can capture the complexities of community challenges and may help increase the effectiveness and sustainability of programs and interventions. Successful community development hinges on focusing on the relationship between people and institutions; fostering participation, responsiveness, and accountability; and continually learning about outcomes while maintaining conditions for success (Gaventa, 2002).

Systems thinking and planning tools can help increase understanding of how factors in the community are interrelated and select a mutable factor or factors that have the greatest potential to move the outcome. In our case, the CLD exercise enabled the coalition to see beyond its prior emphasis on providing alcohol-free events and instead led the coalition to identify a wider set of intervention options, including a focus on parental supervision, community norms, and college events. However, given the limited experience and exposure to systems science tools, the ability to use these tools to identify situations in which feedback may occur and undercut an intervention was not achieved.

As shown in the results section, the CCHC-created CLD is flawed. It lacks some relevant variables, and errors are present in the connection/loop structure. These flaws and issues would need to be addressed in order to obtain the benefits of systems thinking tools. Although the members of the CCHC coalition showed enthusiasm and quickly gained basic understanding of the concept of systems thinking and systems thinking tools, adequate training is necessary to properly use and benefit from these tools.

Systems thinking concepts and tools are not a replacement but a complement for other tools that focus on linear relationships (Trochim et al., 2006). Although these systems science tools have the potential to improve CBPR intervention outcomes and sustainability, a cost is incurred for proper training of CBPR partners. In addition to gaining access to appropriate systems science training, CBPR partners would have to weigh the cost of financial and human resources and training time against perceived benefits of conducting systems science informed interventions.

Given the known utility of CBPR for addressing health issues, especially health disparities (Israel et al., 2010), the emerging evidence of the utility of systems thinking approaches to public health (Leischow et al., 2008; Mabry et al., 2010) and the confluence between these two paradigms, making systems thinking tools and training more widely accessible to both academic and community CBPR stakeholders, may be fruitful.

Furthermore, these systems thinking tools can be used in accordance with the guiding principles of CBPR. For example, tools such as the CLD, if used correctly, can help inform recreation of a logic model based on newly decided intervention components. However, we also suggest that these tools are used at the beginning of the intervention or program planning process, as changing program components or strategies may not be feasible once significant investment has been made toward the current strategy.

Although systems thinking provides a means for creating dynamic hypotheses about systems (e.g., causes of health status/issues within a community), to test these hypotheses, and to conclude accurately about where the relative leverage for change lies, system dynamics simulation methods are needed. Causal loop diagrams may provide an initial basis for simulation modeling in which a variety of futures can be explored and leverage points can be more accurately identified. The system dynamics framework enables the use of computer software to simulate models of the study situation in order to address questions concerning the likely outcome of various alternative intervention strategies. The simulations can incorporate a variety of components to model a community, such as health care services, population dynamics, health beliefs of the population, access to alcohol, and community resources. Thus, a system dynamics approach could be a potentially valuable tool for CBPR in terms of identifying how resource allocation within one component will affect other components and desired outcomes.

Additionally, other tools are confluent with systems thinking and CBPR principles, including but not limited to Behavior Over Time graphs and concept mapping. Behavior over time graphs are similar to other basic graphs in that they have horizontal and vertical axes. These graphs are used to plot a behavior of interest (vertical axis) over time (horizontal axis) so that the behavior can be examined from a systems thinking perspective. These graphs have been and continue to be used across the public health discipline in the assessment of community health issues such as condom use and sexual behavior (Bentley et al., 1998) and adolescent smoking, physical activity, and food choices (Kelder, Perry, Klepp, & Lytle, 1994).

Concept mapping represents the process and graphical results of a group’s problem-solving efforts (Trochim et al., 2006). It has been successfully used as a participatory health tool (Burke et al., 2005). Concept mapping takes knowledge generated from a group and creates a graphical representation that can be used to identify the relationships among ideas statements generated by the group (Trochim, Cook, & Setze, 1994). This resulting concept map provides a structure to guide future actions or next steps. Concept mapping has been used as a tool to identify challenges in the implementation of systems thinking in health care and public health structures (Trochim et al., 2006; Trochim & Kane, 2005) and to assist in planning state-level health promotion activities (Trochim, Milstein, Wood, Jackson, & Pressler, 2004).

In sum, systems thinking tools could be a helpful addition to the CBPR toolkit. However, mechanisms for training CBPR stakeholders in the proper use would be necessary.

Footnotes

Acknowledgements

We would like to thank the Clinton County Healthy Communities Community Health Outreach Coalition for their partnership and leadership regarding the current study.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

References

Anderson

R. A.

Crabtree

B. F.

McDaniel

R. R.

Jr. Steele

D. J.

(2005). Case study research: The view from complexity science. Qualitative Health Research, 15, 669-685.

Bentley

M. E.

Bollinger

R. C.

Gangakhedkar

R. R.

Mehendale

S. M.

Shepherd

M. E.

Spratt

Thilikavahi

(1998). HIV testing and counseling among men attending sexually transmitted disease clinics in Pune, India: Changes in condom use and sexual behavior over time. AIDS, 12, 1869-1877.

Burke

J. G.

Gielen

A. C.

McDonnell

K. A.

O’Campo

Peak

G. L.

Trochim

W. M.

(2005). An introduction to concept mapping as a participatory public health research method. Qualitative Health Research, 10, 1392-1410.

Cabrera

Colosi

Lobdell

(2008). Systems thinking. Evaluation and Program Planning, 31, 299-310.

Cavana

R. Y.

Mares

E. D.

(2004). Integrating critical thinking and systems thinking: From premises to causal loop. System Dynamics Review, 20, 223-235.

Dinno

(2008). Loop analysis of causal feedback in epidemiology: An illustration relating to urban neighborhoods and resident depressive experiences. Social Science & Medicine, 66, 1448-1449.

Forrester

(1968). Industrial dynamics—After the first decade. Management Science, 14, 398-415.

Forrester

(1994). System dynamics, system thinking, and soft OR. System Dynamics Review, 10, 245-256.

Fredericks

K. A.

Deegan

Carman

J. G.

(2008). Using System dynamics as an evaluation tool. American Journal of Evaluation, 29, 251-267.

10.

Gaventa

(2002). Towards participatory local governance: Six propositions for discussion. Currents, 28, 29-35.

11.

Green

L. W.

(2006). Public health asks of systems science: To strengthen our evidence-based practice, can you help us get more practice-based evidence? American Journal of Public Health, 95, 406-409.

12.

Hirsh

G. B.

Levine

Miller

R. L.

(2007). Using system dynamics modeling to understand the impact of social change initiatives. American Journal of Community Psychology, 39, 239-253.

13.

Holland

J. H.

(2006). Studying complex adaptive systems. Journal of Systems Science and Complexity, 19(1), 1-8.

14.

Homer

J. B.

Hirsch

G. B.

(2006). System dynamics modeling for public health: Background and opportunities. American Journal of Public Health, 96, 452-458.

15.

Israel

B. A.

Coombe

C. M.

Cheezum

R. R.

Schulz

A. J.

McGranaghan

R. J.

Lichtenstein . . . Burris

(2010). Community-based participatory research: A capacity-building approach for policy advocacy aimed at eliminating health disparities. American Journal of Public Health, 100, 2094-2102.

16.

Israel

B. A.

Schulz

A. J.

Parker

E. A.

Becker

A. B.

Allen

A. J.

Guzman

J. R.

(2003). Critical issues in developing and following community-based participatory research principles. In Minkler

Wallerstein

(Eds.), Community-based participatory research for health (pp. 56-73). San Francisco, CA: Jossey-Bass.

17.

Kelder

S. H.

Perry

C. L.

Klepp

K. I.

Lytle

L. L.

(1994). Longitudinal tracking of adolescent smoking, physical activity, and food choice behaviors. American Journal of Public Health, 84, 1121-1126.

18.

Kim

D. H.

(1999). Introduction to systems thinking. Boston, MA: Pegasus.

19.

Lane

Monefeldt

Rosenhead

(1998). Emergency—but no accident: A system dynamics study of an accident and emergency department. OR Insight, 11(4), 2-9.

20.

Leischow

S. J.

Best

Trochim

W. M.

Clark

P. I.

Gallagher

R. S.

Marcus

S. E.

Matthews

(2008). Systems thinking to improve the public’s health. American Journal of Preventive Medicine, 35(Suppl. 2), S196-S203.

21.

Lindamer

L.A.

Lebowitz

Hough

R. L.

Garcia

Aguirre

Halpain

M. C.

Depp

Jeste

D. V.

(2009). Establishing an implementation network: Lessons learned from community-based participatory research. Implementation Science, 4(17). doi:10.1186/1748-5908-4-17

22.

Livingood

W. C.

Allegrante

J. P.

Airhihenbuwa

C. O.

Clark

N. M.

Windsor

R. C.

Zimmerman

M. A.

Green

L. W.

(2011). Applied social and behavioral science to address complex health problems. American Journal of Preventive Medicine, 41(5), 525-531.

23.

Mabry

P. L.

Marcus

S. E.

Clark

P. I.

Leischow

S. J.

Méndez

(2010). Systems science: A revolution in public health policy research. American Journal of Public Health, 100, 1161-1163.

24.

Mabry

P. L.

Olster

D. H.

Morgan

G. D.

Abrams

D. B.

(2008). Interdisciplinarity and systems science to improve population health: A view from the NIH Office of Behavioral and Social Sciences Research. American Journal of Preventive Medicine, 35(Suppl. 2), S211-S224.

25.

McDaniel

R. R.

(1999). A view from complexity science. Frontiers of Health Services Management, 16(1), 44-48.

26.

Meade

C. D.

Menard

J. M.

Luque

J. S.

Martinez-Tyson

Gwede

C. K.

(2009). Creating community-academic partnerships for cancer disparities research and health promotion. Health Promotion Practice, 12, 456-462.

27.

Meadows

D. H.

(1999). Leverage points: Places to intervene in a system. Retrieved from http://www.sustainabilityinstitute.org/pubs/Leverage_Points.pdf

28.

Milstein

(2008). Hygeia’s constellation: Navigating health futures in a dynamic and democratic world. Retrieved from http://www.cdc.gov/syndemics/monograph

29.

Minkler

(2005). Community-based research partnerships: Challenges and opportunities. Journal of Urban Health, 82(Suppl. 2), 113-112.

30.

Olney

C. A.

(2005). Using evaluation to adapt health information outreach to the complex environments of community-based organizations. Journal of the Medical Library Association, 93(Suppl. 4), S57-S67.

31.

Osgood

(2009). Lightening the performance burden of individual-based models through dimensional analysis and scale modeling. System Dynamics Review, 25(2), 24.

32.

PA Department of Health. (2006). Pennsylvania State Health Improvement Plan. Harrisburg, PA: Author.

33.

Rahmandad

Sterman

(2008). Heterogeneity and network structure in the dynamics of diffusion: Comparing agent-based and differential equation models. Management Science, 54, 998-1014.

34.

Resnicow

Page

S. E.

(2008). Embracing chaos and complexity: A quantum change for public health. American Journal of Public Health, 98, 1382-1389.

35.

Richardson

G. P.

(1986). Problems with causal loop diagrams. System Dynamics Review, 2, 158-170.

36.

Richardson

G. P.

(1991). Feedback thought in social science and systems theory. Philadelphia: University of Pennsylvania Press.

37.

Richardson

G. P.

(1997). Problems in causal loop diagrams revisited. Systems Dynamics Review, 13, 247-252. doi: 10.1002/(SICI)1099-1727(199723)13:3<247::AID-SDR128>3.0.CO;2-9.

38.

Sterman

J. D.

(2006). Learning from evidence in a complex world. American Journal of Public Health, 96, 505-514.

39.

Trochim

W. M.

Cabrera

D. A.

Milstein

Gallagher

R. S.

Leischow

S. J.

(2006). Practical challenges of systems thinking and modeling in public health. American Journal of Public Health, 96, 538-546.

40.

Trochim

W. M.

Cook

J. A.

Setze

R. J.

(1994). Using concept mapping to develop a conceptual framework of staff’s views of a supported employment program for individuals with severe mental illness. Journal of Consulting and Clinical Psychology, 62, 766-775.

41.

Trochim

W. M.

Kane

(2005). Concept mapping: an introduction to structured conceptualization in health care. International Journal of Quality Health Care, 17, 187-191.

42.

Trochim

W. M.

Milstein

Wood

B. J.

Jackson

Pressler

(2004). Setting objectives for community and systems change: An application of concept mapping for planning a statewide health improvement initiative. Health Promotion Practice, 5(1), 8-19.

43.

Viswanathan

Ammerman

Eng

Gartlehner

Lohr

K. N.

Griffith

Whitener

(2004). Community-based participatory research: Assessing the evidence (Evidence Report/Technology Assessment No. 99; AHRQ Publication No. 04-E022-2). Rockville, MD: Agency for Healthcare Research and Quality.

44.

Wallerstein

Duran

(2010). Community-based participatory research contributions to intervention research: The intersection of science and practice to improve health equity. American Journal of Public Health, 100(Suppl. 1), S40-S46.