Evaluation Capacity Building in the Context of Military Psychological Health

Abstract

The need for evaluation capacity building (ECB) in military psychological health is apparent in light of the proliferation of newly developed, yet untested programs coupled with the lack of internal evaluation expertise. This study addresses these deficiencies by utilizing Preskill and Boyle’s multidisciplinary ECB model within a post-traumatic stress disorder treatment program. This model outlines a theoretical framework, offers practical strategies, and emphasizes both context and culture, which are paramount in military health-care settings. This study found that the model provides a highly applicable ECB framework that includes ways to identify ECB objectives, tailor activities, and understand outcomes. While there was high utilization of ECB activities by program staff, there was misaligned evaluative thinking, which ultimately truncated sustainable evaluation practice. Based on this research, evaluators can better understand how to provide an ECB intervention in a complex cultural and political environment and assess its effectiveness.

Keywords

evaluation theory evaluation practice evaluation capacity building behavioral health

Evaluation capacity building (ECB) is essentially the design and implementation of strategies focused on increasing knowledge, skills, and beliefs to help individuals and organizations cultivate sustainable evaluation practices within their organizations. The literature defines factors that are building blocks for successful ECB in the private and nonprofit sectors, however little is known about other contexts such as the military sector wherein culture, norms, and resources are vastly different from the civilian sector. This study addresses this knowledge gap by empirically testing an existing multidisciplinary ECB model within the context of a post-traumatic stress disorder (PTSD) treatment program at an Army Military Treatment Facility. The main goals of this study were to design, implement, and assess the effectiveness of an ECB intervention in a military health-care setting utilizing a multidisciplinary ECB model created by Preskill and Boyle (2008). This model was utilized because it provides a theoretical framework, offers practical strategies, and emphasizes both context and culture, which are paramount in military health-care settings. Further, it has been tested successfully in nonmilitary domains.

Background and Significance

Military Psychological Treatment Rarely Evidence Based

The need for military ECB is apparent in light of the proliferation of newly developed, yet untested psychological health programs that have cropped up to meet the recent and growing demand. Over 2.7-million service members have returned from conflicts in Iraq and Afghanistan (Watson Institute International & Public Affairs, 2016). Most of those serving have coped exceedingly well under repeated deployments, high “personnel tempo” (which is short turnaround times between deployments), and high “operational tempo” (which is high workload during deployments). However, 20–25% of service members have reported psychological trauma with long-lasting medical, economic, and social impacts for themselves, their families, and society (Hoge et al., 2004; Schell & Marshall, 2008). To meet this need, the government has acted rapidly and implemented a variety of new treatment programs. However, the effectiveness of most of these programs remains untested and leadership decision-making is rarely evidence based (Weinich et al., 2011). We illustrate the development of an evidence base for military psychological health programs with a study that aims to increase evaluation capacity of the staff at a PTSD clinical treatment program.

Military Systems Lack Internal Evaluation Expertise

In 2011, RAND Corporation released a comprehensive descriptive report on psychological health programs funded by the Department of Defense. One of its five main foci of recommendations was capacity building to create an evidence base for formative and summative program evaluation and comparative effectiveness research (Weinich et al., 2011). Findings revealed a lack of systematic collection and analysis of process and outcomes data. Untested programs may offer overlapping or duplicated services, may be inefficient in terms of their utilization of resources, and may offer treatments that are ultimately shown to be ineffective for the treatment of psychological trauma. The opportunity costs may be huge if service members delay proper psychological treatment because they are getting care through an ineffective treatment method. The need for an evidence base coupled with the lack of internal evaluation expertise may be overcome via ECB. The body of literature on evaluation research presents potential solutions in ECB frameworks and models; applying these theoretical models would be an appropriate next step for expanding military personnel evaluation capabilities.

Theoretical Framework

Models of ECB

A series of conceptual ECB frameworks has been published over the last decade (Boyle, Lemaire, & Rist, 1999; King & Volkov, 2005; Milstein & Cotton, 2000; Nielsen, Lemire, & Skov, 2011; Preskill & Boyle, 2008; Taylor-Powell & Boyd, 2008). None of these models has been empirically tested in military settings. However, after conducting an examination of the literature on these models, one stood out as the best fit for ECB in the military context: Preskill and Boyle’s multidisciplinary model.

Boyle et al. (1999) report on lessons from practice by outlining an evaluation regime that includes “evaluation capacity,” practice, organizational arrangements, and the institutionalization of evaluation. Milstein and Cotton (2000) similarly focus on evaluation capacity (which is the precursor to ECB), and they define concepts for the American Evaluation Association presidential strand in which they spearheaded a movement to go beyond the individual to organizational capacity building. King and Volkov (2005) offer a comprehensive ECB model that goes into depth on organizational context and ECB structures and defines resources for ECB and reports on three case studies. Preskill and Boyle create and empirically test at 15 sites a comprehensive, multidisciplinary model that focuses on context, organizational learning capacity, building evaluation knowledge, skills, attitudes, transfer of learning and measures the sustainability of evaluation practice and its diffusion (Preskill & Boyle, 2008). Taylor-Powell and Boyd implement an ECB logic model in a case study evaluation of a complex education extension organization. Nielsen et al. (2011) develop and validate an instrument to measure ECB.

Of these models, Preskill and Boyle’s multidisciplinary model offers the most guidance for implementation of ECB, and it has been empirically tested.

Multidisciplinary Model Applied to the Military Setting

Referring to Preskill and Boyle’s model (Figure 1), the overarching domains (leadership, culture, systems, communication) set the stage for unfolding the context of the evaluation. Authors suggest asking key initial questions such as “Does leadership value learning and evaluation?” “Is there a culture of inquiry?” “Are systems and structures in place to support evaluation?” and “How is evaluation communicated and are there channels for dissemination?” The left circle of the diagram includes prescribed objectives and a variety of ECB activities that guide initiation, planning, design, and implementation of ECB efforts. The right circle outlines the processes, practices, policies, and resources for sustainable evaluation practice. The middle arrow emphasizes the importance of the transfer of learning for successfully sustained practice.

Figure 1.

Preskill & Boyle’s (2008) multidisciplinary model of evaluation capacity building.

Preskill and Boyle’s model is most useful in a military setting because of the emphasis placed on leadership, cultural context, systems, and structures; because of its practical offerings of ECB activities; and because it focuses on beliefs and attitudes toward evaluation. Cultural context is key because military organizational hierarchy is rigidly structured and constitutes a unique type of top-down organizational structure (Soeters, Winslow, & Weibull, 2006). Further, the focus on sustainable evaluation practices and the model’s offering of many different approaches for capacity building activities and learning techniques has the most practical application of theories found in the literature.

Preskill and Boyle’s multidisciplinary model is distinctive because it emphasizes the culture of evaluation and promotes the belief that evaluation is beneficial. The existence of this type of shared belief between commanders and project managers is the crux of successful evaluation work in military settings. Commanders are increasingly being tasked with reviewing impact outcomes for resourcing and remission of programs. Yet, they are hard-pressed to find these data. In the context of military health, the bulk of what normally constitute evaluation consists of monitoring data on utilization of services. For instance, the military health system currently uses work relative value units (RVU) to measure productivity of its providers. Most often, high RVU procedures (e.g., surgery) constitute successful programs and low RVU services (e.g., psychological counseling) raise a red flag for a program to be examined for remission. This type of productivity monitoring offers no gauge of the effectiveness of programs. The gap has created an opportunity for growing evaluative thinking and for increasing resources allocated to outcomes evaluation for internal evaluation efforts. ECB activities will help leadership and staff understand the difference between outputs (RVU production) and outcomes (psychological health-related gains), among other activities. The objective of this study was to offer and evaluate an ECB intervention in order to grow the evidence base of psychological health treatment programs in the Army via three aims: (1) offer tailored ECB activities based on needs assessment, (2) create an outcomes data collection tool and repository, and (3) provide an analysis tool for continuous performance improvement.

Study Context

The Warrior Combat Stress Reset Program (RESET) at Carl R. Darnell Army Medical Center, Fort Hood, TX, is an untested PTSD program that began operating in August 2008. Its goal is to provide innovative, integrative care for service members for the treatment of combat stress–related PTSD. During a 3-week, intensive outpatient treatment period, soldiers participate in activities which include traditional approaches (i.e., group and individual counseling, coping skills training) and alternative approaches (i.e., biofeedback, cranial electrical stimulation, therapeutic massage, Reiki, acupuncture) as well as 8 weeks of follow-up counseling. Since its inception, the RESET program has treated over 700 soldiers on which data have been systematically collected via hard copy survey. While the RESET program director began to collect data, the program team lacked sufficient skills and support for program evaluation and had no provisions to establish an ongoing monitoring system of its efforts.

Research Design and Method

The research utilized a case study methodology to examine how ECB works in a military setting with a community of stakeholders (Yin, 1984). Case studies are particularly effective for discovering the key factors that facilitate and inhibit desired outcomes and for understanding the processes and mechanisms through which these factors interact (Patton, 1990). Case studies provide in-depth information about how a program works within the larger social and organizational contexts in which they are embedded (Jinnett, Coulter, & Koegel, 2002), which is a key for studying capacity building in the military setting. Within this community of stakeholders, each is presumed to have a distinct interest in the evaluation process, so it is not unusual to find them in conflict with one another (Gargani, 2011). Thus, all stakeholders must be included in the analysis. To this end, the approach is supported by the elements of ethnography and stakeholder analysis in that the informants are observed within their cultural context, all stakeholders have been interviewed, and the meanings they ascribe to topics are considered to be important variables in the study (Spradley, 1979).

Since the study’s feasibility is contingent upon partnership in the military setting, we considered stakeholder buy in to be essential during the initial design phase. It is a requirement for any third-party research to be hosted by a site principal investigator. The research reflects the interest and support of the program director, program staff, and commanders. Further, this work was funded by the Department of Defense U.S. Army Medical Research and Material Command.

Logic Modeling ECB Intervention

In order to operationalize the research methods described herein and to gain buy in from leadership, we used a program theory-driven logic model of the ECB intervention (Donaldson, 2007). This tool is a snapshot of the ECB plan, which maps this study’s inputs for structure, the implementation components, and the measures and objectives for evaluating the ECB intervention. See Figure 2 for the RESET ECB logic model.

Figure 2.

Evaluation capacity building intervention logic model.

Needs Assessment

Central to Preskill and Boyle’s model is that the evaluator understand the needs and wants of the stakeholders (Preskill & Boyle, 2008). Without a sense of the objectives valued by stakeholders, it is not possible to design strategies. For these reasons, a needs assessment was included as one of the main aims of the ECB intervention (see Figure 2, logic model). It was conducted with five-key stakeholders, namely, the program director, a program provider, an office staff person, and a psych technician who conducts patient intakes. The needs assessment uncovered the underlying assumptions, motivations, and expectations that the key stakeholders have for the ECB intervention, which are described in the Results section. Second, the needs assessment elucidated the types of outcomes that were valued and the strategies that were deemed worthwhile and feasible from the perspective of the program staff and the evaluator.

Participants: Sample Selection Size and Characteristics

Sample selection procedures were based on purposeful sampling. Criteria for inclusion and exclusion of informants were their interest in and/or need for evaluation. If informants did not have a stake in the evaluation of this program, they were not included (Johnson, 1990). This purposeful sampling strategy is most efficient for the small number of people involved with the program.

Although there are no strict rules for determining sample size for qualitative studies, they typically include 30 or fewer respondents (Morse, 1994; Patton, 1990). The study sample size was 55 participants including RESET program’s key leadership at the Military Treatment Facility including the behavioral health director, the outpatient director, and the hospital commander (n = 3), program administrators in the hospital (n = 4), patients (n = 24), and their family members (n = 24). The upstream stakeholders are the Military Treatment Facility leadership, who are the intended users of these key evaluation findings and capacity building efforts (Davidson, 2005). The midstream stakeholders are the recipients of the capacity building trainings, which include program staff. The downstream stakeholders include active duty service members currently participating in the RESET program and their respective family members.

Guiding Questions and Data Collection

Questions were tailored for each stakeholder group (leaders, administrators, treatment providers, patients, and family) to focus on the issues around evaluation that are salient to them and to provide data organized around domains of knowledge, skills/behaviors, and affective outcomes. Since the Preskill and Boyle model does not dictate a set of measures for these constructs, we created questions aimed to operationalize the ECB aims into qualitative measures. The objectives outlined in the model are reworked into questions and tailored for the various stakeholders in the military setting. Questions we asked of stakeholder groups included the following:

Is the organizational leadership supportive of evaluation and its use?

What are the organizational policies, procedures, rules, and norms regarding evaluation implementation and use?

Are there resources provided for evaluation (i.e., monetary, staffing, IT, and analysis)?

Do you know if there are specific organizational plans for practicing evaluation and for building evaluation capacity at the hospital?

Does leadership mandate or incentivize to do evaluation?

Are you asked to access and use any evaluation resources such as data from other programs when making decisions about the current resourcing here?

What are program director and staff attitudes toward evaluation? Are they supportive of evaluation?

Is acquisition of new skills and knowledge related to evaluation and performance improvement encouraged?

What made you decide that RESET was the best treatment option for you?

Would you want to know how other soldiers did after coming to the program?

Data were collected using multiple methods: interviews, organizational data, field notes, and observational notes (Miller & Crabtree, 1992; Yin, Bateman, & Moore, 1983) to achieve triangulation (Patton, 1990; Tashakkori & Teddlie, 1998; Yin, 1984). Data were drawn from multiple points of view to reduce the influence of any single respondent (Edgerton & Langness, 1974). The qualitative interviews were conducted face to face and digitally recorded. Interview audiotapes were transcribed, and the validity of the transcription process was checked by comparing tapes to transcripts.

Interviews were conducted at baseline and 6 months postintervention using a semistructured interview protocol. The interview questionnaires started with open-ended questions followed up by probes (Bernard, 2006). The interview questions were drawn from models in the literature and focused on respondents’ backgrounds, their attitudes, knowledge, and activities related to evaluation; the organizational culture related to evaluation; and the types of institutional programs and resources dedicated to evaluation processes.

Administrative and organizational data were collected in the form of organizational charts, briefings, program materials, and conference presentation slides that included evaluation results. These data support the structural component of a program evaluation and can provide context on the program and insights into the evaluation work that staff has conducted.

Data Analysis Plan

Our qualitative data analysis was based on grounded theory (Glaser & Strauss, 1967) and guided by the model, which defines three overarching domains with various objectives related to each one as displayed in Figure 1:

Knowledge objectives

Participants understand evaluation terms and concepts

Participants understand evaluation’s contribution to decision-making

Participants understand strengths and weaknesses of different evaluation approaches and of different data collection methods

Participants understand stakeholders roles and involvement

Participants understand relationships among a program’s goals, objectives, activities, and expected outcomes

Skills/behaviors objectives

Participants engage in logic modeling

Participants develop evaluation questions, data collection instruments, analyze data, and interpret results

Participants teach others

Affective objectives

Participants believe that evaluation can be useful, positive, and incorporated into program design

Participants believe that evaluation contributes to program’s success, brings value, and is worth the time and money

These objectives provided the framework for analysis of the ECB approach and the strategies that were utilized to meet these objectives were also informed by the Preskill and Boyle model. From the descriptions provided by respondents, we identified patterns, concepts, and themes. To help synthesize the large amount of qualitative data this study yielded, we uploaded interview transcripts into a computer-based text analysis software designed specifically for narrative interview and/or field notes (Fielding & Lee, 1991; Muhr, 2010; Pfaffenberger, 1988). This technique allowed for the easy retrieval of text segments. We conducted an interrater reliability exercise which produced a Fleiss’ κ average score of .89 which is an acceptable level for interrater reliability (Fleiss, 1981). After establishing interrater reliability of the coders using κ measures with the University of Pittsburgh Coding Analysis Toolkit, all transcripts were coded and reviewed for common themes (MacQueen, McLellan-Lemal, Bartholow, & Miltstein, 2008; University of Pittsburgh, 2012).

Ethical Protections of Human Subjects

The protection of human subjects is especially important with populations like military service members who are mandated to comply with commander orders. This work gained approval at several levels for the protection of human subjects during research including institutional review boards at Claremont Graduate University and Department of the Army Brooke Army Medical Center and second-level military review by Human Research Protection Office of the U.S. Army Medical Research and Materiel Command.

Findings

From Needs Assessment to Actions

The program director, staff, providers, and evaluation team developed activities in response to the aforementioned needs assessment. The ECB activities were selected and tailored to the program needs because they were not only valued by stakeholders but also feasible and from the standpoint of the external evaluation consultants’ time and skillset.

Involvement in an evaluation process: The program director and staff participated in the design and implementation of the program evaluation. At each step in the process, the evaluation team elicited input from the program leadership and staff including selecting quantitative health outcomes and qualitative interview probes.

Training: The evaluation team provided classes/workshops on evaluation methods (e.g., logic modeling, survey tools, and qualitative data analysis) to the program staff.

Technical assistance: The evaluation team supported the program by developing a data repository and statistical reporting tool that would allow the program to continue to self-monitor after the withdrawal of the evaluation team.

Written materials: The program staff was provided with a variety of written documents and texts to learn about evaluation principles and practices.

Mentoring/coaching: The evaluation team built an ongoing relationship with program staff to provide individualized technical and professional support.

Meetings: Program staff allocated time and space to discuss evaluation activities specifically for the purpose of learning from and about evaluation. This commitment is notable as the time spent with evaluators was beyond the stated scope of their job responsibilities.

Results Regarding Evaluation Practice

Stakeholder groups report varying insights on their ECB activities. We also discuss infrastructure supports to sustain evaluation practice.

Clinic providers

Evaluative thinking is high for these individuals. They have been collecting satisfaction data and clinical chart outcomes pre-/posttreatment since program inception as mandated by their program director. They utilized these data for process quality-of-care improvement and for clinical reasons, such as to tailor the treatments to each participant. They reported that their capacity for analyzing these data with the purpose of outcomes evaluation has been low because of resourcing (data entry labor, data analysis expertise). One provider said:

Well, I gather this was a good thought to collect data from the word go, when this program began. Because this program I think is the only program that we have a good record of the data of everything that is happening here.

Participants

Participants had heard about the program’s success through word of mouth, which seemed to be the strongest influence for them to sign up for RESET. Most participants interviewed mentioned they would have found information helpful from previous participants on satisfaction and health outcomes when they initially made the decision to seek care. The general feeling was that “… success rate and how other people felt afterward, yes, I would have liked to been exposed to that information.”

A few RESET patients felt that this kind of PTSD program is specialized and that outcomes from other program participants would not be helpful, but this sentiment was expressed by the minority.

Family members

Family members were asked if they would want to know information on how other participants did after coming to the program. Many felt that it would have been helpful and encouraging to know that it was efficacious for others. More pointedly, they wished for information on long-term effects experienced by service members who attended the program and whether these positive impacts were sustained, “It would help me to understand and know if this program really works for people, how many people, and it’s not just a short time benefit.”

Administrative staff

According to administrative personnel, the extent of evaluation at this installation involves monitoring productivity through program use and throughput. Administrators track billable hours (called resource value units or RVUs), which show that productivity is very high. The RVUs produced from the RESET program give it continuously high marks with hospital administrators. There is an existing hospital-wide monitoring system called the interactive customer evaluation, in which self-reported complaints or compliments are submitted via paper cards in boxes. However, these data are not systematically analyzed. Hospital administrators expressed the desire for program staff to collect data for comparative analysis, funding, and resource allocation.

Leadership

A top-tier Medical Command memo suggested programs be analyzed to prove worth and be data driven, supporting evaluation Armywide. However, while evaluation is supported by leaders, it is not resourced. Program directors and staff take it upon themselves to collect data beyond clinical charting. Otherwise what is referred to as evaluation is essentially monitoring through productivity data (RVUs) as mentioned previously.

Infrastructure

The main aims of the ECB intervention were to resource data collection and analysis for increasing the evidence base of psychological health treatment programs and to enable continuous performance improvement. The outcomes measures were refined to include current versions of validated PTSD measures and reintegration measures as well as including additional measures on posttraumatic growth. A data repository was built using Excel, and statistical formulas were programmed so that staff would have a sustained evaluation tool after the ECB intervention ended. Staff was trained in data entry and use of analysis tools. At 6-month postintervention, the staff was using tools continuously. However, issues around resourcing data entry were arising as front office staffing was cut back, limiting the amount of time allowed for data entry. The program director was able to use outcomes data to present at conferences and to leadership. The reputation of the program grew and other military hospitals sent leaders to tour the program’s facility and meet with the program director for possible replication.

Leadership wants to use data for decisions on program expansion, replication, and remission. But the data they seek for decision-making is not resourced. This job is left solely up to program directors and staff to create and maintain evaluation systems. There are no command-directed outcomes or satisfaction data mandated or incentivized (which is the normal channel in the military context for initiating and sustaining reporting systems). Other clinical program directors from local and distant military installations are contacting RESET to ask how they set up their data systems. At the time of the evaluation, the RESET program director was conducting tours of the clinic for these interested parties on an almost monthly basis.

Implications for Capacity Building

Preskill and Boyle note that ECB involves the design and implementation of teaching and learning strategies to help individuals, groups, and organizations learn about what constitutes effective, useful, and professional evaluation practice. They go on to state that the main goal is sustainable evaluation practice. To be considered sustainable, organizations provide leadership support, incentives, resources, and opportunities to transfer their learning about evaluation to their everyday work as well as to the development of systems, processes, policies, and plans that help embed evaluation work into the way the organization accomplishes its strategic mission and goals (p. 444).

Overall, we consider this a successful ECB intervention and application of the Preskill and Boyle model in a military setting. The model offers a clear framework for this effort including ways to identify objectives, specific ECB activities, and ways to understand findings. There was a high utilization of ECB activities within program staff stakeholders. The RESET program shifted its evaluation policies and procedures to incorporate strategies learned during the training, technical assistance, and mentoring activities (i.e., institutionalized evaluation policies and procedures, shared evaluation beliefs and commitment, and strategic plan for evaluation). The success of the capacity building effort was based on highly motivated program director and staff supportive of evaluation.

The model’s recognition of the importance of context and culture was especially relevant in this Military Treatment Facility where the hierarchical leadership structure, high turnover of administrative staff, and general staffing shortages presented unique challenges.

Before the ECB intervention, program staff regularly used clinical case conferences to inform treatment planning and make changes to program activities based on informal feedback from patients and clinical acumen. Now they are able to use retrospective patient outcomes to make data-driven decisions regarding treatment modalities (i.e., use of evaluation findings). The evaluation consultants left the treatment staff with a data management system that can accommodate either paper and pencil data or direct electronic data capture and can analyze data with just a few clicks at multiple time points (pre- and posttreatment and 8- and 12-week follow-ups; i.e., integrated knowledge management evaluation system).

While most of the goals of this ECB effort were realized, it fell short with regard to some intended outcomes derived from the multidisciplinary model, specifically on the right side of the model under “sustainable evaluation practices” related to top-level organizational support of the program’s evaluation efforts. Many of these shortcomings were due to a misalignment in evaluative thinking across stakeholders. Specifically, there was disconnect between program staff and the leadership who do not hold shared beliefs and commitment to evaluation (i.e., organization-wide evaluation frameworks and processes). The organization was not a fertile ground for evaluation sustainability when applying the Preskill and Boyle model definition. It’s no surprise that there are challenges with the sustainability of evaluation practice when there is misaligned evaluation orientations. The usefulness of Preskill and Boyle’s ECB model in Military Health Facility application lies mainly in the ECB strategies defined in the left side of the model graphic. The sustainability side of the graphic fell short in this setting.

Limited resources around staffing and continuing education hindered sustainability. Shortly after the evaluation team withdrew from the setting, there were cuts to administrative staff who were responsible for data entry and follow-up data collection (i.e., sustained resources dedicated to evaluation). Clinical staff does not have an evaluation mandate and are not authorized to use regular work hours for these activities. Although there was interest on the part of the program director to attend conferences and gain further training in evaluation, military-wide cuts to nonessential travel made this unfeasible (i.e., continuous learning about evaluation).

A critical challenge to this effort was the inability to include command leadership in planning and implementation of the EBC activities. Although we were given permission to work with program staff, evaluation activities were limited to that specific program and did not extend out to the entire behavioral health department or to the rest of the hospital. While next steps would be to work with command leadership, this study has raised issues not addressed in the ECB model. If an organization is not supportive of resourcing evaluation activities, but the program staff is highly motivated to do these activities, there may still be some positive ECB outcomes that can be achieved and sustained.

Footnotes

Authors’ Note

The views, opinions, and/or findings contained in this report are those of the author(s) and should not be construed as an official Department of the Army position, policy, or decision unless so designated by other documentation. In the conduct of research where humans are the subjects, the investigator(s) adhered to the policies regarding the protection of human subjects as prescribed by Code of Federal Regulations (CFR) Title 45, Volume 1, Part 46; Title 32, Chapter 1, Part 219; and Title 21, Chapter 1, Part 50 (Protection of Human Subjects).

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by support from the U.S. Army Medical Research and Materiel Command under Award No. W81XWH-08-1-0408.

References

Bernard

H. R.

(2006). Research methods in anthropology: Qualitative and quantitative approaches (4th ed.). Lanham, MD: AltaMira Press.

Boyle

Lemaire

Rist

R. C.

(1999). Introduction: Building evaluation capacity. In Boyle

Lemaire

(Eds.), Building effective evaluation capacity: Lessons from practice (pp. 1–19). New Brunswick, NJ: Transaction.

Davidson

E. J.

(2005). Evaluation methodology basics. Thousand Oaks, CA: Sage.

Donaldson

S. I.

(2007). Program theory-driven evaluation science: Strategies and applications. Mahwah, NJ: Erlbaum.

Edgerton

R. B.

Langness

(1974). Methods and styles in the study of culture. San Francisco, CA: Chandler & Sharp.

Fielding

N. G.

Lee

R. M.

(1991). Using computers in qualitative research. Newbury Park, CA: Sage.

Fleiss

J. L.

(1981). Statistical methods for rates and proportions (2nd ed.). New York, NY: John Wiley.

Gargani

(2011). Theories of value and value of theories: A theory-based evaluation perspective. Paper presented at the American Evaluation Association Conference, Anaheim, CA.

Glaser

B. G.

Strauss

(1967). The discovery of grounded theory: Strategies for qualitative research. New York, NY: Aldine.

10.

Hoge

Castro

Messer

McGurk

Cotting

Koffman

(2004). Combat duty in Iraq and Afghanistan, mental health problems and barriers to care. New England Journal of Medicine, 351, 13–22.

11.

Jinnett

Coulter

Koegel

(2002). Cases, contexts, and care: The need for grounded network analysis. Social Networks and Health, 8, 101–110.

12.

Johnson

(1990). Selecting ethnographic informants. Thousand Oaks, CA: Sage.

13.

King

J. A.

Volkov

(2005). A framework for building evaluation capacity based on the experiences of three organizations. CURA Reporter, 35(3), 10–16.

14.

MacQueen

McLellan-Lemal

Bartholow

Miltstein

(2008). Team-based codebook development: Structure, process, and agreement. In Guest

MacQueen

(Eds.), Handbook for team-based qualitative research (pp. 119–136). Lanham, MD: AltaMira Press.

15.

Miller

W. L.

Crabtree

B. F.

(1992). Primary care research: A multimethod typology and qualitative road map. In Crabtree

B. F.

Miller

W. L.

(Eds.), Doing qualitative research (pp. 17–28). Newberry Park, CA: Sage.

16.

Milstein

Cotton

. (2000). Defining concepts for the presidential strand on building evaluation capacity. Proceedings from American Evaluation Association Conference. Retrieved from http://www.eval.org/p/cm/ld/fid=137

17.

Morse

J. M.

(1994). Designing funded qualitative research. In Denzin

N. K.

Lincoln

Y. S.

(Eds.), Handbook of qualitative research (pp. 220–235). Thousand Oaks, CA: Sage.

18.

Muhr

(2010). ATLAS.ti: The qualitative data analysis & research software (Version V6.0) [Software]: ATLAS.ti scientific software development GmbH. Retrieved from http://www.atlasti.com/

19.

Nielsen

S. B.

Lemire

Skov

(2011). Measuring evaluation capacity—Results and implications of a Danish study. American Journal of Evaluation, 32, 324–344. doi:10.1177/1098214010396075

20.

Patton

M. Q.

(1990). Qualitative evaluation and research methods. Newbury Park, CA: Sage.

21.

Pfaffenberger

(1988). Microcomputer applications in qualitative research. Newbery Park, CA: Sage.

22.

Preskill

Boyle

(2008). A multidisciplinary model of evaluation capacity building. American Journal of Evaluation, 29, 443–459.

23.

Schell

Marshall

(2008). Survey of individuals previously deployed for OEF/OIF. In Tanielian

Jaycox

(Eds.), Invisible wounds of war: Psychological and cognitive injuries, their consequences, and services to assist recovery (pp. 87–113). Santa Monica, CA: RAND Corporation.

24.

Soeters

Winslow

Weibull

(2006). Military culture. In Caforio

(Ed.), Handbook of the sociology of the military (pp. 237–254). New York, NY: Kluwer Academic/Plenum.

25.

Spradley

(1979). The ethnographic interview. New York, NY: Holt, Rinehart and Winston.

26.

Tashakkori

Teddlie

(1998). Mixed methodology: Combining qualitative and quantitative approaches. Thousand Oaks, CA: Sage.

27.

Taylor-Powell

Boyd

H. H.

(2008). Evaluation capacity building in complex organizations. In Braverman

M. T.

Engle

Arnold

M. E.

Rennekamp

R. A.

(Eds.), Program evaluation in a complex organizational system: Lessons from Cooperative Extension. New Directions for Evaluation, 120, 55–69.

28.

University of Pittsburgh. (2012). Coding analysis toolkit. Retrieved from http://cat.ucsur.pitt.edu/

29.

Watson Institute International & Public Affairs. (2016). Costs of war: US veterans and military families. Retrieved from http://watson.brown.edu/costsofwar/costs/human/veterans

30.

Weinich

Beckjord

Famer

Marin

Gillen

Acosta

J.,

… Scharf

(2011). Programs addressing psychological health and traumatic brain injury among U.S. military servicemembers and their families. TR-950-OSD. Santa Monica, CA: RAND Corporation.

31.

Yin

R. K.

(1984). Case study research: Design and methods. Thousand Oaks, CA: Sage.

32.

Yin

R. K.

Bateman

P. G.

Moore

G. B.

(1983). Case studies and organizational innovation: Strengthening the connection. Washington, DC: Cosmos.