Generating a Core Set of Outcomes for Hospital-Based Violence Intervention Programs

Abstract

Hospital-based violence intervention programs (HVIPs) engage individuals who have experienced violent victimization in postmedical care programming, with the goal of reducing the incidence and impact of future injuries. Although there is some empirical support for HVIPs’ impact on violence and crime-related outcomes, proper impact assessment is limited by a lack of systematized research on outcomes that relate to the proximal goals and activities of the programs themselves. To address this critical gap, we conducted a two-stage Delphi method to elicit and prioritize these outcomes using the wisdom and experience of those who are engaged in service delivery (i.e., HVIP community-based practitioners, program coordinators, and embedded researchers; N = 79). Through this process, respondents prioritized outcomes related to posttraumatic stress symptoms, beliefs about aggression, coping strategies, and emotional regulation, which have not been consistently measured using validated or standardized tools. Results suggest that, rather than limiting program outcomes to those related to repeat violent injury or criminality, hospital- and community-based violence prevention programs seek to improve and measure mental health and socioemotional outcomes as a benchmark for healing and recovery after a violent injury. Prioritization of these outcomes broadens the definition of recovery to include psychosocial health and well-being. In addition, inclusion of these outcomes in effectiveness studies will serve to bolster the relevance of findings, and provide support for continued development and refinement of HVIP practice.

Keywords

violence prevention Delphi method program evaluation

To maximize the impact and implications of results from clinical trials, appropriate outcomes must be specified a priori (see Sinha, Smyth, & Williamson, 2011, for a review). Failure to do so may lead researchers to prioritize outcomes that are most conveniently measured or of individual interest, rather than those of greatest importance to participants or practitioners. Similarly, publication bias may cause researchers to report only favorable outcomes, limiting reporting of null or negative findings (Chan, Hróbjartsson, Haahr, Gøtzsche, & Altman, 2004). Finally, a lack of systematic outcomes hinders the ability to share best practices among researchers, and is a significant barrier to meta-analyses that rely on similar outcomes across studies. Noting these limitations in their respective fields, several groups of researchers have established core outcome sets (e.g., Byrne et al., 2017, diabetes; Kelly et al., 2018, medication adherence; Shorter et al., 2017, brief alcohol interventions). However, researchers have not developed a core outcome set in the field of hospital-based violence prevention, despite the public health burden of such violence. Corso, Mercy, Simon, Finkelstein, and Miller (2007) estimated that interpersonal violence results in nearly US$37 billion per year in combined medical and employment costs, and further, interpersonal violence is among the leading causes of nonfatal injuries that require hospitalization (Centers for Disease Control and Prevention, 2016).

Hospitals treat violently injured patients for their physical wounds but are less likely to address the long-term mental and emotional toll that may result from these injuries. Although not outwardly visible, the psychological outcomes associated with violence-related injuries are numerous and include posttraumatic stress symptoms, depression, anxiety, and substance use (see Fowler, Tompsett, Braciszewski, Jacques-Tiura, & Baltes, 2009, for a review). As a practical matter, emergency health care providers often have limited time and access to the community resources necessary to attend to assault-injured victims’ nonmedical needs (Aiken, 2002). There is an urgent need for services to support recovery of violently injured individuals by promoting healing and safety.

In response to this need, the past two decades have seen the development and evolution of hospital-based violence intervention programs (HVIPs) that connect injured patients to much needed community-based services. HVIP teams—often comprised of social workers, intervention specialists, and/or peer mentors—meet injured individuals during or soon after hospitalization and collaborate with them and their families to identify postinjury goals and needs related to physical and psychological healing. The National Network of Hospital-Based Violence Intervention Programs (NNHVIP) has explicit criteria for membership, including programs that care for a minimum number of clients and supporting them in a community context for months after the violence event. The NNHVIP recognizes more than 30 HVIPs in the United States, United Kingdom, and Canada (NNHVIP, 2016), most located in urban, underserved communities in which violence is common (Richters & Martinez, 1993). Although each HVIP operates independently, the postassault recovery needs for patients are similar across programs, creating a distinct need to compare program outcomes to establish and share best practices (Juillard et al., 2016). At the time that this study was conducted, there were 22 existing NNHVIP members.

A few prior studies demonstrated that HVIPs show promise in decreasing repeat victimization and future criminal behavior (e.g., Cheng, Haynie et al., 2008; Cooper, Eslinger, & Stolley, 2006; Cunningham et al., 2012). However, the research base is limited due to its heavy focus on these types of outcomes, and also, because when there are assessments of other outcomes, the number and type of outcomes has varied widely between studies (Aboutanos et al., 2001; Cheng, Wright, Markakis, Copeland-Linder, & Menvielle, 2008; Cunningham et al., 2012; Zun, Downey, & Rosen, 2006). This hinders the ability to share best practices between programs and presents a significant barrier to meta-analyses that require measurement of similar outcomes across studies (Clarke, 2007; Sinha et al., 2011). When determining which outcomes would be important to assess, it is necessary to ensure that outcomes are relevant and meaningful to people most affected by the programs, salient to those involved in intervention delivery, and represent program priorities (Spoth et al., 2013).

The goal of this study was to develop a core set of person-centered outcomes that could be used across HVIPs to individually and collectively assess program impact. Applying a Delphi method (Delbecq, Van de Ven, & Gustafson, 1975), we iteratively surveyed a diverse group of HVIP staff members (e.g., community-based practitioners, program coordinators, and embedded researchers) to identify and prioritize meaningful outcomes that are likely to change through HVIP participation. Establishing a core set of outcomes would accelerate violence prevention research by enabling systematic and consistent multisite evaluations of HVIPs.

Method

Study Design

We used a Delphi survey method to develop a prioritized list of outcomes of relevance to HVIPs. The Delphi method is a multistage, iterative process designed to explore a group’s judgments or questions surrounding a topic, and to summarize those assessments (Delbecq et al., 1975). A unique benefit of the Delphi methodology is that it elicits expert opinion to develop consensus around a specific topic of interest. In addition, the Delphi method can be conducted via electronic surveys, which facilitates involvement of geographically diverse expert stakeholders. The Delphi method has been used previously to prioritize prevention strategies and identify research priorities (e.g., Blum et al., 2012).

Study Overview

During Spring 2014, we recruited 88 staff members from the 22 existing NNHVIP member programs to participate in a two-stage Delphi method. In Round 1, staff members were asked to identify outcomes they hope their clients will achieve through participation in their HVIP. Members of the study team coded responses via interround content analysis. In Round 2, respondents ranked the importance of each outcome identified in Round 1 relative to other outcomes. Participants did not receive compensation. This study was reviewed and deemed exempt by our hospital’s institutional review board.

Identification of Study Respondents

Investigators contacted the program directors of the 22 NNHVIP member programs to request their individual participation and that of three to five of their staff members. To synthesize perspectives of experts with diverse educational and occupational backgrounds, directors were encouraged to nominate individuals with a variety of professional experiences, specifically focusing on those who work directly with injured participants in HVIP programs. This process resulted in identification of 88 potential study respondents, as described in the “Results” section.

Delphi Round 1—Outcome Elicitation

Each of the 88 study respondents received an individual email that explained the project and included a link to an online survey in which they could free-list up to ten HVIP outcomes. For the purposes of this study, an outcome was defined as

youth or family factors that change as a result of HVIP participation . . . important outcomes may occur in many areas of your participants’ lives, such as improvements in their physical, mental, economic, or social functioning. When an outcome is achieved, changes can be seen in attitudes, behaviors, relationships, etc.

Interround Content Analysis

Using content analysis (Krippendorf, 1980), we grouped responses according to whether they were measured in previous research studies or whether they were novel ideas generated from the current Delphi process. As commonly done in Delphi methodology, responses that were unclear to investigators were presented to an expert panel for clarification (Delbecq et al., 1975; Rudolph et al., 2009). The expert panel consisted of senior NNHVIP staff (i.e., experienced case managers or case manager supervisors) from different programs. Following that, the research team used the repetition technique (Ryan & Bernard, 2003), in which members of the research team collaboratively examined the outcomes to discern that were most similar and occurred frequently in order. This methodology is used in qualitative research (Ryan & Bernard, 2003) to group thematically similar content (e.g., Hagaman & Wutich, 2017).

Delphi Round 2—Outcome Prioritization

Round 2 of the Delphi method was conducted to establish consensus regarding which outcomes are of greatest priority to HVIP staff members. All 88 respondents who were invited to participate in Round 1 were recruited via email to participate in Round 2. To determine the relative importance of each outcome, we conducted a maximum-difference scaling method using a forced-choice paradigm to establish highly ranked and lower ranked categories (Louviere, Lings, Islam, Gudergan, & Flynn, 2013) using Sawtooth Software (Sawtooth Software, Inc., 2014). Respondents reviewed 33 sets of five outcomes and were asked to “choose one outcome you think is most important and one outcome that you think is least important for HVIPs to address.” Outcomes were grouped together in sets randomly to reduce bias and maximize the chance an outcome is presented with as many other outcomes as possible. Each outcome was listed in three unique sets.

Analytic Plan

We compared each respondent’s selections across multiple sets using hierarchical Bayesian estimation to calculate the relative importance of each outcome as compared with the others (i.e., by determining individual-level preference scores for each respondent). This information is displayed as a linear score for each outcome, such that an outcome with a score of 10 would be twice as important as an outcome with a score of 5. Individual respondent data were averaged to generate aggregate-level preferences for each outcome, including a rank order priority and a quantitative assessment of the relative importance of each outcome as compared with the others. Thus, each mean represents a “relative importance value,” and the sum of the means of all outcomes equal 100.

Results

Identification of Study Respondents

Of the 88 staff members in NNHVIP nominated by program directors, 79 (89.7%) responded to Round 1, including at least one member from all 22 NNHVIP member programs. Differences in participation across programs was due primarily to the number of staff in each program. The majority of respondents were female (58.2%) and between the ages 31 and 50 years (63.3%). Twenty-eight (35.4%) respondents identified themselves as program directors or managers, 10 (12.7%) identified as program coordinators, 28 (35.4%) identified as case managers, nine (11.3%) identified as mental health clinicians, and four (5.0%) reported that they fulfilled other roles. Seventeen (21.5%) respondents reported that they had worked in the violence prevention field for fewer than 6 years, 13 (16.5%) reported that they had worked in the field for between 6 and 9 years, and 49 (62.0%) said that they had worked for 10 years or more in the field.

Round 1—Outcome Elicitation

The 79 Round 1 respondents generated a total of 572 responses (per respondent: range = 1-10, M = 7.24 responses, SD = 2.43 responses). We classified 194 responses (33.9%) into 18 outcomes that had been assessed in previous studies and 117 responses (20.4%) into 41 outcomes that had not been previously assessed. There were 136 responses (23.7%) for which the meaning was unclear to the research team (e.g., “tutorial”); thus, these outcomes were classified as “unknown.” Finally, 125 responses (21.9%) were classified as “process” outcomes, meaning the participant named an activity that demonstrates how care is delivered rather than the effect of that delivery on client health or functioning (e.g., “linking to mental health services”). The expert panel reviewed the responses that the research team had classified as either “unknown” or “process” and determined that, together, they represented 32 outcomes. Then, using the using the repetition technique (Ryan & Bernard, 2003), investigators consolidated redundant outcomes (e.g., self-worth and self-esteem) and eliminated conceptually broad outcomes (e.g., “resiliency”) that were more accurately represented by narrower outcomes, which were retained (e.g., “better coping strategies”). Because the goal of the study was to identify individual-level outcomes amenable to change through short-term HVIP participation, investigators also removed outcomes that were considered out of scope of HVIPs (e.g., affecting health and functioning of [nonparticipating] family members, reducing neighborhood crime rates). By the end of Round 1, 55 unique HVIP outcomes had been identified.

Round 2—Outcome Prioritization

Seventy-five of the 79 respondents from Round 1 (94.9%) completed Round 2. They prioritized the 55 outcomes by completing the maximum-difference forced-choice activity. Mean importance ratings for each outcome are shown in Table 1. Most of the highest priority outcomes (ranked 10th or higher) were violence related. They included reduced violence exposure (#4) and victimization (#1), fewer violence-related injuries (#2: hospital recidivism, #3: mortality, #5: posttraumatic stress symptoms), reduced risk for violent retaliation (#6), and decreased aggression (#9). All these outcomes have been considered in prior research on HVIPs. Notably, two of the “top 10” outcomes and many “second-tier” outcomes (ranked 11-20) reflected positive attributes, skills, or strengths that have not been included in prior research. These include better coping (#7) and emotion regulation/control (#10), enhanced social support from adult role models (#11) and family members (#18), improved future orientation (#14), and increased life satisfaction (#15).

Table 1.

Final List of Outcomes and Their Source, Presented in Order of Prioritization.

Outcome	M	SD	Prior Research	Delphi Results
1. Decreased violence victimization	4.21	1.60	X	X
2. Reduced hospital recidivism due to violent injury	4.17	1.68	X	X
3. Reduced participant mortality	4.02	2.01	X	X
4. Reduced exposure to violence	4.02	1.62	X	X
5. Decreased posttraumatic stress symptoms	3.99	1.47	X	X
6. Reduced risk for retaliation	3.53	1.58	X	X
7. Better coping strategies	3.44	1.38		X
8. Change in belief regarding need to retaliate	3.27	1.59	X	X
9. Decreased aggression	3.23	1.25	X	X
10. Improved emotional regulation and control	3.18	1.66		X
11. Establishing a relationship with a positive adult role model	2.99	1.68		X
12. Change in perceived efficacy of aggression	2.58	1.57	X	X
13. Increased conflict avoidance	2.48	1.45	X	X
14. Improved future orientation	2.44	1.64	X	X
15. Increased life satisfaction	2.20	1.46		X
16. Less gang involvement	2.15	1.66	X	X
17. Less involvement with antisocial peers	2.06	1.54	X
18. Improved sense of family belonging	2.02	1.56		X
19. Fewer arrests/convictions	2.02	1.57	X	X
20. Increased feelings of safety in neighborhood	1.94	1.38		X
21. Improved self-esteem	1.91	1.40		X
22. Improved peer relationships	1.87	1.31		X
23. Obtaining employment	1.83	1.71		X
24. Reduced suicidal risk	1.69	1.70		X
25. Reduced alcohol and substance use	1.68	1.31	X	X
26. Fewer instances of hospitalization	1.66	1.59	X	X
27. Enhanced beliefs regarding the importance of education and job training	1.57	1.22	X	X
28. Decreased depression symptoms	1.51	1.16		X
29. Increased feelings of safety at home	1.47	1.26		X
30. Improved employment self-efficacy	1.45	1.66		X
31. Improved mental health literacy	1.45	1.22		X
32. Improved financial stability	1.44	1.67		X
33. Decreased self-harm behaviors	1.40	1.45		X
34. High school graduation/obtaining GED	1.40	1.43		X
35. Reduced delinquency	1.31	1.13	X
36. Reduced rate of homelessness	1.28	1.48		X
37. Enrolling in and attending job/vocational training	1.25	1.37		X
38. Improved physical health	1.15	1.28		X
39. Increased happiness	1.12	1.40		X
40. Decreased anger	1.10	0.86		X
41. Improved social skills	1.07	0.89	X	X
42. Increased parenting knowledge and use of parenting skills	1.03	1.16		X
43. Increased community involvement and engagement	0.98	1.18		X
44. Improved academic performance	0.95	1.07		X
45. Completion of job/vocational training	0.88	1.29		X
46. Decreased anxiety	0.75	0.75		X
47. Receipt of victim of crime benefits	0.75	1.31		X
48. Greater compliance with medical follow-up	0.71	0.89	X	X
49. Reduced instances of prostitution	0.69	1.17		X
50. Increased feelings of safety at school	0.64	0.84		X
51. Improved school attendance	0.63	0.55	X	X
52. Increased adaptability	0.54	0.16		X
53. Improved sleep quality	0.46	0.81		X
54. Safer sex knowledge and practices	0.32	0.42		X
55. Improved body image	0.14	0.52		X

Note. X marks origin of outcome (i.e., X under “Delphi results” column means that the outcome was identified through this Delphi process. An X under “prior research” column means that the outcome was identified in published studies).

Discussion

By establishing consensus among “on-the-ground” stakeholders, this study offers a common set of individual-level outcomes for HVIPs and similar programs and alters the framework through which the effectiveness of these programs can be measured. In addition to commonly cited domains of violence exposure, injury, and aggression, HVIP interventionists and leaders identified mental health and coping, social support, and well-being as important indicators of HVIP success. The prioritization of positive outcomes by those who work most closely with injured individuals broadens the definition of recovery to include both reducing violence exposure and its negative sequelae and establishing and strengthening assets that support healing.

Notably, 17 of the top 25 most highly prioritized outcomes relate to psychosocial health, suggesting the importance of socioemotional factors in healthy postinjury functioning. For example, learning effective coping strategies and using healthier emotional regulation skills can better position program participants to avoid responding aggressively to provocations (Sullivan, Helms, Kliewer, & Goodman, 2010) and decrease the likelihood of future violent victimization (Lynch & Cicchetti, 1998). Increased life satisfaction and improved future orientation not only contribute to autonomy, general contentment, and self-respect but also are key elements of one’s ability to make positive decisions that render participants less likely to engage in violent behaviors (Stoddard, Zimmerman, & Bauermeister, 2011; Tay & Diener, 2011). In addition, posttraumatic stress symptoms and perceived efficacy of aggression are critical mediating factors in the cycle of violence (Farrell et al., 2008; Ozkol, Zucker, & Spinazzola, 2011). For example, if HVIPs are effective in reducing posttraumatic stress symptoms, program participants may more effectively regulate their emotions and, subsequently, respond to provocations in a nonaggressive manner (Sullivan et al., 2010). Similarly, participants’ beliefs regarding the use of aggression, such that they are more inclined to use prosocial means to resolve conflict while still maintaining social status, may prevent situations from escalating into violence (Farrell et al., 2008).

Our results suggest that prior studies may not have assessed the full spectrum of positive change that program staff, who work most closely with victims of violence, consider to be important and measureable. A comprehensive literature review identified eight studies that had examined the effectiveness of HVIPs, and those studies collectively assessed 23 outcomes (Aboutanos et al., 2001; Cheng, Haynie, et al., 2008; Cheng, Wright, et al., 2008; Cooper et al., 2006; Cunningham et al., 2012; Johnston, Rivara, Droesch, Dunn, & Copass, 2002; Shibru et al., 2007; Zun et al., 2006). In the present study, Round 1 respondents identified 18 of these 23 outcomes, and an additional 41 novel outcomes that had not previously been described or assessed in published literature. Furthermore, in Round 2, 10 of the 25 outcomes most highly prioritized by HVIP stakeholders had not been assessed in prior studies. Thus, inclusion of these outcomes in future research on violence prevention programs will increase the meaningfulness and relevance of study findings.

The application of the Delphi method to violence prevention outcomes bridges a gap between research and practice by involving program staff members who work directly with program participants in the determination of important outcomes that can be used for both clinical and research purposes. A principal tenet of the Delphi method suggests the sample should include “experts” from the field (Delbecq et al., 1975). Nearly half of the survey respondents self-identified as frontline violence intervention specialists or mental health clinicians, and were uniquely positioned to identify outcomes they observe among program participants. The remainder of study respondents identified as program directors or coordinators who were more likely to identify outcomes that are valued by funders and hospital administrators. In addition, study respondents reported a wide range of years of experience working as a professional in the violence prevention field, further ensuring a unique range of perspectives was represented in the final list of outcomes and, most important, that the results of this study are meaningful to those most deeply invested in the work.

Having a consensus-based core set of outcomes is expected to promote cross-site collaboration between HVIPs and help support program development and refinement. Subsequent research efforts, focused on developing and/or selecting psychometrically valid instruments that assess the key constructs and domains identified in this study, will be critical in overcoming the variability in how participant outcomes have been measured, as well as the quality of that measurement. For example, some initial studies of HVIP effectiveness used single items to measure certain outcomes (Cheng, Haynie, et al., 2008; Cheng, Wright, et al., 2008), whereas others did not include the psychometric properties of their assessment tools (Aboutanos et al., 2001; Cooper et al., 2006). Well-validated outcome measurement tools will facilitate the development of a strong evidence base from which to assess the program effectiveness; examine mediators, moderators, and mechanisms of change; and potentially enhance the ability for these programs to obtain funding through a variety of sources.

Although the focus of this study was on individual-level outcomes amenable to change through short-term HVIP participation, future research should examine HVIPs’ impact on community-level outcomes. Importantly, the Center for Disease Control’s model of violence prevention (Dahlberg & Krug, 2002) is rooted in Bronfenbrenner’s (1977) social–ecological theory, a central tenet of which states that individual behavior results from interactions among environmental- and individual-level variables. The goal of the present study was to establish a systematic list of outcomes that would be most amenable to change among individual program participants. However, it is our hope that increased understanding regarding (a) which individual-level outcomes consistently demonstrate change and (b) increased cross-site collaboration and continued program refinement provide impetus for examining ecological-level change. If some individual-level outcomes consistently demonstrate change across multiple studies, this will allow researchers to generate hypothesized pathways through which ecological-level constructs, by extension, may change. In addition, increased cross-site collaboration will help provide researchers with the necessary statistical power to examine research questions regarding ecological constructs (e.g., use of multilevel modeling to examine between-site differences).

Although the respondent characteristics are a strength of the study, the process by which the sample was selected may be a limitation. Selection bias may have been introduced as respondents were nominated by program directors rather than being randomly selected. That said, it is important to note that most HVIPs have fewer than five interventionists, so the nomination of three to five program staff likely included the majority of potential participants in the field. Another potential limitation is the concern that the Delphi method does not allow an opportunity for group discussion (Walker & Selfe, 1996). However, it is important to recognize that consensus regarding the prioritized set of domains does not mean that they are final or finite. Rather, results from a Delphi method are intended to spur additional research questions and discussion among researchers in a field (Hasson, Keeney, & McKenna, 2000). A final limitation is that the Delphi method was applied only to program staff and not program participants. Future efforts to refine the prioritized domains will apply methods to include the assault-injured participants as well as program funders to ensure that the universe of key outcomes are identified and measured.

Conclusion

HVIPs lie at the growing intersection of health care and community-based violence prevention interventions. Through a Delphi method, this study delineates meaningful and measurable outcomes prioritized by “on-the-ground” HVIP experts, and identifies novel outcomes that have not been previously reported in the literature. Results enhance the potential to further develop an evidence base for these violence prevention programs—critically important, given that many HVIPs serve urban, underserved communities in which violence is common. In addition, the broader definition of key outcomes and postinjury recovery such as mental health and emotional and social well-being may be more salient to program participants and more proximal or mechanistic to criminal behavior and retaliation outcomes. Further research to identify and apply valid measures for these constructs must include theory testing and valid causal inference, and explore properties of measures, predictors of response, and mechanisms of change (Gottfredson et al., 2015).

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

W. John Monopoli

Author Biographies

W. John Monopoli, MA, is a graduate student in clinical psychology at Ohio University. He studies the development and dissemination of evidence-based interventions for youth. Prior to coming to Ohio, he worked for Children’s Hospital of Philadelphia’s Violence Prevention Initiative.

Rachel K. Myers, PhD, is a research associate and Violence Prevention Initiative Fellow at the Children’s Hospital of Philadelphia. In this role, she offers a unique perspective to the causes and impact of violence on children and youth, and helps set direction and priorities for the initiative.

Brooke S. Paskewich, PsyD, is the program manager for the Violence Prevention Initiative. In this role, she managers the infrastructure and operational efforts of the initiative, as well as the signature programs housed within it.

Katherine B. Bevans, PhD, is a child health outcomes methodologist and associate professor of psychology at Temple University. She is the lead investigator on several Patient-Centered Outcomes Research Institute (PCORI) contacts aimed at enhancing patient and family engagement in research.

Joel A. Fein, MPH, MD, is the codirector of the Children’s Hospital of Philadelphia’s Violence Prevention Initiative, which is a multitiered, trauma-informed approach to violence prevention. His research focuses on violence prevention, mental health, and procedural pain management.

References

Aboutanos

M. B.

Jordan

Cohen

Foster

R. L.

Goodman

Halfond

R. W.

. . . Ivatury

R. R.

(2001). Brief violence interventions with community case management services are effective for high-risk trauma patients. The Journal of Trauma: Injury, Infection, and Critical Care, 71, 228-237. doi:10.1097/TA.0b013e31821e0c86

Aiken

L. H.

(2002). Hospital nurse staffing and patient mortality, nurse burnout, and job dissatisfaction. Journal of the American Medical Association, 288, 1987-1993. doi:10.1001/jama.288.16.1987

Blum

N. J.

Feldman

H. M.

Barbaresi

W. J.

Schonfeld

D. J.

Hansen

R. L.

Forrest

C. B.

(2012). Research priorities for developmental-behavioral pediatrics. Journal of Developmental & Behavioral Pediatrics, 33, 509-516. doi:10.1097/DBP.0b013e31825a7101

Bronfenbrenner

(1977). Toward an experimental ecology of human development. American Psychologist, 32, 513-531. doi:10.1037/0003-066X.32.7.513

Byrne

O’Connell

Egan

A. M.

Dinneen

S. F.

Hynes

O’Hara

M. C.

. . . Coyne

(2017). A core outcomes set for clinical trials of interventions for young adults with type 1 diabetes: An international, multi-perspective Delphi consensus study. Trials, 18, 1-8. doi:10.1186/s13063-017-2364-y

Centers for Disease Control and Prevention. (2016). 10 leading causes of death by age group, United States—2016. Retrieved from https://www.cdc.gov/injury/images/lc-charts/leading_causes_of_death_age_group_2016_1056w814h.gif

Chan

A.-W.

Hróbjartsson

Haahr

M. T.

Gøtzsche

P. C.

Altman

D. G.

(2004). Empirical evidence for selective reporting of outcomes in randomized trials. Journal of the American Medical Association, 291, 2457-2465. doi:10.1001/jama.291.20.2457

Cheng

T. L.

Haynie

Brenner

Wright

J. L.

Chung

Simons-Morton

(2008). Effectiveness of a mentor-implemented, violence prevention intervention for assault-injured youths presenting to the emergency department: Results of a randomized trial. Pediatrics, 122, 938-946. doi:10.1542/peds.2007-2096

Cheng

T. L.

Wright

J. L.

Markakis

Copeland-Linder

Menvielle

(2008). Randomized trial of a case management program for assault-injured youth: Impact on service utilization and risk for reinjury. Pediatric Emergency Care, 24, 130-136. doi:10.1097/PEC.0b013e3181666f72

10.

Clarke

(2007). Standardising outcomes for clinical trials and systematic reviews. Trials, 8, Article 39. doi:10.1186/1745-6215-8-39

11.

Cooper

Eslinger

D. M.

Stolley

P. D.

(2006). Hospital-based violence intervention programs work. The Journal of Trauma: Injury, Infection, and Critical Care, 61, 534-540. doi:10.1097/01.ta.0000236576.81860.8c

12.

Corso

P. S.

Mercy

J. A.

Simon

T. R.

Finkelstein

E. A.

Miller

T. R.

(2007). Medical costs and productivity losses due to interpersonal and self-directed violence in the United States. American Journal of Preventive Medicine, 32, 474-482. doi:10.1016/j.amepre.2007.02.010

13.

Cunningham

R. M.

Chermack

S. T.

Zimmerman

M. A.

Shope

J. T.

Bingham

C. R.

Blow

F. C.

Walton

M. A.

(2012). Brief motivational interviewing intervention for peer violence and alcohol use in teens: One-year follow-up. Pediatrics, 129, 1083-1090. doi:10.1542/peds.2011-3419

14.

Dahlberg

L. L.

Krug

E. G.

(2002). Violence—A global public health problem. In Krug

Dahlberg

L.L.

Mercy

J.A.

Zwi

A.B.

Lozano

(Eds.), World report on violence and health (pp. 1-56). Geneva, Switzerland: World Health Organization

15.

Delbecq

A. L.

Van de Ven

A. H.

Gustafson

D. H.

(1975). The Delphi techniques. In Delbecq

Van de Ven

Gustafson

(Eds.), Group techniques for program planning (pp. 83-107). Glenview, IL: Scott Foresman Company.

16.

Farrell

A. D.

Erwin

E. H.

Bettencourt

Mays

Vulin-Reynolds

Sullivan

. . . Meyer

. (2008). Individual factors influencing effective nonviolent behavior and fighting in peer situations: A qualitative study with urban African American adolescents. Journal of Clinical Child and Adolescent Psychology, 37, 397-411. doi:10.1080/15374410801955821

17.

Fowler

P. J.

Tompsett

C. J.

Braciszewski

J. M.

Jacques-Tiura

A. J.

Baltes

B. B.

(2009). Community violence: A meta-analysis on the effect of exposure and mental health outcomes of children and adolescents. Development and Psychopathology, 21, 227-259. doi:10.1017/S0954579409000145

18.

Gottfredson

D. C.

Cook

T. D.

Gardner

F. E. M.

Gorman-Smith

Howe

G. W.

Sandler

I. N.

Zafft

K. M.

(2015). Standards of evidence for efficacy, effectiveness, and scale-up research in prevention science: Next generation. Prevention Science, 16, 893-926. doi:10.1007/s11121-015-0555-x

19.

Hagaman

A. K.

Wutich

(2017). How many interviews are enough to identify metathemes in multisited and cross-cultural research? Another perspective on Guest, Bunce, and Johnson’s (2006) landmark study. Field Methods, 29, 23-41. doi:10.1177/1525822X16640447

20.

Hasson

Keeney

McKenna

(2000). Research guidelines for the Delphi survey technique. Journal of Advanced Nursing, 32, 1008-1015. doi:10.1046/j.1365-2648.2000.t01-1-01567.x

21.

Johnston

B. D.

Rivara

F. P.

Droesch

R. M.

Dunn

Copass

M. K.

(2002). Injury risk: A randomized, controlled trial. Pediatrics, 110, 267-274.

22.

Juillard

Cooperman

Allen

Pirracchio

Henderson

Marquez

. . . Dicker

R. A.

(2016). A decade of hospital-based violence intervention: Benefits and shortcomings. Journal of Trauma and Acute Care Surgery, 81, 1156-1161. doi:10.1097/TA.0000000000001261

23.

Kelly

Tong

Tymms

March

Craig

J. C.

De Vera

. . . Zeljkovic

(2018). Outcome measures in rheumatology—Interventions for medication adherence (OMERACT-Adherence) core domain set for trials of interventions for medication adherence in Rheumatology: 5 phase study protocol. Trials, 19, 1-13. doi:10.1186/s13063-018-2565-z

24.

Krippendorf

(1980). Content analysis: An introduction to its methodology. Beverly Hills, CA: Sage.

25.

Louviere

Lings

Islam

Gudergan

Flynn

(2013). An introduction to the application of (case 1) best-worst scaling in marketing research. International Journal of Research in Marketing, 30, 292-303. doi:10.1016/j.ijresmar.2012.10.002

26.

Lynch

Cicchetti

(1998). An ecological-transactional analysis of children and contexts: The longitudinal interplay among child maltreatment, community violence, and children’s symptomatology. Development and Psychopathology, 10, 235-257. doi:10.1017/S095457949800159X

27.

National Network of Hospital-Based Violence Intervention Programs. (2016). NNHVIP Annual Member Survey (unpublished). Available at: http://nnhvip.org/

28.

Ozkol

Zucker

Spinazzola

(2011). Violence exposure and depressive symptoms among adolescents and young adults disconnected from school and work. Journal of Community Psychology, 38, 607-621. doi:10.1002/jcop

29.

Richters

J. E.

Martinez

(1993). The NIMH Community Violence Project: I. Children as victims of and witnesses to violence. Psychiatry: Interpersonal and Biological Processes, 56, 7-21.

30.

Rudolph

Hiscock

Price

Efron

Sewell

South

Wake

(2009). What research questions matter to Australian paediatricians? National Delphi study. Journal of Paediatrics and Child Health, 45, 704-710. doi:10.1111/j.1440-1754.2009.01598.x

31.

Ryan

G. W.

Bernard

H. R.

(2003). Techniques to identify themes. Field Methods, 15, 85-109. doi:10.1177/1525822X02239569

32.

Sawtooth Software [Computer software, 2014]. Orem, UT: Sawtooth Software Inc.

33.

Shibru

Zahnd

Becker

Bekaert

Calhoun

Victorino

G. P.

(2007). Benefits of a hospital-based peer intervention program for violently injured youth. Journal of the American College of Surgeons, 205, 684-689. doi:10.1016/j.jamcollsurg.2007.05.029

34.

Shorter

G. W.

Heather

Bray

J. W.

Giles

E. L.

Holloway

Barbosa

. . . Newbury-Birch

(2017). The “Outcome Reporting in Brief Intervention Trials: Alcohol” (ORBITAL) framework: Protocol to determine a core outcome set for efficacy and effectiveness trials of alcohol screening and brief intervention. Trials, 18, 1-7. doi:10.1186/s13063-017-2335-3

35.

Sinha

I. P.

Smyth

R. L.

Williamson

P. R.

(2011). Using the Delphi technique to determine which outcomes to measure in clinical trials: Recommendations for the future based on a systematic review of existing studies. PLoS Medicine, 8. doi:10.1371/journal.pmed.1000393

36.

Spoth

Rohrbach

L. A.

Greenberg

Leaf

Brown

C. H.

Fagan

. . . Hawkins

J. D.

(2013). Addressing core challenges for the next generation of type 2 translation research and systems: The translation science to population impact (TSci Impact) framework. Prevention Science, 14, 319-351. doi:10.1007/s11121-012-0362-6

37.

Stoddard

S. A.

Zimmerman

M. A.

Bauermeister

J. A.

(2011). Thinking about the future as a way to succeed in the present: A longitudinal study of future orientation and violent behaviors among African American youth. American Journal of Community Psychology, 48, 238-246. doi:10.1007/s10464-010-9383-0

38.

Sullivan

T. N.

Helms

S. W.

Kliewer

Goodman

K. L.

(2010). Associations between sadness and anger regulation coping, emotional expression, and physical and relational aggression among urban adolescents. Social Development, 19, 30-51. doi:10.1111/j.1467-9507.2008.00531.x

39.

Tay

Diener

(2011). Needs and subjective well-being around the world. Journal of Personality and Social Psychology, 101, 354-365. doi:10.1037/a0023779

40.

Walker

A. M.

Selfe

(1996). The Delphi method: A useful tool for the allied health researcher. British Journal of Therapy and Rehabilitation, 3, 677-681. doi:10.12968/bjtr.1996.3.12.14731

41.

Zun

L. S.

Downey

Rosen

(2006). The effectiveness of an ED-based violence prevention program. American Journal of Emergency Medicine, 24, 8-13. doi:10.1016/j.ajem.2005.05.009