Piloting of COPES: An Empirically Informed Psychosocial Intervention on an Adolescent Psychiatric Inpatient Unit

Abstract

Objectives:

This study evaluated the feasibility and initial efficacy of an empirically informed psychosocial intervention on an adolescent psychiatric inpatient unit.

Methods:

Data were obtained for 463 adolescents 12–16 years of age on a psychiatric inpatient unit. Information collected included demographics, psychiatric diagnoses, length of inpatient stay, completion of four treatment modules, rehospitalizations, and emergency room visits during 12 months after discharge from index admission.

Results:

Around 98.70% of patients completed at least one out of the four treatment modules and 93.95% of patients completed two modules. There were no significant barriers to completing treatment modules on the basis of participant characteristics (demographics, psychiatric diagnosis, number of diagnoses, or length of stay). Completion of the four treatment modules, particularly modules on developing a safety plan and enhancing life, predicted lower risk for rehospitalization and emergency room contact in the 12 months postdischarge.

Conclusion:

Findings suggest that the intervention is feasible to implement regardless of common barriers in an inpatient psychiatric setting, such as complex psychopathology and brief duration of hospitalization. Completion of treatment modules significantly reduces risk for subsequent emergency intensive service utilization, suggesting this intervention may be an effective method for reducing acute clinical events.

Introduction

While considerable evidence supports the efficacy of cognitive behavioral therapy for treating a variety of psychological disorders, there is a paucity of research using this approach in inpatient settings (Knesper 2010). The number of youth hospitalized for thoughts of suicide or self-harm has more than doubled during the last decade (Plemmons 2017). However, few studies have examined the effectiveness of psychosocial treatments delivered in the inpatient setting.

Unfortunately, studies of youth discharged from inpatient treatment show high rates of rehospitalization and repeat psychiatric crises (Chung et al. 2008; Yen et al. 2014). The risk appears particularly high within the first 12 months of posthospitalization, with rates of readmission or repeat contact with psychiatric crises services being as high as 46% (James et al. 2010). Providing quality care in this setting has become increasingly challenging given reduced lengths of hospitalization, and there is an increased need for delivering effective brief interventions (Glick et al. 2011; Curtin et al. 2016).

Although cognitive behavioral treatment (CBT) has been well supported in outpatient settings for a host of disorders (e.g., anxiety, depression), its feasibility and clinical utility in short-term inpatient settings have yet to be empirically determined. Manuals designed to address suicidality in other settings tend to involve lengthy protocols lasting 6–12 months and involve a large number of sessions (e.g., Esposito-Smythers et al. 2011). While long-term psychotherapy and sufficient dosage of therapy are important, these approaches are not suitable for the brief duration of inpatient treatment. However, because the highest risk period for recurrent suicidal events is 1–4 weeks after hospital discharge, it is important that treatment be initiated as soon as possible during hospitalization to equip patients with basic skills before discharge (Brent et al. 2013).

While inpatient hospitalization creates an important window of opportunity, few psychosocial treatments have been adapted for the inpatient setting. The limited research available suggests that psychiatric hospitalization of youth is most helpful when there is a solid therapeutic alliance, longer stay, problem-solving skill-based therapy utilizing a cognitive behavioral theoretical orientation, better premorbid family functioning, and integration with aftercare services (Blanz and Schmidt 2000; Green et al. 2007). Although these studies are suggestive of the potential benefits of psychosocial treatments in these settings, the literature regarding effective inpatient therapeutic content of care is extremely limited, and such studies tend to have a variety of methodological issues that limit their generalizability and interpretability to inform inpatient care. In particular, these studies tend to have small sample sizes and do not provide sufficient follow-up data.

Delivery of these treatments has become increasingly challenging in the inpatient setting given reduced lengths of hospitalization and increased acuity among youth (Glick et al. 2011). Some brief intervention models for suicidality have been implemented in the community, but focus on school settings with youth who are less acute than those in the inpatient setting (Schilling et al. 2014). Others, such as Applied Suicide Intervention Skills Training, focus on training general skills of talking with those at risk and acting as gatekeepers to refer them for treatment (Rodgers 2010; Quinnett 2012). Another approach, ED-SAFE, is a brief suicide intervention model that was recently developed for emergency departments. However, this model is specifically designed for adult populations, and therefore is not well suited for use with youth (Miller et al. 2017). To date, we are not aware of any brief skill-based suicide intervention programs with demonstrated efficacy in the inpatient setting.

The present study aimed to adapt an empirically supported CBT manual (Esposito-Smythers et al. 2011) for use with acute adolescent inpatients with a broad range of psychopathology within the time constraints of a brief hospital stay, and to assess whether implementation of this adapted manual is associated with reduction in acute psychiatric service utilization over a 12-month period. First, it was hypothesized that the treatment would be feasible to implement as demonstrated by completion of treatment modules, despite factors commonly cited as reducing the ability to provide empirically informed treatment in inpatient settings, including age, gender, diagnostic status, and length of stay (Glick et al. 2011). Second, it was hypothesized that module completion would be associated with lower prospective risk for return to intensive psychiatric services as evidenced by reduced rehospitalizations and use of emergency services during the 12 months following hospital discharge, supporting effectiveness of the intervention.

Methods

Participants

Participants were drawn from patients admitted between April 2015 and March 2016 to a psychiatric inpatient facility in Rhode Island with two 17-bed inpatient units (total of 34 beds) due to an imminent threat to themselves or others. Youth over the age of 16 were excluded since they would age out of adolescent psychiatric services over the course of the follow-up period. Thus, they would be ineligible for the same service utilization at age 18, which is the clinical outcome of interest in the current study. In the first year of the full implementation stage, there were 530 nonrepeat adolescent admissions between the ages of 12 and 16. Of these, 463 completed the standard hospital assessment protocol and were included in subsequent analyses. The majority of participants were female (64.58%) and had a mean age of 14.45 (standard deviation [SD] = 1.20). In terms of race and ethnicity, 74.34% were Caucasian, 7.52% were African American, 2.43% were multiracial, 15.71% were other, and 21.75% were Latino. Diagnoses based on a structured interview (see below for description) are listed in Table 1. The mean number of diagnoses was 2.38 (SD = 1.48). The average length of stay was 9.34 days (SD = 8.79), which was influenced by several factors, including insurance coverage, disposition plan, and family involvement.

Table 1.

Diagnoses Based on the Children's Interview for Psychiatric Syndromes

Diagnosis	N (%)
Major depressive disorder/dysthymia	322 (69.55)
Mania/hypomania	28 (6.05)
Attention-deficit/hyperactivity disorder	133 (28.73)
Generalized anxiety disorder/social phobia	248 (53.56)
Oppositional defiant disorder/conduct disorder	181 (39.09)
PTSD/acute stress	102 (22.03)
Psychosis/schizophrenia	61 (13.17)
Anorexia/bulimia	26 (5.62)

PTSD, post-traumatic stress disorder.

Overview of the coping, problem solving, enhancing life, and safety planning intervention

The COping, Problem solving, Enhancing life, and Safety planning (COPES) treatment protocol was based on an empirically supported CBT manual designed for outpatient treatment of suicidal adolescents (Donaldson et al. 2005; Esposito-Smythers et al. 2011; Spirito et al. 2015) that was adapted for use in an inpatient setting. Using this approach, teens received psychoeducation and skills training from a multidisciplinary team that focused on enhancing safety and improving coping skills and problem solving around the factors that led to hospitalization. The intervention began with an assessment battery consisting of a clinical interview, structured diagnostic interview, and behavior checklists (completed by the parent and teen). Psychoeducation specific to the primary presenting problem (e.g., suicide, depression, trauma, substance use, mood instability) was provided in written and verbal form from the treatment team to review common symptoms, etiology, treatment approaches, and information on reducing suicide and self-injury.

With the help of various members on the multidisciplinary team, adolescents completed four skill training modules:COPES. The Coping Plan module helped teens develop a list of regulatory strategies, including things they can say to themselves and things they can do to feel better. Patients identify triggers and physiologic signs of distress, and then generate a list of coping thoughts and behaviors that can be used to alleviate this distress.

The Problem-solving module helped teens identify the triggers that led to hospitalization and problem-solving ways to address these issues. For example, when addressing the primary trigger for a suicide attempt, the therapist helped the adolescent identify and define the problem situation, generate potential options, evaluate pros and cons of each potential option, and then select the best option for solving the problem that triggered the suicide attempt. The therapist then reframed the suicide attempt as a failure in problem solving. This explanation provided adolescents with a better sense of control over future problems. The therapist explained that many teenagers who attempt suicide pick the only option that they think they have, which is to hurt themselves. The therapist emphasized that the more adolescents practice coming up with a list of options, the more potential solutions they have to choose from when distressed, and the less likely they will feel that the only thing they can do is to hurt themselves. The third module focused on Enhancing life by following an individualized plan for taking medication, eating healthily, engaging in pleasant activities, and improving sleep to enhance self-care and reduce vulnerability to negative emotions. Finally, the Safety plan module helped teens develop a safety plan that centers on ways teens can keep themselves safe and obtain support when needed after leaving the hospital. Patients made a list of dangerous items to remove from their surroundings (e.g., alcohol, knives, razors, medications) to reduce access to means, identified their warning signs and vulnerabilities that indicated they may become unsafe, generated helpful coping strategies, and identified people that could help the teen manage negative feelings.

The skill modules were typically presented in the order described, although clinicians and staff could modify the order as needed. The skills were continually strengthened throughout hospitalization in individual, group, and family therapy sessions by milieu staff, nurses, psychiatrists, psychologists, and social workers. Typically, the skills were initially presented to teens by the milieu staff and completed collaboratively. Each module included an accompanying worksheet that was reviewed in individual therapy sessions with the psychologist or psychiatrist. Before discharge, the completed modules and worksheets were discussed in family meetings led by a social worker.

Implementation of the intervention

The implementation process followed the stages recommended by Fixsen et al. (2005). In the Exploration Stage, hospital staff aimed to identify the need for change and explore new treatment approaches. During the Installation Stage, administrators and clinicians worked to train staff and adjust policies and procedures to support new practices. All staff, including psychiatrists, psychologists, social workers, nurses, and milieu staff, received the same training. Training consisted of didactics, reviewing videotaped examples of each module being used with patients, live supervision of staff conducting these sessions before the approval of independent skill delivery, and weekly supervision meetings. In the Initial Implementation Stage, the multidisciplinary team began using the treatment protocol with patients, and the implementation team problem solved barriers such as logistical difficulties and personnel issues. In particular, we worked to enhance communication regarding module completion and reduce the diffusion of responsibility in completing modules. Six months after initial implementation, we entered the Full Implementation Stage when the system was fully in use. Data presented in the current study are from the full implementation stage (beginning April, 2015).

Procedures

The investigation was carried out in accordance with the latest version of the Declaration of Helsinki. Following approval from the hospital's Institutional Review Board, data were collected from an electronic medical record system called LifeChart, using Epic software. Because this was a quality improvement study, informed consent was not required. As part of standard intake procedures, patients received a structured diagnostic interview administered by a psychology research assistant within 72 hours of hospital admission. The four modules were reviewed with patients throughout their hospital stay. When a module was completed, a copy of the completed module was scanned into LifeChart. This allowed for module completion to be calculated by the presence of completed forms in LifeChart or specific mention of module completion in LifeChart progress notes. Follow-up data were collected following index hospitalization for each patient for the time period between index hospitalization and May 2016 (2- to 12-month follow-up period for each patient depending upon length of time between index admission and end of data collection in May 2016). These data included information from LifeChart that captures subsequent emergency room visits and rehospitalizations. LifeChart includes information throughout Rhode Island's healthcare system, from hospitals and clinics, to ambulatory centers and community partners. The system includes all emergency rooms in the state and all but one psychiatric hospital, allowing us to track patient movement through the vast majority of the mental healthcare system in the state. Especially given the unique nature of Rhode Island being a small state, this provides fairly comprehensive coverage of acute service utilization.

Measures

The Childhood Inventory of Psychiatric Syndromes (ChIPS) (Weller et al. 2000) was used as part of standard unit procedures to identify diagnoses based on Diagnostic and Statistical Manual of Mental Disorders, 4th ed., Text Revision (DSM-IV-TR; American Psychiatric Association 2000) criteria. The ChIPS is a highly structured diagnostic interview covering 20 Axis-I disorders designed for 6- to 18-year-old youth. Reliability and validity studies of the ChIPS reveal sound psychometric properties with youth in both inpatient and outpatient settings (Fristad et al. 1998; Teare et al. 1998a, 1998b). The ChIPS is an ideal diagnostic instrument for the inpatient setting where reliable yet quick assessment is required for treatment planning during the brief duration of stay.

Data analyses

Descriptive statistics were used to report on the feasibility of the intervention. To examine factors that may be barriers to module completion (i.e., age, sex, length of stay, diagnostic status), Pearson correlations were calculated for the continuous variable of number of modules completed and χ ² and independent samples t-tests were conducted for the dichotomized variables of specific module completion. Cox regression (survival) analyses were conducted to assess the relationship between the completion of treatment modules and time (in days) to subsequent use of intensive services (defined as rehospitalization on the inpatient unit or emergency room visit). This analysis was first conducted with the number of modules completed. The time to intensive service utilization (inpatient admission or emergency room visit) was the outcome measure. To ensure that the relationship between intervention adherence and intensive service utilization was not related to other factors, we included the following covariates: age, sex, length of stay, and number of diagnoses. This analysis was repeated in four separate models with completion of each specific module, in turn, serving as the predictor variable of interest.

Results

Feasibility

A total of 98.70% of patients completed at least one of the treatment modules and 42.98% completed all four components. The Coping plan was completed by 94.17% of patients; 81.64% completed Problem solving; 52.05% completed the Enhancing life module; and 84.67% completed the Safety plan. Although all patients received some amount of general safety planning before discharge, these general plans varied widely with clinician. The COPES safety planning module differed from the general safety planning in providing a structured, more in-depth safety plan, which was completed with input from the patient, family, and treatment team. We also sought to determine if module completion was related to factors commonly cited as reducing the ability to provide empirically informed treatment in inpatient settings (i.e., age, sex, length of stay, and diagnoses, including depression, mania, attention-deficit/hyperactivity disorder, anxiety, psychosis, conduct problems, post-traumatic stress disorder (PTSD)/acute stress, and eating disorders). None of the factors examined was associated with the number of modules completed. We also examined the association between these patient characteristics and the completion of each of the four modules (Coping plan, Problem solving, Healthy living, and Safety plan). Conduct disorder was the only diagnosis associated with lower completion rate of any module (the Enhancing life module). It should be noted, however, that the effect size was small (odds ratio = 0.62, 95% confidence interval = 0.42–0.90).

Survival analyses

Overall, 39.74% (n = 184) of patients were readmitted to a psychiatric unit or seen in the emergency room for a psychiatric concern after discharge from the inpatient unit. In a multivariate survival analysis, the number of modules completed during the index admission significantly predicted the time to subsequent intensive service utilization (Table 2). Specifically, greater number of modules completed predicted a longer time to subsequent intensive service contact after age, sex, length of hospital stay, and number of diagnoses was covaried. In terms of specific modules, completion of the Enhancing life and Safety plan modules, respectively, was also significantly predictive of longer time to intensive service utilization after accounting for the same covariates. Figures 1 and 2 depict the survival curves for intensive service utilization as a function of completing these specific modules.

FIG. 1.

Enhancing life module.

FIG. 2.

Safety plan module.

Table 2.

Summary of Cox Regression Models Predicting Time to Intensive Psychiatric Services

Predictor	OR	95% CI
Number of modules completed
Number of modules	0.796^**	0.690–0.918
Age	0.907	0.805–1.023
Sex (female)	1.512^*	1.078–2.120
Length of stay (days)	1.011^*	1.001–1.020
Number of diagnoses	1.074	0.971–1.188
Coping plan module completed
Coping plan module	0.716	0.413–1.242
Age	0.906	0.803–1.021
Sex (female)	1.451^*	1.033–2.038
Length of stay (days)	1.010^*	1.000–1.020
Number of diagnoses	1.064	0.962–1.178
Problem-solving module completed
Problem-solving module	0.755	0.528–1.079
Age	0.905	0.803–1.021
Sex (female)	1.493^*	1.065–2.095
Length of stay (days)	1.010^*	1.000–1.020
Number of diagnoses	1.071	0.968–1.185
Enhancing life module completed
Enhancing life module	0.689^*	0.515–0.923
Age	0.906	0.803–1.022
Sex (female)	1.532^*	1.091–2.150
Length of stay (days)	1.010^*	1.000–1.019
Number of diagnoses	1.068	0.966–1.181
Safety plan module completed
Safety plan module	0.649^*	0.450–0.936
Age	0.900	0.798–1.016
Sex (female)	1.497^*	1.067–2.100
Length of stay (days)	1.011^*	1.001–1.020
Number of diagnoses	1.065	0.962–1.179

p < 0.05, ^** p < 0.01.

OD, odds ratio; CI, confidence interval.

Discussion

The present study examined the feasibility and initial effectiveness of implementing a CBT protocol on an acute psychiatric inpatient unit with adolescent patients presenting with a broad range of psychopathology. As predicted, the treatment was feasible to implement as demonstrated by high completion rates of three out of four modules, and the majority of patients received the program. Results suggest that the feasibility of administering treatment was not diminished by the challenges of brief hospitalization, multiple diagnoses, or severe psychopathology. Length of stay was not related to treatment feasibility and only conduct disorder was associated with decreased completion of the Enhancing life component of the intervention (although the associated effect size was modest). The primary hypothesis was confirmed as it was demonstrated that module completion was associated with reduced risk for subsequent hospitalizations and use of emergency psychiatric services. Specifically, the more modules that were completed, the less likely teens were to be hospitalized soon after discharge.

Results further demonstrated that those who completed the specific modules related to Enhancing life and the Safety plan were more protected from rehospitalization. The protective effect of the Safety plan supported by the present study aligns with national guidelines for suicide prevention (U.S. Department of Health and Human Services 2012), which name safety planning as a critical component of the care that should be provided to individuals at high risk of attempting suicide. According to these guidelines, providers should work directly with patients to create individualized safety plans, and safety plans should include instructions on how to access psychiatric care during times of crises. The Safety plan completed by participants in this study met both of these criteria. It should be noted that due to the flexible but suggested order of session completion, teens were less likely to complete these two modules that were presented later in the protocol. Therefore, it is unknown if these results are related to the specific content of the Enhancing Life and Safety Plan modules or the number of modules received.

Limitations

The current study included few limitations of note. First, participants were not randomized to treatment and there was no control group for comparisons. Although this study is an important first step in demonstrating the feasibility of the COPES intervention, future studies may use a more stringent test of the treatment to provide further support for the effectiveness of the intervention. In addition, measures of competency or fidelity in completion of the modules were not included. Given that an inpatient psychiatric unit requires a large number of staff members, variability in the quality of implementation may exist, and although standardized training was adopted, there may be differences in staff members' level of competence in delivering the intervention. This issue may be to some degree reduced in the case of the current intervention, given its low intensity nature. Nonetheless, future studies would benefit from closer examination of the content of the completed modules as it is likely that patients who had more thoughtful completion of the modules had greater benefit than those who had poor quality completion. The present study does not include pre and posttreatment outcomes measures, which could be considered a study limitation. However, the current investigation focuses on rehospitalization and utilization of intensive psychiatric services as an outcome instead, since this type of service utilization is extremely meaningful in the practical clinical context. Utilizing rehospitalization and use of intensive psychiatric services as the main outcome provides an important clinical measure in and of itself. Furthermore, although a unique strength of the current study was its comprehensive coverage of acute psychiatric service use in the mental health system in Rhode Island, a limitation that should be noted with any study assessing recurrence of acute psychiatric service use based on medical records, is that it is possible that a minority of patients received these services at follow-up in medical centers outside of our system, thus resulting in an underestimation of the rate of repeat acute psychiatric service use. While service utilization is an important factor to examine, future studies should consider additional indicators of impairment, including posthospitalization symptomatology and quality of life. Future studies may also identify moderators, such as intelligence, level of engagement, or other factors that may influence treatment feasibility and response. Although we examined some factors that may influence feasibility, future studies may evaluate additional factors, including differences in insurance coverage.

Conclusions

While previous literature failed to identify empirically supported treatments for adolescents in this setting, the current study showed CBT module completion reduced risk for subsequent psychiatric hospitalization and use of emergency psychiatric services despite the challenges of brief hospitalization, multiple diagnoses, or severe psychopathology. While the need to address multiple symptoms in severe patients presents challenges, it is important to conduct more rigorous evaluations of treatments within these real-world settings. The preliminary results presented here provide initial support for using a CBT model and future effectiveness research is needed to further inform the use of empirically supported treatments in the inpatient setting.

Clinical Significance

Results of this study have direct clinical implications for inpatient psychiatric treatment in youth. These findings support the use of a brief CBT modular intervention as an empirically-informed treatment for the highly acute and heterogeneous patient population typically found on inpatient psychiatric settings. The use of these treatment protocols helps ensure that each patient receives standardized, quality treatment in the hospital setting. Thus, brief CBT interventions show promise for teaching basic psychotherapy skills during hospitalization and reducing repeat psychiatric admissions, which may lead to fewer disruptions in psychosocial functioning and cost savings for the healthcare industry.

Footnotes

Acknowledgments

The authors would like to acknowledge the support of Brian Seuch, Sophia Batalau, and Samantha Moul in collecting data for this study.

Disclosures

No competing financial interests exist.

References

American Psychiatric Association: Diagnostic and Statistical Manual of Mental Disorders

4th ed., Text Revision. Washington, DC: American Psychiatric Association; 2000.

Blanz

, Schmidt

: Practitioner review: Preconditions and outcome of inpatient treatment in child and adolescent psychiatry. J Child Psychol Psychiatry, 41:703–712, 2000.

Brent

, McMakin

, Kennard

, Goldstein

, Mayes

, Douaihy

: Protecting adolescents from self-harm: A critical review of intervention studies. J Am Acad Child Adolesc Psychiatry, 52:1260–1271, 2013.

Chung

, Edgar-Smith

, Palmer

, Bartholomew

, Delambo

: Psychiatric rehospitalization of children and adolescents: Implications for social work intervention. Child Adolesc Soc Work J, 25:483–496, 2008.

Curtin

, Warner

, Hedegaard

: Increase in Suicide in the United States, 1999–2014. (NCHS Data Brief, No 241). Hyattsville (Maryland): National Center for Health Statistics, 2016.

Donaldson

, Spirito

, Esposito-Smythers

: Treatment for adolescents following a suicide attempt: Results of a pilot trial. J Am Acad Child Adolesc Psychiatry, 44:113–120, 2005.

Esposito-Smythers

, Spirito

, Kahler

, Hunt

, Monti

: Treatment of co-occurring substance abuse and suicidality among adolescents: A randomized clinical trial. J Consult Clin Psychol, 79:728–739, 2011.

Fixsen

, Naoom

, Blase

, Friedman

, Wallace

: Implementation Research: A Synthesis of the Literature (FMHI #231). Tampa (Florida): University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network, 2005.

Fristad

, Cummins

, Verducci

, Teare

, Weller

: Study IV: Concurrent validity of the DSM-IV revised Children's Interview for Psychiatric Syndromes (ChIPS). J Child Adolesc Psychopharmacol, 8:227–236, 1998.

10.

Glick

, Sharfstein

, Schwartz

: Inpatient psychiatric care in the 21st century: The need for reform. Psychiatr Serv, 62:206–209, 2011.

11.

Green

, Jacobs

, Beecham

, Dunn

, Kroll

, Tobias

, Briskman

: Inpatient treatment in child and adolescent psychiatry—a prospective study of health gain and costs. J Child Psychol Psychiatry, 48:1259–1267, 2007.

12.

James

, Charlemagne

, Gilman

, Alemi

, Smith

, Tharayil

, Freeman

: Post-discharge services and psychiatric rehospitalization among children and youth. Adm Policy Ment Health, 37:433–445, 2010.

13.

Knesper

: Continuity of care for suicide prevention and research: Suicide attempts and suicide deaths subsequent to discharge from the emergency department or psychiatric inpatient unit. Newton, MA: Education Development Center, 2010.

14.

Miller

, Camargo

, Arias

, Sullivan

, Allen

, Goldstein

, Boudreaux

: Suicide prevention in an emergency department population: The ED-SAFE study. JAMA Psychiatry, 74:563–570, 2017.

15.

Plemmons

: Trends in suicidality and serious self-harm for children 5–17 years at 32 U.S. children's hospitals, 2008–2015. Paper presented at the Pediatric Academic Societies Meeting, San Francisco, CA, 2017.

16.

Quinnett

: Question, Persuade, Refer: Ask a Question, Save a Life. Oklahoma City: Oklahoma State Regents for Higher Education, 2012.

17.

Rodgers

: Review of the Applied Suicide Intervention Skills Training Program (ASIST): Rationale, Evaluation Results, and Directions for Future Research. Calgary, Canada: LivingWorks Education Incorporated, 2010.

18.

Schilling

, Lawless

, Buchanan

, Aseltine

: “Signs of suicide” shows promise as a middle school suicide prevention program. Suicide Life Threat Behav, 44:653–667, 2014.

19.

Spirito

, Wolff

, Seaboyer

, Hunt

, Esposito-Smythers

, Nugent

, Miller

: Concurrent treatment for adolescent and parent depressed mood and suicidality: Feasibility, acceptability, and preliminary findings. J Child Adolesc Psychopharmacol, 25:131–139, 2015.

20.

Teare

, Fristad

, Weller

, Salmon

: (1998a). Study I: Development and criterion validity of the Children's Interview for Psychiatric Syndromes (ChIPS). J Child Adolesc Psychopharmacol, 8:205–211.

21.

Teare

, Fristad

, Weller

, Salmon

: (1998b). Study II: Concurrent validity of the DSM-III-R Children's Interview for Psychiatric Syndromes (ChIPS). J Child Adolesc Psychopharmacol, 8:213–219.

22.

U.S. Department of Health and Human Services: 2012 National Strategy for Suicide Prevention: Goals and Objectives for Action. Washington, DC: HHS. 2012.

23.

Weller

, Weller

, Fristad

, Rooney

, Schecter

: Children's Interview for Psychiatric Syndromes (ChIPS). J Am Acad Child Adolesc Psychiatry, 39:76–84, 2000.

24.

Yen

, Fuller

, Solomon

, Spirito

: Follow-up treatment utilization by hospitalized suicidal adolescents. J Psychiatr Pract, 20:353–362, 2014.