Multi-Family Groups for Multi-Stressed Families

Abstract

Purpose:

To examine the influence of caregiver stress on attendance among urban families involved in a multiple family group (MFG) intervention, as well as pre/post changes in childhood behavioral difficulties, caregiver stress, caregiver depressive symptoms, caregiver coping by substance use, and caregiver motivation to change.

Methods:

Univariate analyses assessed pre/post differences in outcomes between and within stress groups. Post hoc analyses accounted for missing data and the potential influence of regression to the mean.

Results:

High caregiver stress was associated with lower rates of attendance in MFG over time. Despite attending the fewest sessions, highest stressed caregivers experienced the most pre/post improvement on measures of caregiver stress and childhood behavioral difficulties.

Discussion:

Involvement in an MFG intervention is associated with high rates of attendance and meaningful improvements in child and caregiver outcomes, with the most distressed families reporting the greatest reduction in both caregiver stress and childhood behavioral difficulties.

Keywords

high-stress families multiple family group caregiver stress childhood behavioral difficulties urban families

Introduction

Nationwide, child mental health presents a US$8.9 billion problem (Department of Health and Human Services, 2012), with conduct difficulties accounting for the majority of all youth mental health referrals and with rates being even higher in children within urban communities (Frick, 1998; Ghandour, Kogan, Blumberg, Jones, & Perrin, 2012; Loeber, Burke, Lahey, Winters, & Zera, 2000; Tolan & Henry, 1996). Estimates of childhood conduct problems suggest that 5–10% of children, ages 6–17 years old, have persistent oppositional or aggressive behavior problems (Angold & Costello, 2001; Costello, Mustillo, Erkanli, Keeler, & Angold, 2003; Ghandour et al., 2012), and nationwide, conduct difficulties account for one third to one half of all youth mental health referrals (Frick, 1998; Ghandour et al., 2012; Kazdin & Whitley, 2003; Loeber et al., 2000). If left untreated, conduct-related concerns at the latency age often develop into significantly more serious behavioral problems (Bassarath, 2001).

Previous studies have highlighted the “burden” (Kazdin & Blase, 2011) of current mental health needs within low-income, multi-stressed caregivers, coupled with extremely low rates of prolonged engagement among the neediest of families (Harrison, McKay, & Bannon, 2004; McKay & Bannon, 2004; McKay, Gonzales, Stone, Ryland, & Kohner, 1996). A robust body of evidence reveals that inner-city youth evidencing conduct problems and their families typically also suffer from multiple, simultaneous, and multilevel stressors that significantly affect the likelihood of engagement in needed mental health care and greatly challenge clinic providers in the delivery of services (Franco, Pottick, & Huang, 2010; Hill, Ross, & Angel, 2005). Such stressors include poverty and unemployment (Chaffin, Kelleher, & Hollenberg, 1996; Harrison et al., 2004; McKay & Bannon, 2004), domestic violence (Fleck-Henderson, 2000; Mills et al., 2000), parent substance abuse (Brown, Cohen, & Johnson, 1998; Chaffin et al., 1996; Russell, Harris, & Gockel, 2008), parent psychiatric illness (Culp, Culp, Soulis, & Letts, 1989; Dinwiddle & Bucholz, 1993; Kaplan, Pelcovitz, Salzinger, & Caneles, 1983; Monuteaux, Faraone, Gross, & Biederman, 2007), underresourced schools, and lack of child mental health clinics (Gopalan et al., 2010; McKay & Bannon, 2004). It is clear that the effects of hierarchical stressors faced by many of the urban families involved in child mental health clinics place these families at the highest risk of dropout (Kazdin, Holland, & Crowley, 1997; Kazdin & Whitley, 2003; Prinz & Miller, 1994, 1996).

We also know that there is a dearth of literature on engaging, family centered, research-supported treatment modalities that effectively reduce the burden of mental illness within families living in low-income urban communities, specifically for poor urban families experiencing high levels of stress. Prior studies suggest research-supported treatments run the highest risk of treatment failure with high stress, high need families (Weisz, Jensen-Doss, & Hawley, 2006), and this highly vulnerable subset of mental health service seekers has been identified as predisposed to negative outcomes, poor parenting self-perceptions, and premature dropout (Gopalan et al., 2010; Weisz, Doss, & Hawley, 2005). Additionally, the effectiveness of research-supported child mental health interventions often varies depending on numerous demographic and health-related factors (Kazdin & Blase, 2011) and the prevalence of disruptive behavior disorders among the urban poor is disproportionate to that of other groups (Tolan & Henry, 1996). Kazdin and Whitley (2003) suggest that a variety of both caregiver and child-level characteristics exist as a “package,” and cohesively this “package” must be addressed to optimize the effectiveness of community-based child mental health treatment.

Conceptually, better understanding of these child- and caregiver-level characteristics is needed to determine which (if any) of these covariates may help explain families’ attendance, engagement, and risk of dropout in mental health services. The importance of engaging families in child mental health services is well documented in the literature, with the notion that the potential impact of research-supported treatments for the most vulnerable of families may be limited unless families are engaged in the treatment in the first place (Harrison et al., 2004; Kazdin et al., 1997; Prinz & Miller, 1994). Therefore, family engagement (often depicted in the literature as “attendance”) can be paramount for maximizing treatment (Hoagwood et al., 2012; Weisz et al., 2006).

One emerging family-based approach in the mental health services literature for engaging families is multiple family groups (MFG), which in prior studies has shown to be a promising modality for effectively engaging and treating child disruptive behavior disorders while minimizing caregiver’s perceptual barriers to care and improving overall family functioning (Gopalan & Franco, 2009; Gopalan et al., 2010; McKay, Gonzales, Quintana, Kim, & Abdul-Adil, 1999; Mckay et al., 1996, 2010, 2011; McKay, Harrison, Gonzales, Kim, & Quintana, 2002; Stone, McKay, & Stoops, 1996). MFG is designed to engage families, increase child mental health service use, and to reduce disruptive behavior disorders in urban, low-income children of color. Early community-based clinical trials have demonstrated that MFG shows great promise as a strategy embedded within community-based clinics (Franco, Dean-Assael, & McKay, 2008; Gopalan et al., 2011; Gopalan & Franco, 2009; McKay et al., 1999, 2002, 2010; Mckay, Gonzales, et al., 1996; Stone et al., 1996). Many of these families, however, experience a set of mental health and nonmental health–related stressors that make prolonged attendance in mental health services extremely difficult, especially for low-income families (Gopalan et al., 2011; McKay et al., 2004).

To date, MFG has been tested primarily as a “one-size-fits-all” approach to engaging urban families and promoting declines in child conduct problems (McKay et al., 2011); however, what is not as well established is how families at various stress levels respond to a multi-family group intervention. The primary aim of this study was to examine the influence of caregiver stress on attendance among a group of highly stressed families involved in MFG. Primarily, it is hypothesized that (1) caregivers in a high-stress group will attend significantly fewer MFG sessions as compared to caregivers in the low-stress group. Secondary aims of this study were to evaluate changes in childhood behavioral difficulties and caregiver stress, as well as caregiver depressive symptoms, caregiver coping by substance use, and caregiver motivation to change from baseline to post-MFG. The results of this study are expected to have broader implications for future research and practice related to the implementation of family-based mental health services for highly stressed families.

Method

Procedures

Existing data from the “Family groups for urban youth with disruptive behavior” clinical trial (PI: McKay, R01 MH072649) funded by the National Institute of Mental Health were used to assess outcomes for a subsample of families with the highest stressed caregivers versus outcomes for families with less stressed caregivers. Families in the original clinical trial were recruited from October 2006 to October 2010 from 13 different community-based outpatient child mental health clinics in the greater New York City area and randomized to either the treatment (MFG) or control (services as usual). The Institutional Review Board at Mount Sinai School of Medicine approved use of the data for secondary analysis in this study, and the data were void of any identifiable information. Preliminary outcomes from the “Family groups for urban youth with disruptive behavior” clinical trial are published elsewhere (Chacko et al., 2015), but unique to this article is its exclusive focus on outcomes for high-stress families, for whom prior authors suggest treatment often fails (Weisz et al., 2006).

Sample and Recruitment

The current study included a total of N = 191 families (caregivers and their children) from 13 community-based clinics who were enrolled in the intervention arm of the “Family groups for urban youth with disruptive behavior” clinical trial, and all received the MFG intervention through their outpatient community mental health clinic. Table 1 details the demographic information pertaining to the sample in this study. Of note, the comparison of low stress versus high stress remains the focus of the implications for this study, despite the inclusion of data from mid-stress caregivers in the analysis for comparative purposes and to aid interpretability of study findings. Also of note is that with the exception of caregiver education status in Table 1 (middle-stress caregivers reported a smaller proportion of “eighth grade or less” as compared to low-stress and high-stress caregivers), there are no statistically meaningful differences in demographic characteristics across stress groups on all baseline characteristics.

Table 1.

Demographic Characteristics of Study Participants by Caregiver Stress Group.

	Total (n = 191)		Low Stress (n = 47)		Mid Stress (n = 88)		High Stress (n = 56)
Demographic Characteristic	n ^a	%^b	n ^a	%^b	n ^b	%^b	n ^b	%^b
Caregiver age (mean ± SD)	191	35.2 ± 7.6	47	33.8 ± 7.9	88	36.5 ± 7.4	56	34.3 ± 7.6
Primary caregiver
Mother	155	81	36	77	77	88	42	75
Father	4	2	1	2	1	1	2	4
Grandparent	4	2	0	0	2	2	2	4
Mother and father	4	11	7	15	6	7	8	14
Other	6	3	3	6	2	2	1	2
Caregiver marital status
Single	77	40	16	34	35	40	26	46
Married or cohabitating	72	38	22	47	29	33	21	38
Divorced	6	3	0	0	5	6	1	2
Separated	27	14	7	15	15	17	5	9
Widowed	4	2	2	4	1	1	1	2
Other	3	2	0	0	1	1	2	3
Caregiver education status
Eighth grade or less^c	23	12	10	21	5	6	8	14
Some high school	54	28	20	43	20	23	14	25
Completed HS/GED	41	21	8	17	23	26	10	18
Some college	44	23	7	15	21	24	16	29
Completed college	14	7	2	4	8	9	4	7
Some grad/Prof. school	4	2	0	0	2	2	2	4
Completed grad/Prof. school	9	5	0	0	8	9	1	2
Caregiver ethnicity
White/Caucasian	17	9	2	4	8	9	7	12
Black/African American	53	28	12	26	28	32	13	22
Hispanic/Latino	107	56	29	62	46	52	32	57
Native American	2	1	1	1	1	1	0	0
Asian/Pacific Islander	1	1	0	0	1	1	0	0
Other	10	5	3	6	3	3	4	7
Caregiver employment status
Employed full time	44	23	7	15	23	26	14	25
Employed part time	37	19	9	19	21	24	7	13
Student	13	7	3	6	5	6	5	9
Retired	3	2	1	2	1	1	1	2
Disabled	23	12	11	23	5	6	7	13
Unemployed	62	32	14	30	29	33	19	34
Other	8	4	1	2	4	4	3	5
Family income
Less than $9,999	83	43	26	55	37	42	20	36
US$10,000–$19,999	45	24	7	15	21	24	17	30
US$20,000–$29,999	26	14	7	15	10	11	9	16
US$30,000–$39,999	13	7	4	9	6	7	3	5
US$40,000–$49.999	3	2	0	0	2	2	1	2
Over US$50,000	13	7	1	2	7	8	5	9

Note. HS = high school; GED = general equivalency diploma. ^aNot all totals sum to n = 191 due to missing data. ^bPercentages based on total sample (n = 191), representing low- and high-stress caregivers. ^cMiddle-stress caregivers reported a smaller proportion of “Eighth grade or less” versus low-stress and high-stress caregivers. Otherwise, no other between-group differences are significant at the p = .05 level.

Primary inclusion criteria included (1) youth between the age of 7 and 11 and (2) youth meeting diagnostic criteria for oppositional defiant disorder (ODD) or conduct disorder (CD) as defined in Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision (American Psychiatric Association, 2000), which were assessed by a trained research assistant. Exclusion criteria included youth with mental retardation and/or any caregivers with cognitive limitations that would make it impossible to actively participate in the MFG intervention.

Of the 13 total clinics involved in this study, the majority of the clinics (n = 9) received 16 weeks of MFG, whereas four (n = 4) of them received a condensed, 12-week version of MFG to accommodate both clinic and family needs around feasibility. Additionally, the current study reports attendance and outcomes only for families assigned to the treatment (MFG) arm of the previous study (n = 191). Figure 1 depicts the sampling strategy.

Figure 1.

Sampling strategy: Stratification by parent stress index score.

Measures

Caregiver stress

Caregiver stress was assessed using the Parent Stress Index–Short Form (PSI-SF; Abidin, 1995), which is a 36-item assessment that examines parenting stress across three subscales, from which a total stress score is derived: parental distress, parent–child dysfunctional interaction, and difficult child (Abidin, 1995). This measure and its subscales have been well validated for internal consistency (Cronbach’s α = .83) and reliability (Haskett, Ahern, Ward, & Allaire, 2006; Reitman, Currier, & Stickle, 2002) across several settings, and the total score is often used a measure of general caregiver distress. Response Items 1 to 33 are scored from 1 to 5 (strongly disagree to strongly agree), and example items include the following: “I often have the feeling that I cannot handle things very well;”“I expected to have a closer and warmer feeling for my child than I do and this bothers me;” “Sometimes my child does things that bother me just to be mean.” Questions 33 to 36 are reverse scored from 5 to 1, and an example question is “I feel that I am: (response items 1 to 5 range from not very good at being a parent to a very good parent).

High-, mid-, and low-stress categories were derived in this study by calculating percentile scores from the total score on the PSI (low-stress group—25th percentile; mid-stress group—middle 50th percentile; high-stress group—75th percentile). When coded as a continuous variable in this study, higher scores indicate higher amounts of caregiver stress (Reitman et al., 2002). Additionally, a cutoff score of 90 or above on the PSI indicates that there may be clinically significant caregiver-level impairment (Schaeffer et al., 2005).

Attendance

Attendance (number of weeks attended) was assessed weekly as “Yes” (attended MFG) or “No” (did not attend MFG) by the MFG group facilitator.

Childhood behavioral difficulties

Childhood behavioral difficulties were assessed using the IOWA Conners Rating Scale (IOWA CRS), which aimed to assess youth disruptive behavioral symptoms (Waschbusch & Willoughby, 2008). Caregivers completed this 10-item self-report scale that included subscales regarding behaviors inherent in children with ODD. Behaviors were rated on a scale from 0 to 3 (e.g., Not at all to Very much). Specifically, ODD symptomatology was captured on the oppositional defiant subscale (Items 6 to 10) of the IOWA CRS, and scores ranged from 0 to 15. Sample statements on the ODD subscale include the following: “defiant;” “uncooperative;” and temper outbursts, with explosive and unpredictable behavior.” Cronbach’s α = .80. When scored as a continuous variable, higher scores indicate a greater amount of childhood behavioral difficulties. Additionally, prior studies exist (Pelham, Milich, Murphy, & Murphy, 1989) comparing IOWA CRS scores against normative data to establish cutoff scores from teacher report, which indicate that youth behaviors may have reached levels of clinical concern. The cutoff score for IOWA CRS for children kindergarten through third grade is 9 and for fourth- and fifth-grade children, the cutoff score is 6 (Pelham et al., 1989).

Caregiver depression

Caregiver depression was assessed using the Center for Epidemiological Studies–Depression Scale (CES-D; Radloff, 1977). The CES-D is a 20-item assessment that examines depression across four subscales, which can be added together to obtain a total depression score: Depressive Affect, Somatic Symptoms, Positive Affect, and Interpersonal Problems (Radloff, 1977). This measure has been validated both in the general public and in psychiatric populations and has been shown to have high interval validity (Cronbach’s α = .85–.90; Radloff, 1977). There are 20 response items with response ranging from 0 (rarely/none of the time) to 3 (most/all of the time) and 4 of the items are worded positively and reverse scored for added reliability of caregiver’s responses. Example items include the following: “I felt depressed;” “I was happy;” “I felt people disliked me;” and “I could not get going.” When coded as a continuous variable, higher scores indicate higher amounts of caregiver stress. Additionally, existing normative literature on this measure indicates that a score of 16 or above indicates that a caregiver may be exhibiting clinically significant levels of depression (Radloff, 1977).

Caregiver coping by substance use

Caregiver Coping by Substance Use was assessed using the COPE Inventory (Carver, Scheier, & Weintraub, 1989), which is a 28-item measure aimed at assessing adults’ coping strategies when faced with stressors. Carver describes coping strategies as sometimes being “problematic,” while at other times being adaptive (Carver, 1997; Carver et al., 1989). This measure was adapted from the longer 60-question COPE measure (Carver et al., 1989) and since has been adapted into an updated “brief COPE” measure (Carver, 1997). The COPE substance use subscale includes response items (scored 1–4) from “Don’t do this at all” to “Do this a lot.” Example items include the following: “I’ve been using alcohol or other drugs to make myself feel better” and “I’ve been using alcohol or other drugs to help me get through it” (Carver, 1997). Cronbach’s α = .90.

Caregiver motivation to change

Caregiver motivation to change was assessed using the “Program Satisfaction” subscale of the Metropolitan Area Child Study Process Measure (Tolan, Hanish, McKay, & Dickey, 2002), which is a validated tool used in the child services research that is designed to assess the processes that are associated with positive intervention effects as perceived by caregivers. The Program Satisfaction subscale has been identified by Tolan, Hanish, McKay, and Dickey (2002) as capturing key aspects of caregiver motivation and positive intervention effects within child mental health interventions, and in this study, it was used as a proxy for motivation to change. Tolan et al. describe “motivation to improve” as a driving component of program satisfaction along with how well the caregiver and child bond with the therapist, as Tolan and colleagues assert that children and caregivers are more likely to return to mental health services and will be more motivated to make improvements if they have not only built an alliance with the therapist (or in some cases, the child builds an alliance with the caregiver) but also if the services are perceived as useful.

The subscale used in this study contains 14 questions rated on a scale from 1 to 4 ranging from Not at all to Very much, and includes the following sample questions: “Even though I have been frustrated, I have kept on trying to reach my goals;” “In group, I have tried hard to improve;” “I usually look forward to the next session;” and “I believe the group is helping my child.” Cronbach’s α = .89. When coded and analyzed as a continuous variable, higher scores indicate increased motivation to change.

Data Analysis

Descriptive statistics and univariate analyses were used to analyze the data. Of note, 4 of the 13 urban child mental health clinics involved in this study opted for a 12-week version of the MFG intervention instead of the full 16-week version as offered by the majority of the clinics. To aid in the comparison and generalization of these findings to the entire MFG sample, total attendance was standardized across all clinics (converted to percentages and then multiplied by 16). As a result, total attendance in this study is based on a standardized (16-week) measure of attendance.

When conducting inferential statistics to test whether mean attendance was equal across the caregiver stress groups, a series of independent samples t-tests were used to compare mean attendance between caregiver stress groups. Additionally, both the paired samples t-tests (Cody & Smith, 2005) for normally distributed data and the Wilcoxon signed rank test (Wilcoxon, 1945) for nonparametric data were used to assess significant median/mean differences between baseline (pre) and posttest measures to detect any significant within-group differences in the primary outcome variables (Table 2). In both cases, these tests are optimized for analyzing “matched” or pre/post measures on the same set of research subjects. Results of these pre/post differences tests are summarized numerically in Table 3 for the main outcomes of interest, stratified by caregiver stress group (the horizontal dotted line represents the clinically significant threshold, as described in the literature). Of note, outcomes for caregiver motivation to change were only assessed from mid-test to posttest. Additionally, subgroup totals (n) may sum to less than study total (N = 191) due to attrition, resulting in missing paired data at posttest assessment.

Table 2.

Mean and Median Attendance Between Stress Groups.

	Low Stress Caregivers	Mid Stress Caregivers	High Stress Caregivers
n	47	88	56
Mean	10.40	9.62	8.34
95% CI	[9.01, 11.79]	[8.54, 10.71]	[6.86, 9.82]
SD	4.68	5.09	5.51
Median	12	11	8
IQR	11–13	10–13	5–12

Note. CI = confidence interval; SD = standard deviation; IQR = interquartile range; Median = time point (week) at which 50% of families have dropped out of MFG. Attendance was recorded weekly by group facilitators. Pairwise comparison of mean attendance between low caregiver stress versus high stress attendance is significant at p ≤ .05. There are no other significant pairwise comparisons.

Table 3.

Within-Group Pre/Post Mean Differences—Stratified by Caregiver Stress Group.^a

	Low Stress Caregiver						Mid Stress Caregivers						High Stress Caregivers
	n	Mean (SD)	Median	% Diff	SE	p	n	Mean (SD)	Median	% Diff	SE	p	n	Mean (SD)	Median	% Diff	SE	p
Caregiver stress^b	39	0.49 (12.38)	0	0.7	1.98	.79	64	8.19 (19.45)	8	8.38	2.43	.0003***	35	19.51 (23.21)	14.0	15.31	3.92	.001***
Caregiver depression^c	37	−0.91 (10.26)	−1	7.18	1.69	.57	62	2.55 (8.02)	2	13.76	1.02	.02*	36	5.25 (11.15)	4	20.5	1.86	.001***
Childhood behavioral difficulties^d	36	0.92 (4.08)	0	11.5	0.68	.22	67	1.15 (3.31)	1	13	0.40	.007**	38	2.31 (4.84)	1	20.8	0.78	.01**
Caregiver motivation to change^e,f	33	1.18 (3.1)	2	3.32	0.54	.01**	55	0.96 (4.57)	1	2.78	0.62	.12	35	0.60 (3.88)	0	11.6	0.66	.33
Caregiver coping by substance use^g	38	0.32 (1.21)	0	7.48	0.2	.25	69	0.12 (0.93)	0	2.72	0.11	.48	40	0.40 (2.02)	0	8.6	0.32	.28

Note. % Diff = the percentage of difference between pre/post outcome measures. All statistically significant differences at p ≤ .05 also indicate improved treatment effects. Positive (+) mean difference implies outcomes improved. Negative (−) mean difference implies outcomes worsened.

^aSubgroup totals (n) sum to less than study total (N) due to missing data. ^bParent Stress Index–Short Form (PSI-SF; Abidin, 1995). Coded as a continuous variable. ^cCenter for Epidemiological Studies Depression Scale (CES-D; Radloff, 1977). Coded as a continuous variable. ^dIOWA Conners Rating Scale (Loney & Milich, 1982).Coded as continuous variable. ^eMetropolitan Area Child Study Process Measure (MACS; Program Satisfaction Subscale; Tolan et al., 2002). Coded as a continuous variable. ^fMid/Post (Motivation measures were not assessed at baseline). ^gCOPE Inventory (Coping by Substance Use subscale. Carver et al., 1989). Coded as a continuous variable.

*p ≤ .05. **p ≤ .01. ***p ≤ .001.

Results

Attendance

Independent samples t-tests revealed a statistically significant difference in mean attendance between low and high caregiver stress groups at the p < .05 level; t(100) = 2.04, p = .044 (see Table 2). Mean attendance per caregiver stress group was as follows—low stress: mean = 10.40, SD = 4.68, 95% CI = [9.01, 11.79]; mid stress: mean = 9.62, SD = 5.09, 95% CI = [8.54, 10.71]; high stress: mean = 8.34, SD = 5.51, 95% CI = [6.86, 9.82].

Caregiver Stress

Caregivers in the high caregiver stress group reported a significant decrease in stress from pre to post, with caregivers in the high caregiver stress group reporting a 15.31% decrease in stress symptoms (mean difference = 19.51, SD = 23.21, and p = .001) from baseline to MFG posttest. Caregivers in the low caregiver stress group reported no statistically significant difference in stress from pre to post (see Table 3 and Figure 2).

Figure 2.

Pre/post caregiver stress by caregiver stress group.

Childhood Behavioral Difficulties

Analyses of differences in IOWA CRS outcomes from pre to post revealed that high-stress caregivers reported highly significant decreases in child disruptive behavioral symptoms, with high-stress caregivers reporting a 20.8% decrease in symptoms (mean difference = 2.31, SD = 4.84, and p = .01). Mid-stress caregivers also reported substantial decreases in child behavioral difficulties (13% decrease; mean difference = 1.15, SD = 3.31, and p = .007). Low-stress caregivers did not report a statistically significant difference in pre/post child behavioral outcomes (see Table 3 and Figure 3).

Figure 3.

Pre/post child disruptive behavior by caregiver stress group.

Caregiver Depression, Coping by Substance Use, and Motivation to Change

Pre/post analysis revealed that caregiver depression decreased significantly for caregivers in the high-stress group (20.5% decrease; mean difference = 5.25, SD = 11.15, and p = .001). Caregivers in the mid-stress group also experienced significant decreases in depression symptoms from pre to post (13.76% decrease; mean difference = 2.55, SD = 8.02, and p = .02). On the other hand, caregivers in the low-stress group reported increases in caregiver depression from pre to post; however, these increases were not statistically significant. Despite all caregivers reporting declines in coping by substance use at posttest, none of the pre/post differences were statistically significant. There were no meaningful pre/post differences in caregiver motivation to change among mid and high-stressed caregivers. However, low-stressed caregivers experienced statistically significant increases in motivation to change. Low-stressed caregivers reported a 3.32% increase in motivation to change at posttest (mean difference = 1.18, SD = 3.1, p = .01).

Post Hoc Analyses

Two post hoc analyses were conducted. First, to account for missing data and potential selection bias in the mean differences between pre- and post-MFG outcomes, missing data for posttest measures were imputed using the lost observation carried forward method, followed by use of either the Wilcoxon signed rank test or paired samples t-tests to assess significant median/mean differences between baseline and posttest measures to detect any significant within-group differences in the primary outcome variables. Results of these pre/post differences tests are summarized numerically in Table 4. Compared to the outcomes in Table 3, pre/post outcomes that include imputed values (Table 4) display smaller mean values and smaller standard errors, which were expected. However, despite imputing missing data onto the posttest outcomes and statistically accounting for potential bias, the statistical significance of each pre/post difference remains unchanged.

Table 4.

Post Hoc Within-Group Pre/Post Mean Differences—Stratified by Caregiver Stress Group With Imputed Missing Data.^a

	Low Stress Caregiver						Mid Stress Caregivers						High Stress Caregivers
	n	Mean (SD)	Median	% Diff	SE	p	n	Mean (SD)	Median	% Diff	SE	p	n	Mean (SD)	Median	% Diff	SE	p
Caregiver stress^a	47	0.40 (11.25)	0	0.6	1.64	.79	88	5.95 (16.96)	0	8.17	1.81	.0003***	56	12.20 (20.59)	2	14.80	2.75	.001***
Caregiver depression^b	43	−0.79 (9.51)	0	7.01	1.45	.57	83	1.90 (8.02)	0	12.40	0.77	.02*	53	3.57 (9.48)	0	19.63	1.30	.001***
Childhood behavioral difficulties^c	44	0.75 (3.70)	0	11.1	0.56	.22	82	0.94 (3.02)	0	12.5	0.33	.007**	53	1.66 (4.22)	0	20.1	0.58	.01**
Caregiver motivation to change^d,e	33	1.18 (3.1)	2	3.32	0.54	.01**	55	0.96 (4.57)	1	2.78	0.62	.12	35	0.60 (3.88)	0	11.6	0.66	.33
Caregiver coping by substance use^f	47	0.26 (1.09)	0	7.41	0.16	.25	86	0.09 (0.93)	0	1.91	0.09	.48	55	0.29 (1.73)	0	7.9	0.23	.28

Note. % Diff = the percentage of difference between pre/post outcome measures. All statistically significant differences at p ≤ .05 also indicate improved treatment effects. Positive (+) mean difference implies outcomes improved. Negative (−) mean difference implies outcomes worsened.

^aMissing data have been imputed for these analyses. ^bParent Stress Index–Short Form (PSI-SF; Abidin, 1995). Coded as a continuous variable. ^cCenter for Epidemiological Studies Depression Scale (CES-D; Radloff, 1977). Coded as a continuous variable. ^dIOWA Conners Rating Scale (Loney & Milich, 1982). Coded as continuous variable. ^eMetropolitan Area Child Study Process Measure (MACS; Program Satisfaction Subscale; Tolan et al., 2002). Coded as a continuous variable. ^fMid/Post (Motivation measures were not assessed at baseline). ^gCOPE Inventory (Coping by Substance Use subscale. Carver et al., 1989). Coded as a continuous variable.

*p ≤ .05. **p ≤ .01. ***p ≤ .001.

Additionally, post hoc analyses were conducted to estimate potential effects of regression to the mean on caregiver stress and childhood behavioral difficulties, yielding adjusted pre/post measures of caregiver stress and childhood behavioral difficulties. The results (Table 5) suggest that after adjusting for the expected effects of regression to the mean, pre/post changes in caregiver stress among high-stress caregivers remain statistically significant (mean difference = 13.70, SD = 24.39, and p = .001), as these caregivers reported a 5.54% decrease from pre to post. However, the adjusted value for childhood behavioral difficulties among high-stress caregivers is not statistically significant (p = .12), which suggests that the statistically significant pre/post differences originally reported for childhood behavioral difficulties may in part be explained by regression to the mean (Barnett, van der Pols, & Dobson, 2005).

Table 5.

Post Hoc Outcomes Adjusted for Effect of Regression to the Mean Among High-Stress Caregivers.

	Unadjusted Pre/Post Outcomes						Adjusted for Regression to the Mean
	n	Mean (SD)	Median	% Diff	SE	p	n	Mean (SD)	Median	% Diff	SE	p
Caregiver stress^a	35	19.51 (23.21)	14.0	15.31	3.92	.001***	35	13.70 (24.39)	8.40	5.54	4.12	.001***
Childhood behavioral difficulties^b	38	2.31 (4.84)	1	20.8	0.78	.01**	38	0.09 (0.93)	0.49	6.04	0.82	.12

Note. % Diff = the percentage of difference between pre/post outcome measures. All statistically significant differences at p ≤ .05 also indicate improved treatment effects. Positive (+) mean difference implies outcomes improved. Negative (−) mean difference implies outcomes worsened. Subgroup totals (n) sum to less than study total (N) due to missing data.

^aParent Stress Index–Short Form (PSI-SF; Abidin, 1995). Coded as a continuous variable. ^bIOWA Conners Rating Scale (Loney & Milich, 1982). Coded as continuous variable.

*p ≤ .05. **p ≤ .01. ***p ≤ .001.

Discussion and Future Implications

The existing literature on attendance rates among help-seeking children and their caregivers within child mental health clinics suggests that 48–62% of families do not show up for their initial intake appointment (Mckay, Mccadam, & Gonzales, 1996), and conservative estimates conclude that families’ average length of stay in clinics is four sessions (Harrison et al., 2004). Long term, as few as 9% of youth on average still attend weekly services after 3 months of clinic-based care (McKay & Bannon, 2004). Historically, estimates suggest that 20% to 80% of families end up dropping out of services between initial engagement (e.g., at intake) and completion of most mental health interventions, which can limit accurate estimates of potential treatment effects and long-term outcomes (Ambruster & Kazdin, 1996; Kazdin et al., 1997). Prior work by Kazdin, Holland, and Crowley (1997) indicated that high levels of caregiver stress, perceptions of obstacles to treatment, and negative caregiver perceptions regarding their relationship with the therapist were predictive of premature treatment dropout.

Given current knowledge of the effects of various stressors on family outcomes (Felner et al., 1995; Franco et al., 2010; Gutman, McLoyd, & Tokoyawa, 2005; Kotchick, Dorsey, & Heller, 2005; Kruger, Reischl, & Gee, 2007; Lin, Thompson, & Kaslow, 2009), developing an effective, engaging, and innovative community-based mental health service for families experiencing the highest levels of stress will help these families access and benefit from much-needed community mental health treatment. Prior studies have suggested it is necessary to engage the highest stressed, help-seeking urban families in at least eight sessions to provide an opportunity for families to experience meaningful mental health–related changes (McKay & Bannon, 2004). Insofar as “dose” of MFG, families of every stress group (high stress, mid stress, and low stress) in this study received at least eight “doses” of weekly MFG sessions on average, with the lowest stressed families attending a significantly higher amount of sessions as compared to the highest stressed families. This adds to the literature regarding families’ average length of stay in clinic-based mental health services (Harrison et al., 2004).

Furthermore, despite attending the fewest sessions on average, caregivers in the highest stress group experienced the greatest levels of pre/post improvement on measures of caregiver stress and childhood behavioral difficulties. To that end, one possible interpretation is the highest stressed caregivers may have benefited the “most” as evidenced by exhibiting the most significant decreases in caregiver stress. However, despite making statistically significant improvements in caregiver stress and child conduct symptomology, families in the highest stressed group still exhibited clinically significant caregiver stress and child behavioral symptoms. Measures of caregiver depression followed a similar pattern, with caregivers in the high-stress group exhibiting the greatest statistically significant pre/post decrease; however, this effect diminished when accounting for potential effects of regression to the mean.

In contrast to within-group differences among high-stress caregivers, lowest stressed caregivers began and completed MFG below the clinical cutoff score of 90 on the Parental Stress Index (Schaeffer et al., 2005; Trute & Hiebert-Murphy, 2005). Measures of childhood behavioral difficulties began above the clinical cutoff score on the IOWA Conners (Pelham et al., 1989) for lowest stressed families, but at posttest, these scores were no longer clinically significant, suggesting that MFG achieved the desired childhood disruptive behavioral outcomes for the lowest stressed families. Having said that, further adjustments might need to be made to MFG programming and to the delivery of the MFG treatment modality to optimize clinical outcomes for families of all stress levels. Additionally, pre/post increases in caregiver motivation to change were reported among low-stress caregivers, which could be associated with shifts in caregiver stress, caregiver depression, and levels of childhood behavioral difficulties and potentially making it easier to overcome barriers to attendance (Olin et al., 2010). However, these associations were not explicitly examined in this study.

There were limitations in comparing the high-stress versus low-stress caregiver groups as defined in this study. A primary concern is one of detecting “signal from noise,” as caregivers defined at baseline as “high stress” may not differ substantially from those defined as “low stress” in ways that are clear and easy to define. Furthermore, previous studies suggest that when asked to provide pre- and postparenting stress on the PSI, caregivers are most likely to report that their stress has improved over time, partially as a result of regression to the mean and regardless the degree to which they reported to being stressed at baseline (Barnett et al., 2005). Nevertheless, after accounting for the estimated effects of regression to the mean in post hoc analyses, the analyses in this article suggests significant changes in caregiver stress over time for families involved in MFG. Future studies should incorporate appropriate study design elements that anticipate some of the limitations described in this study.

All of these limitations speak to the challenges in “bridging the gap” between mental health research and mental health practice, and prior publications have cited many of the barriers to achieving the rigor and sample size needed to adequately carry out ideal studies with community-based populations (Weisz et al., 2005). Future qualitative studies exploring caregivers’ perceptions of their experiences in MFG may guide efforts to better tailor the implementation of existing MFG treatment strategy based on families’ stressors and needs. Additionally, future quantitative and mixed methods studies could be enhanced by more advanced study designs and data analytical approaches. Overall, the findings from this study indicate that MFG may be a highly engaging and beneficial treatment modality for families with children with conduct problems as well as caregivers at all stress levels.

Footnotes

Declaration of Conflicting Interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: National Institutes of Health: R01 MH072649 (PI: McKay), F31 MH095335 (PI: Jackson), and K12 GM000680 (PI: Eaton).

References

Abidin

R. R.

(1995). Parenting Stress Index: Professional manual (3rd ed.). Odessa, FL: Psychological Assessment Resources.

Ambruster

Kazdin

A. E.

(1996). Dropping out of child psychotherapy: Issues for research and implications for practice. Clinical Child Psychology and Psychiatry, 1, 133–156.

Angold

Costello

E. J.

(2001). The epidemiology of depression in children and adolescents. In Goodyer

(Ed.), The depressed child and adolescent: Developmental and clinical perspectives (2nd ed., pp. 143–178). New York, NY: Cambridge University Press.

American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (Revised 4th ed.). Washington, DC: Author.

Barnett

A. G.

van der Pols

J. C.

Dobson

A. J.

(2005). Regression to the mean: What it is and how to deal with it. International Journal of Epidemiology, 34, 215–220.

Bassarath

(2001). Conduct disorder: A biopsychosocial review. Journal of the American Academy of Child and Adolescent Psychiatry, 46, 609–616.

Brown

Cohen

Johnson

J. G.

(1998). A longitudinal analyisis of risk factors for child maltreatment: Findings of a 17-year prospective study of officially recorded and self-reported chld abuse and neglect. Child Abuse & Neglect, 22, 1065–1078.

Carver

C. S.

(1997). You want to measure coping but your protocol’s too long: Consider the brief COPE. International Journal of Behavioral Medicine, 4, 92–100.

Carver

C. S.

Scheier

M. F.

Weintraub

J. K.

(1989). Assessing coping strategies: A theoretically based approach. Journal of Personality and Social Psychology, 56, 267–283.

10.

Chacko

Gopalan

Franco

Dean Assael

Jackson

Marcus

Hoagwood

McKay

(2015). Multiple family group service model for children with disruptive behavior disorders: Child outcomes at post-treatment. Journal of Emotional and Behavioral Disorders, 23, 67–77.

11.

Chaffin

Kelleher

Hollenberg

(1996). Onset of physical abuse and neglect: Psychatric, substance abuse, and social risk factors from prospective community data. Child Abuse & Neglect, 20, 191–203.

12.

Cody

R. P.

Smith

J. K.

(2005). Applied statistics and the SAS programming language (5th ed.). Upper Saddle River, NJ: Pearson Prentice Hall.

13.

Costello

E. J.

Mustillo

Erkanli

Keeler

Angold

(2003). Prevalence and development of psychiatric disorders in childhood and adolescence. Archives of General Psychiatry, 60, 837–844.

14.

Culp

R. E.

Culp

A. M.

Soulis

Letts

(1989). Self-esteem and depression in abusive, neglecting, and non-maltreating mothers. Infant Mental Health Journal, 10, 243–251.

15.

Department of Health and Human Services. (2012). Expenditures for treatment of mental health disorders among children, ages 5-17, 2007-2009: Estimates for the U.S. Civilian Noninstrutionalized Population (pp. 1–7). Retrieved from http://meps.ahrq.gov/data_files/publications/st357/stat357.pdf

16.

Dinwiddle

S. H.

Bucholz

K. K.

(1993). Psychatric diagnoses of self-reported child abusers. Child Abuse & Neglect, 17, 465–476.

17.

Felner

R. D.

Brand

DuBois

D. L.

Adan

A. M.

Mulhall

P. F.

Evans

E. G.

(1995). Socio-economic disadvantage, proximal environmental experiences, and socioemotional and academic adjustment in early adolescence: Investigation of a mediated effects model. Child Development, 66, 774–792.

18.

Fleck-Henderson

(2000). Domestic violence in the child protection system: Seeing double. Children and Youth Services Review, 22, 333–354.

19.

Franco

L. M.

Dean-Assael

K. M.

McKay

M. M.

(2008). Multiple family groups to reduce youth behavioral difficulties. In LeCroy

C. W.

(Ed.), Handbook of evidence-based treatment manuals for children and adolescents (pp. 546–590). New York, NY: Oxford Press.

20.

Franco

L. M.

Pottick

K. J.

Huang

C. C.

(2010). Early parenthood in a community context: Neighborhood conditions, race–ethnicity, and parenting stress. Journal of Community Psychology, 38, 574–590.

21.

Frick

P. J.

(1998). Callous-unemotional traits and conduct problems: A two-factor model of psychopathy in children. In Hare

R. D.

Cooke

D. J.

Forth

(Eds.), Psychotherapy: Theory, research, and implications for society (pp. 161–187). Dordrecht, the Netherlands: Kluwer.

22.

Ghandour

R. M.

Kogan

M. D.

Blumberg

S. J.

Jones

J. R.

Perrin

J. M.

(2012). Mental health conditions among school-aged children: Geographic and sociodemographic patterns in prevalence and treatment. Journal of Developmental and Behavioral Pediatrics, 33, 42–54.

23.

Gopalan

Bannon

Dean-Assael

Fuss

Gardner

LaBarbera

McKay

M. M.

(2011). Multiple family groups: An engaging intervention for child welfare-involved families. Child Welfare, 90, 135–156.

24.

Gopalan

Franco

L. M.

(2009). Multiple family approach. In Gitterman

Salmon

(Eds.), Encyclopedia of social work with groups (pp. 86–88). New York, NY: Routeldge.

25.

Gopalan

Goldstein

Klingenstein

Sicher

Blake

McKay

M. M.

(2010). Engaging families into child mental health treatment: Updates and special considerations. Journal of the Canadian Academy of Child and Adolescent Psychiatry, 19, 182–196.

26.

Gutman

McLoyd

Tokoyawa

(2005). Financial strain, neighborhood stress, parenting behaviors, and adolescent adjustment in urban African American families. Journal of Research on Adolescence, 15, 229–245.

27.

Harrison

M. E.

McKay

M. M.

Bannon

W. M.

(2004). Inner-city child mental health service use: The real question is why youth and families do not use services. Community Mental Health Journal, 40, 119–131.

28.

Haskett

M. E.

Ahern

L. S.

Ward

C. S.

Allaire

J. C.

(2006). Factor structure and validity of the Parenting Stress Index-Short Form. Journal of Clinical Child & Adolescent Psychology, 35, 302–312.

29.

Hill

T. D.

Ross

C. E.

Angel

R. J.

(2005). Neighborhood disorder, psychophysiological distress, and health. Journal of Health and Social Behavior, 46, 170–186.

30.

Hoagwood

K. E.

Jensen

P. S.

Acri

M. C.

Olin

Lewandowski

E. R.

Herman

R. J.

(2012). Outcome domains in child mental health research since 1996: Have they changed and why does it matter? Journal of the American Academy of Child and Adolescent Psychiatry, 51, 1241–1260.

31.

Kaplan

S. J.

Pelcovitz

Salzinger

Caneles

(1983). Psychopathology of parents of abused and neglected children and adolescents. Journal of the American Academy of Child Psychiatry, 22, 238–244.

32.

Kazdin

A. E.

Blase

S. L.

(2011). Rebooting psychotherapy research and practice to reduce the burden of mental illness. Perspectives on Psychological Science, 6, 21–37.

33.

Kazdin

A. E.

Holland

Crowley

(1997). Family experience of barriers to treatment and premature termination from child therapy. Journal of Consulting and Clinical Psychology, 65, 453–463.

34.

Kazdin

A. E.

Whitley

M. K.

(2003). Treatment of parental stress to enhance therapeutic change among children referred for aggressive and antisocial behavior. Journal of Consulting and Clinical Psychology, 71, 504–515.

35.

Kotchick

B. A.

Dorsey

Heller

(2005). Predictors of parenting among African American single mothers: Personal and contextual factors. Journal of Marriage and Family, 67, 448–460.

36.

Kruger

D. J.

Reischl

T. M.

Gee

G. C.

(2007). Neighborhood social conditions mediate the association between physical deterioration and mental health. American Journal of Community Psychology, 40, 261–271.

37.

Lin

Thompson

M. P.

Kaslow

N. J.

(2009). The mediating role of social support in the community environment—Psychological distress link among low-income African American women. Journal of Community Psychology, 37, 459–470.

38.

Loeber

Burke

J. D.

Lahey

B. B.

Winters

Zera

(2000). Oppositional defiant and conduct disorder: A review of the past 10 years, part I. Journal of the American Academy of Child and Adolescent Psychiatry, 39, 1468–1484.

39.

Loney

Milich

(1982). Hyperactivity, inattention and aggression in clinical practice. In Woolraich

(Ed.), Advances in developmental and behavioral pediatrics (pp. 113–147). Greenwich, CT: JAI.

40.

McKay

M. M.

Bannon

W. M.

(2004). Engaging families in child mental health services. Child and Adolescent Psychiatric Clinics of North America, 13, 905–921.

41.

McKay

M. M.

Gonzales

Quintana

Kim

Abdul-Adil

(1999). Multiple family groups: An alternative for reducing disruptive behavioral difficulties of urban children. Research on Social Work Practice, 9, 593–607.

42.

McKay

M. M.

Gonzales

J. J.

Stone

Ryland

Kohner

(1996). Multiple family therapy groups: A responsive intervention model for inner city families. Social Work with Groups, 18, 41–56.

43.

McKay

M. M.

Gopalan

Franco

Assael

K. D.

Chacko

Jackson

Fuss

(2011). A collaboratively designed child mental health service model: Multiple family groups for urban children with conduct difficulties. Research on Social Work Practice, 21, 664–674.

44.

McKay

M. M.

Gopalan

Franco

L. M.

Kalogerogiannis

Umpierre

Olshtain-Mann

… Goldstein

(2010). It takes a village to deliver and test child and family-focused services. Research on Social Work Practice, 20, 476–482.

45.

McKay

M. M.

Harrison

M. E.

Gonzales

Kim

Quintana

(2002). Multiple-family groups for urban children with conduct difficulties and their families. Psychiatric Services, 53, 1467–1468.

46.

McKay

M. M.

Hibbert

Hoagwood

Rodriguez

Murray

Legerski

Fernandez

(2004). Integrating evidence-based engagement interventions into “real world” child mental health settings. Brief Treatment and Crisis Intervention, 4, 177–186.

47.

Mckay

M. M.

Mccadam

Gonzales

J. J.

(1996). Addressing the barriers to mental health services for inner city children and their caretakers. Community Mental Health Journal, 32, 353–361.

48.

Mills

L. G.

Friend

Conroy

Fleck-Henderson

Krug

Magen

R. H.

… Trudeau

J. H.

(2000). Child protection and domestic violence: Training, practice, and policy issues. Children and Youth Services Review, 22, 315–332.

49.

Monuteaux

M. C.

Faraone

S. V.

Gross

M. L.

Biederman

(2007). Predictors, clinical characteristics, and outcome of conduct disorder in girls with attention-deficit/hyperactivity disorder: A longitudinal study. Psychological Medicine, 37, 1731–1741.

50.

Olin

S. S.

Hoagwood

K. E.

Rodriguez

Ramos

Burton

Penn

… Jenson

P. S.

, (2010). The application of behavior change theory to family-based services: Improving parent empowerment in children's mental health. Journal of Child and Family Studies, 19, 462–470.

51.

Pelham

W. E.

Milich

Murphy

D. A.

Murphy

H. A.

(1989). Normative data on the IOWA Conners teacher rating scale. Journal of Clinical Child Psychology, 18, 259–262.

52.

Prinz

R. J.

Miller

G. E.

(1994). Family-based treatment for childhood antisocial behavior: Experimental influences on dropout and engagement. Journal of Consulting and Clinical Psychology, 62, 645–650.

53.

Prinz

R. J.

Miller

G. E.

(1996). Parental engagement in interventions for children at risk for conduct disorder. In Peters

R. D.

McMahon

R. J.

(Eds.), Preventing childhood conduct disorders, substance abuse and delinquency (pp. 161–183). Thousand Oaks, CA: Sage.

54.

Radloff

L. S.

(1977). The CES-D Scale: A self-report depression scale for research in the general population. Applied Psychological Measurement, 1, 385–401.

55.

Reitman

Currier

R. O.

Stickle

T. R.

(2002). A critical evaluation of the Parenting Stress Index-Short Form (PSI-SF) in a head start population. Journal of Clinical Child & Adolescent Psychology, 31, 384–392.

56.

Russell

Harris

Gockel

(2008). Parenting in poverty: Perspectives of high-risk parents. Journal of Children and Poverty, 14, 83–98.

57.

Schaeffer

C. M.

Bruns

Weist

Stephan

S. H.

Goldstein

Simpson

(2005). Overcoming challenges to using evidence-based interventions in schools. Journal of Youth and Adolescence, 34, 15–22.

58.

Stone

McKay

M. M.

Stoops

(1996). Evaluating multiple family groups to address the behavioral difficulties of urban children. Small Group Research, 27, 398–415.

59.

Tolan

P. H.

Hanish

L. D.

McKay

M. M.

Dickey

M. H.

(2002). Evaluating process in child and family interventions: Aggression prevention as an example. Journal of Family Psychology, 16, 220–236.

60.

Tolan

P. H.

Henry

(1996). Patterns of psychopathology among urban poor children: Comorbidity and aggression effects. Journal of Consulting and Clinical Psychology, 64, 1094–1099.

61.

Trute

Hiebert-Murphy

(2005). Predicting family adjustment and parenting stress in childhood disability services using brief assessment tools. Journal of Intellectual and Developmental Disability, 30, 217–225.

62.

Waschbusch

D. A.

Willoughby

M. T.

(2008). Parent and teacher ratings on the IOWA Conners Rating Scale. Journal of Psychopathology and Behavioral Assessment, 30, 180–192.

63.

Weisz

J. R.

Doss

A. J.

Hawley

K. M.

(2005). Youth psychotherapy outcome research: A review and critique of the evidence base. Annual Review of Psychology, 56, 337–363.

64.

Weisz

J. R.

Jensen-Doss

Hawley

K. M.

(2006). Evidence-based youth psychotherapies versus usual clinical care: A meta-analysis of direct comparisons. The American Psychologist, 61, 671–689.

65.

Wilcoxon

(1945). Individual comparisons by ranking methods. Biometrics Bulletin, 1, 80–83.