Pandemic parenting: A pilot study of in-person versus internet-DOCS K-5 for caregivers of school-age children with disruptive behaviors

Abstract

Behavioral parent training (BPT) programs are needed to address disruptive behavior disorders among school-aged children. Given the prolonged COVID-19 pandemic and associated mental health consequences, adapting BPTs to telehealth modalities is necessary to ensure continued services to children and families. This pilot study evaluated the use of a telehealth vs in-person modality to deliver the Developing Our Children’s Skills K-5 (DOCS K-5) BPT. Participants were caregivers of children enrolled in elementary school exhibiting disruptive behaviors who participated in either in-person DOCS K-5 (n = 21) or internet-DOCS K-5 (i-DOCS K-5; n = 34). Pre- and post-intervention outcome measures were collected for child disruptive behavior, parenting stress, and caregiver symptoms of depression while consumer satisfaction was assessed at post-test only. Multiple linear and Poisson regression models were performed to assess the effect of session modality on the outcomes. Child disruptive behavior, parenting stress and depression, and consumer satisfaction scores were not significantly different across groups, even after adjusting for baseline characteristics. The results of this study provide preliminary evidence that the i-DOCS K-5 modality is as effective as the in-person program. Study findings may be beneficial to practitioners treating school-age children and utilizing telehealth interventions during the COVID-19 pandemic and onward.

Keywords

parent training HOT DOCS telehealth early intervention COVID-19

Introduction

Approximately 7.4% of school-aged children have a diagnosed disruptive behavior disorder, with children ages 6–11 years demonstrating the highest rates of challenging behaviors (Ghandour et al., 2019). Disruptive behaviors can manifest as physical and verbal aggression, noncompliance with adult demands, and other disruptive behaviors such as screaming, whining, and arguing. Ineffective caregiver-child interactions reinforce disruptive behaviors, leading to an intensification over time as the child transitions into adolescence and later adulthood (Jones et al., 2013). By adolescence and early adulthood, severe and long-lasting consequences include school failure and drop out, drug and alcohol use, gang activity, violent acts, adult criminality, and higher death and injury rates among others (Walker & Sprague, 1999). Thus, it is critical that parenting strategies and the caregiver-child relationship are a target for behavioral intervention among school-aged children.

Behavioral parent training programs (BPTs) seek to reduce child disruptive behaviors through effective parenting practices. In BPTs, caregivers learn techniques to 1) reduce the intensity and frequency of child disruptive behaviors, 2) improve the quality of the caregiver-child relationship, and 3) manage their own parenting stress (Agazzi et al., 2019; Baumel & Faber, 2018). BPTs are one of the most effective treatments for children with disruptive behaviors because of the focus on healthier caregiver-child interactions and parenting techniques (Chorpita et al., 2011; Leijten et al., 2013). While Florean et al. (2020) identified seven well-studied BPTs for treatment of child disruptive behaviors in their recent meta-analysis, it is well documented that few children between the ages of 3 and 17 have good access to such interventions (Ghandour et al., 2019). Barriers to treatment access relate to transportation, lack of staff trained/certified in the interventions, inconvenience of time of appointments and/or clinic locations, insurance types accepted, etc. (Baumel & Faber, 2018). These barriers to treatment indicate a need for more accessible BPTs for families with school-aged children.

Existing research on in-person BPTs for school-aged youth have demonstrated overall positive outcomes. Some of these programs seek to address behavioral concerns, while other programs address symptoms of specific disorders, including ADHD and autism spectrum disorder (Bearss et al., 2015; Loren et al., 2015). Caregivers of school-aged children have reported decreases in the frequency and intensity of their children’s disruptive behaviors after participating in face-to-face BPTs (Carr et al., 2017; Hobbel & Drugli, 2013; Nobel et al., 2020). Additionally, in-person BPTs lead to increases in parenting confidence and decreases in parenting stress (Loren et al., 2015).

Traditional in-person BPTs were adapted to telehealth modalities prior to the recent COVID-19 pandemic, but have become increasingly in-demand and covered by third-party payers with the onset of the pandemic. Such adaptations were necessary since children with preexisting behavioral, developmental, and mental health concerns are at-higher risk for increased mental health problems during a pandemic and are at-risk for longer-term negative outcomes (Prime et al., 2020; SAMHSA, 2018). These telehealth adaptations have included tools like online modules, smart phone applications, video vignettes, group discussion, and forums (Agazzi et al., 2021; Hewitt & Loring, 2020; Jones et al., 2014; Wainer & Ingersoll, 2015). For example, the telehealth version of the Positive Parenting Program (Triple P Online; Sanders et al., 2012) is an 8-week curriculum consisting of web modules that caregivers can navigate. Another telehealth adaptation of a BPT, Internet Based Parent-Child Interaction Therapy (iPCIT; Comer et al., 2017) allows practitioners to provide remote, live video coaching to caregivers with their children in their homes. The Incredible Years online program (Webster-Stratton et al., 2008; Webster-Stratton et al., 2004) uses video vignettes for caregivers to watch and apply new parenting skills with a variety of situations and behaviors. Finally, a telehealth adaptation of the HOT DOCS program (e.g., i-HOTDOCS; Agazzi et al., 2021) combines group discussion, PowerPoint lecture, video vignettes, and group problem-solving activities through a videoconferencing platform.

The aforementioned interventions have resulted in reduced disruptive behaviors in children as well as improved parenting behaviors and parenting stress (Nieuwboer et al., 2013a; 2013b). Program completers of the telehealth adaptation of Triple P have endorsed improvements in their parenting stress, their children’s disruptive behaviors, and in overall parenting style, and these results were maintained at 6 months follow-up (Baker et al., 2017). Likewise, studies using iPCIT and i-HOT DOCS have demonstrated improvements in child behavior and parenting stress (Agazzi et al., 2021; Comer et al., 2017). Further, telehealth adaptations of BPTs have resulted in high levels of treatment satisfaction, with telehealth implementation scores either similar to or better than scores for in-person groups (Agazzi et al., 2021; Chacko et al., 2016). BPTs adapted for telehealth have also expanded reach to families in underserved geographical areas (Traube et al., 2020). In addition, BPTs delivered using telehealth platforms can help caregivers address their child’s disruptive behaviors in more natural settings such as the home or community (Ros-DeMarize et al., 2021). Recent meta-analyses (Florean et al., 2020; Thongselratch et al., 2020) support the efficacy of online parenting programs, including synchronous and asynchronous self-guided programs, for the reduction of child disruptive behaviors. Florean et al. (2020) also reported a medium effect for reductions in ADHD symptoms, similar to effects of traditionally delivered interventions. Indeed, both reviews documented reductions in parental distress, while Thongseiratch et al. (2020) also reported online programs as effective for improving parental mental health (e.g., anxiety, depression). Previous studies have documented that traditional BPTs reduce parental depression (Furlong et al., 2012), and this effect is worth of attention given the higher rates of depression reported among diverse caregivers (e.g., Hispanic or Latinx; National Center for Health Statistics, 2021). Telehealth adaptations of BPTs have other benefits like cost effectiveness for both consumers and practitioners (Reese et al., 2015) and multimodal dissemination of psychoeducational resources (Agazzi et al., 2021; Jones et al., 2013).

Telehealth BPTs present with a unique set of limitations. First, access to and understanding of relevant technology can be a significant barrier for both caregivers and practitioners, particularly if there is unstable internet connection or no secure network (Ros-DeMarize et al., 2021). Caregivers also may struggle to participate in either telehealth or in-person BPT services simply because they do not know where to locate these services or even that these services exist (Sayal et al., 2010). Finally, family stressors and changes to routines at home may make caregiver completion of self-directed, modularized BPTs more difficult (Baumel & Faber, 2018).

Purpose of study

This is the first pilot study evaluating the Developing Our Children’s Skills K-5 (DOCS K-5) curriculum, a new BPT focused on behavioral concerns among school-aged children, worthy of empirical investigation. The aims of this study were to: 1) describe the development of DOCS K-5 and its telehealth adaptation, 2) evaluate changes in outcomes from pre to post-test for all participants, 3) and compare outcomes between modalities for standard in-person vs. telehealth. We hypothesized that i-DOCS K-5 would not significantly differ from standard clinic-based DOCS K-5 on the following outcomes: (a) treatment satisfaction; (b) child behavior; (c) caregiver stress; (d) caregiver depressive symptoms.

Method

Participants and recruitment

Participants included 55 caregivers ages 18+ who participated in a pilot program of DOCS K-5, either in-person (DOCS K-5) or via telehealth (i-DOCS K-5) modality. Caregivers were recruited through social media advertising, local pediatricians, and elementary schools. To meet inclusion criteria, caregivers must have had children currently enrolled in elementary school for whom they perceived to have disruptive behaviors (e.g., no severity score or diagnosis was required), attended in-person DOCS K-5 or i-DOCS K-5 sessions, signed consent to participate and completed questionnaires. Over a 2-year recruitment, 59 caregivers signed consent and participated in either DOCS K-5 or i-DOCS K-5. Of these, four caregivers dropped out of the in-person program citing lack of childcare and transportation barriers leaving 51 (92.7%) who met program completion (e.g., attended five of six sessions and completed post-test). Listwise deletion for missing data led to a final sample size of 51 complete records at post-test. Comparison of the included and excluded observations found no significant differences. With regards to sample size for pilot studies, our study sample was within the range of recommended sample sizes (i.e., 20–70; Whitehead et al., 2016). A formal power analysis was not conducted due to the nature of the study. Specifically, pilot studies serve to identify and address issues that could occur with respect to future study conceptualization, study design, data collection, data management, and data analysis (Moore et al., 2011). However, the difference in effects we find in pilot studies can be used to plan future power and sample size of a larger trial.

Procedures

The Institutional Review Board at the University of South Florida (Pro00032638) approved this study, and written informed consent was obtained for all participants.

Intervention overview

DOCS K-5 serves as an open intervention for caregivers who perceive their children (e.g., elementary school age) to have disruptive behaviors or simply want to learn how to prevent disruptive behaviors. This means that children do not have to meet specific scores on behavior rating scales or diagnostic criteria for caregiver participation. DOCS K-5 is an extension of the HOT DOCS program, a BPT for caregivers of children ages 0–5 years (Agazzi et al., 2021). Both HOT DOCS and DOCS K-5 use principles of Positive Behavior Support (PBS; Carr et al., 2002). Specifically, the programs follow the PBS principles of 1) focus on the function of behavior and 2) use of functional assessment to inform interventions targeting replacement skills (Dunlap, 2006). Existing HOT DOCS research includes observational studies, a wait-list control trial, nonrandomized pre-post analyses, and qualitative analyses, with results indicating reductions in caregiver-reported child behavior problems, reductions in parenting stress, increases in knowledge of parenting techniques, and high program satisfaction (Agazzi et al., 2010, 2021; Ogg et al., 2014; Salinas et al., 2011).

DOCS K-5 has been delivered through both in-person and telehealth modalities. For in-person delivery, the program has been delivered in various community centers throughout one county in central Florida. Caregivers receive physical copies of the program materials corresponding which each session (see Table 1 for session content). During each session, the instructor engages participants in a brief review and discussion of previously learned material/techniques. Then, the instructor delivers new session content through use of PowerPoint lectures, video vignettes, and group activities and discussion. Finally, caregivers are assigned homework tasks, including 5 minutes of daily quality time (i.e., playing with child) to enhance the parent-child relationship, and a parenting tip that encourages practice of positive parenting strategies.

Table 1.

DOCS K-5 class content, parenting tips.

Class name	Topics discussed	Parenting tip
Understanding child behavior	Behavior basics; problem-solving behavior	Labeled praise
Developing preventions	Strategies to prevent behavior problems	Give clear directions
New skills for children	Teaching children new skills	Teach a new skill
New responses for caregivers	New responses to child behavior	Use follow through/removal of privileges
Strengthening family relationships	Problem-solve adult behavior; develop preventions, new skills, and new responses for families	Hold a family meeting
Advocacy: Health & education	Advocacy in child health & education	Use community resource map

Telehealth technology

With the onset of COVID-19, in-person DOCS K-5 sessions were suspended in favor of a telehealth version administered Microsoft Teams. Each i-DOCS K-5 group consisted of 10–15 caregivers and involved the same content as the in-person program as previously described. Participant materials were mailed to caregivers’ homes, and additional handouts were emailed in PDF format. Caregivers also were provided with technology support as needed, such as helping them install and navigate the functions of Microsoft Teams.

Instructors and materials

Across modalities, sessions were taught by one or two certified instructors. These instructors included a pediatric psychologist, a school psychology graduate student, an infant-toddler developmental specialist, and one school psychology postdoctoral fellow. Instructors completed weekly integrity checklists (average = 100%) to maintain program fidelity and used attendance sheets to document program attrition. Instructors met bi-monthly via teleconference to review data collection, session flow, and to problem-solve any issues.

Overview of data collection

Pre-test measures were completed either by paper-and-pencil in the first in-person session or through Qualtrics links emailed prior to the first in-person or telehealth session. Paper-and-pencil post-test measures were given to caregivers for completion during the final session, or emailed via Qualtrics to caregivers a few days prior to the last meeting to ensure completion prior to session arrival.

Measures

Demographics

Caregiver and child demographic data were collected with a program specific demographic form.

Child behavioral problems

Child behavior problems were assessed at both pre- and post-intervention with the conduct and hyperactivity scales on the Strengths and Difficulties Questionnaire (SDQ; Goodman, 2001) in the first year of this project. This measure contains 25 questions on a 3-point Likert scale, with five items addressing conduct problems and five items addressing hyperactivity. Interrater reliability, internal consistency, and concurrent validity of the SDQ are deemed acceptable (Goodman, 2001). However, due to difficulties using a broad-band measure to evaluate short-term improvement of child behavior, the program adopted the Eyberg Child Behavior Inventory (ECBI; Eyberg and Pincus, 1999) in year two moving forward. The ECBI contains 36 questions on a 7-point Likert scale that addresses intensity of various child behaviors (e.g., aggression, whining). Scores range from 36 to 252, with a clinical cutoff score of 131 for Intensity scale and 15 for Problem scale. The ECBI has demonstrated high test-retest reliability, internal consistency, and inter-rater reliability (αs = .75, .93, and .79, respectively; (Funderburk et al., 2003) as well as acceptable content and construct validity (Boggs et al., 1990).

Parenting stress

The DOCS Parenting Stress Measure assessed caregiver parenting stress at pre- and post-intervention (DOCS PSM, adapted from the Autism Parenting Stress Index; (APSI; Silva and Schalock, 2012). This measure contains 17 questions on a 4-point Likert scale addressing daily living skills, development, and adaptive functioning, with adaptations to the wording of several items (e.g., changing your child’s diet to your child’s feeding difficulties). Additionally, four items were added to this measure to assess the impact of parenting costs, time, and stressors on other life domains. Scores range from 0–68. In this study, the DOCS PSM demonstrated excellent internal consistency (α = .93). The APSI has been shown to have good internal consistency and test-retest reliability in the past (Silva & Schalock, 2012).

Caregiver depression

The Patient Health Questionnaire-9 (PHQ-9; Kroenke et al., 2001) assessed caregiver depression symptoms. This scale includes 9-items that assess how often in the previous 2 weeks respondents were affected by depressive symptoms (e.g., depressed mood, sleep difficulty, anhedonia). Respondents rate symptoms using a 4-point Likert scale and scores between 5 and 14 may require treatment and scores above 15 warrant treatment. The PHQ-9 has demonstrated high internal consistency (α = .89). The PHQ-9 also has adequate convergent validity and sensitivity to change when compared to the Beck Depression Inventory-II (BDI-II; Titov et al., 2011).

Participant satisfaction

Caregiver satisfaction was evaluated using the Therapy Attitude Inventory (TAI; Eyberg, 2002) administered at post-test. The TAI contains 10 items on a 5-point Likert scale, with scores ranging from 0 (no satisfaction) to 50 (high satisfaction). Previous research has shown that the TAI has excellent reliability, good test-retest reliability, and acceptable internal consistency (αs = .91, .85, and .78, respectively; Brestan et al., 1999).

Session type

The primary independent variable for this study was the modality used for sessions, namely in-person or telehealth. The researchers applied a binary variable system to represent in-person modality (0) and telehealth modality (1).

Attendance

Program instructors recorded attendance with binary coding in an attendance log wherein participant attendance was coded 0 (absent) or 1 (present). This system was used to document which participants attended the five-session minimum to meet program completion criteria.

Statistical analysis

Descriptive analyses

Where multiple caregivers attended the program, the mother was prioritized for analyses. If only the father attended, his data were used. The following analyses were performed using Stata 16 (StataCorp, 2019). Bivariate and univariate analyses were performed on the caregiver-child populations. For continuous and count variables, t-tests were used to compare whether mean scores significantly differed between the in-person and telehealth groups. For binary variables such as race and ethnicity, Chi-square tests of independence (x²) were performed to identify significant associations between session modality and categorical variables.

Overall program effect

In order to assess overall effect of the program on our hypothesized outcomes, paired t-tests were performed to assess whether there were significant differences between the pre- and post-test scores, as well as the direction of change.

Modality comparison

A series of multivariable regression models were performed to assess the effect of session modality (treatment variable) on the outcome variables: for adults, those outcomes were parenting stress, caregiver depression, and program satisfaction; for children, the outcomes were child behavior as measured by the SDQ and ECBI, depending on when participants attended the program (i.e., year 1 or year 2). Assessment of the outcome variables for normality showed that three of the outcomes (parenting stress, caregiver depression, and program satisfaction) were not normally distributed. Log transformation was a sufficient adjustment for the stress variable, square root transformation was used for caregiver depression, and Box Cox transformation was used for program satisfaction. Poisson regression was used for the child behavior outcome (i.e., ECBI generated non-negative integers or count data) that was not transformed, while linear regression was used for the parenting stress (DOCS PSM) and caregiver depression (PHQ-9). Cohen’s d was calculated for these outcomes as a measure of effect size where appropriate. A Wilcoxon Rank-Sum Test was used for the association between modality of delivery and program satisfaction (TAI). As ECBI information was not collected for a subsection of the study population, regressions were still performed, but modality of session could not be included in modeling and effect sizes could not be calculated (i.e., in year 2, we moved to the telehealth only modality and added the ECBI as a more sensitive measure of change).

Demographic variables were included in the regression models to account for possible confounders. These were included as proxies to account for other personal factors that may affect outcomes related to the DOCS K-5 program. We were particularly interested in identifying any gaps that the demographic information might identify for the purposes of strengthening the program to better serve all of the participants. If particular demographic groups were identified as exhibiting significantly less improvement in outcomes, this information would be useful in focusing further program development on better reaching those populations. For the outcomes that measured parenting stress and program satisfaction, caregiver information on gender, ethnicity, and race was used. For the child behavior outcome, child gender, ethnicity, and race were included in the modeling. Gender, ethnicity, and race were all binary measures. The majority of the participants were White; thus, there were insufficient data points to observe multiple race types. For this reason, the binary race variable was White vs. Non-White. Finally, the binary measure of attendance of at least five sessions was included in the modeling. This was included as a measure of participation, with the assumption that those who attended at least five sessions were more likely to have significantly benefited from the program.

We also included the participant’s pre-test scores on the same outcome measures in our regression modeling. The importance of disentangling the effect of an intervention on baseline differences in outcomes is long established, though there are competing theories on the most appropriate manner in which to do so (Wilder, 1950; Wainer, 1991). As we are primarily interested in controlling for the effect of baseline scores on our outcome, rather than the interpretation of those coefficients, we included their baseline pre-test score in our models as a covariate.

We acknowledge that there are issues around using siblings in regression analysis, as it violates the assumption of independence. To account for this, we performed further analysis of a smaller child sample, including only one child for family. With this reduced sample, we ran similar regressions, as well as uncontrolled ANOVAs. As the results of that analysis were similar to the regression using a more robust sample, we chose to present the results of the regression modeling that included siblings.

Results

Table 2 shows the demographic characteristics of the caregivers and their children. A total of 55 participants was included for analysis. As some caregivers had multiple children for whom they attended the program, there were a total of 74 complete child records. Around 38% of participating families attended in-person DOCS K-5 sessions compared to telehealth. The caregivers retained for analysis were overwhelmingly female and 65.5% identified as non-Hispanic/Latinx ethnicity. White race was reported by 78.2% of the sample. Over 90% of the participants attended at least five sessions. The telehealth group had a higher overall proportion of White and African American attendees, and more respondents reported obtaining an associate degree or higher compared to the in-person group. The in-person and telehealth groups did not differ in terms of sociodemographic information, nor in terms of their baseline outcome measures, as described in Table 2. For SDQ outcomes, the ECBI problem scale, and our parenting stress measure, there were significant differences between the pre- and post-test scores.

Table 2.

Participant Characteristics by Type of Program Attended.

	In person n	Telehealth n		Total N
	(column %)	(column %)	p-value	(column %)
Year attended			<0.01
2019	10 (47.6%)	0 (0.0%)		10 (18.2%)
2020	11 (52.4%)	24 (70.6%)		35 (63.6%)
2021	0 (0.0%)	10 (29.4%)		10 (18.2%)
Gender			0.58
Female	19 (90.5%)	29 (85.3%)		48 (87.3%)
Male	2 (9.5%)	5 (14.7%)		7 (12.7%)
Ethnicity			0.88
Not hispanic or latinx	14 (66.7%)	22 (64.7%)		36 (65.5%)
Hispanic or latinx	7 (33.3%)	12 (35.3%)		19 (34.6%)
Race			0.69
White	17 (81.0%)	26 (76.5%)		43 (78.2%)
Other or refused to answer	4 (19.1%)	8 (23.53%)		12 (21.82%)
Attended at least 5 classes	19 (90.5%)	32 (94.1%)	0.62	51 (92.7%)
	Mean (SD)	Mean (SD)	p-value
Child behavior
SDQ pre-test	16.1 (6.9)	15.7 (8.2)	0.84	15.9 (7.5)
SDQ post-test	14.1 (9.1)	15.3 (8.8)	0.60	14.7 (8.9)
SDQ paired t-test			<0.01
SDQ externalizing scale pre-test	10.6 (4.1)	9.9 (4.3)	0.51	10.2 (4.2)
SDQ externalizing scale post-test	8.5 (5.4)	9.0 (4.9)	0.73	8.7 (5.1)
SDQ externalizing scale paired t-test			<0.01
ECBI problem scale pre-test	N/A	20.1 (9.9)	N/A	20.1 (9.9)
ECBI problem scale post-test	N/A	14.9 (8.2)	N/A	14.9 (8.2)
ECBI problem scale paired t-test			<0.01
ECBI intensity scale pre-test	N/A	131.8 (40.6)	N/A	131.8 (40.6)
ECBI intensity scale post-test	N/A	111.0 (35.6)	N/A	111.0 (35.6)
ECBI intensity paired t-test			0.01
Adult participant ratings
Caregiver depression pre-test	4.0 (3.4)	5.2 (5.7)	0.40	4.8 (4.9)
Caregiver depression post-test	3.7 (3.5)	5.3 (5.6)	0.24	4.7 (4.9)
Caregiver depression wilcoxon sign-rank			0.59
Parenting stress scale pre-test	24.6 (11.0)	24.2 (14.8)	0.93	24.4 (13.3)
Parenting stress scale post-test	19.8 (8.3)	20.4 (12.4)	0.84	20.2 (10.9)
Parenting stress wilcoxon sign-rank			<0.01
Program satisfaction total raw score	41.5 (6.6)	42.6 (5.3)	0.51	42.1 (5.8)

Note. Total participants N = 55; In-person n = 21; Telehealth n = 34; Total child records n = 74; Child behavior = ECBI; Parenting stress = DOCS PSM; Program Satisfaction = TAI.

Program outcomes

Table 3 shows the results of the regression models for caregiver outcomes except for the TAI score. Post-test score differences for child behavior and caregiver stress were not significantly different across modality of session attended. Child behavior pre-test scores significantly predicted child behavior post-test scores, but the effect size was small (β = .04, p < .01 and β =.15, p < .01). Hispanic or Latinx caregivers had significantly higher caregiver depressions scores (PHQ-9; β = 0.69, p < 0.01), and those who reported attending at least five sessions also scored higher on caregiver depression (PHQ-9; β = 1.01, p < .05). Again, effect sizes were small.

Table 3.

Regression results for adult outcomes.

	Parenting stress outcome (log transformed)		Caregiver depression outcome (square root transformed)
	β	SE	β	SE
Modality
In person	N/A	N/A	N/A	N/A
Telehealth	−0.09	0.13	0.08	0.25
Gender
Female	N/A	N/A	N/A	N/A
Male	0.02	0.20	−0.09	0.37
Ethnicity
Not hispanic or latinx	N/A	N/A	N/A	N/A
Hispanic or latinx	0.02	0.14	0.69***	0.25
Race
White	N/A	N/A	N/A	N/A
Not white	0.02	0.16	−0.54	0.30
Attended at least 5 classes	−0.01	.25	1.01**	0.47
Pretest score	0.04***	<0.01	0.15***	0.03
Cohen’s d	0.15		−0.24

Note. N = 55 for both outcomes ***significant at p < .01 level; **significant at p < .05 level.

Results of the Wilcoxon Rank-Sum Test of the association between session modality and TAI scores suggested that participant satisfaction scores (TAI) were not significantly different between caregivers in the two session modalities (p = .64).

Table 4 presents the results of the regression models for child outcomes. The only significant predictor across outcomes was the pre-test score, which was significant for SDQ total, SDQ externalizing, and the ECBI Problem scale. In addition, gender was significantly associated with SDQ externalizing, wherein male children had significantly higher SDQ externalizing scores than female children. Pre-test ECBI Intensity scale scores were not associated with the final ECBI Intensity scale scores. Modality of session was significantly associated with SDQ externalizing, with telehealth children having higher scores on average. The other ECBI and SDQ outcomes were not significantly associated with modality.

Table 4.

Regression results for child outcomes.

	SDQ Total		SDQ Externalizing		ECBI Problem		ECBI Intensity
	β	SE	β	SE	β	SE	β	SE
Modality
In person	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
Telehealth	1.58	0.96	1.38**	0.63	N/A	N/A	N/A	N/A
Gender
Female	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
Male	1.68	1.07	1.66**	0.71	3.13	3.29	−2.97	18.18
Ethnicity
Not hispanic or latinx	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
Hispanic or latinx	1.14	0.98	0.01	0.65	2.14	3.48	8.83	20.22
Race
White	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
Not white	0.26	1.07	0.63	0.69	1.53	3.21	1.98	17.88
Attended at least 5 classes	−2.87	2.23	0.72	1.58	0.59	5.11	−3.64	29.10
Pretest score	0.97***	0.06	0.99***	0.08	0.48***	0.16	0.39	0.22
Cohen’s d	−0.14		0.12		N/A		N/A

Note. ***p < .01 level; **p < .05 level.

Discussion

The purpose of the current pilot study was to describe the DOCS K-5 in-person program as well as the telehealth adaptation (i-DOCS K-5), and compare pre- to post-test outcomes for participants involved in the two modalities (in-person versus telehealth). Results from the study are promising and support telehealth as a viable and accessible option for diverse families of elementary age children. While we did not predict demographic differences in program attendance, the telehealth option resulted in greater numbers of non-white and male caregivers, possibly indicating that telehealth resolved difficulties related to attendance at an in-person session due to barriers associated with work responsibilities, transportation, childcare, or stigma related to mental health treatment.

Consistent with our study hypotheses, there were no significant differences between in-person and telehealth outcomes across parenting stress, child behavior, and treatment satisfaction measures. Thus, caregivers derived similar benefit and satisfaction regardless of the method of delivery. While participants completed different measures for child behavior between year 1 and two of the project, both groups reported significant improvements on measures of child behavior at post-test. Similarly, significant improvements on caregiver stress were reported across both program modalities at post-test. Participant satisfaction scores were positive and indicated that participants overall were moderately satisfied. Thus, consistent with previous findings from systematic reviews and meta-analyses (Chorpita et al., 2011; Florean et al., 2020; Leijten et al., 2013; Thongselratch et al., 2020), both modalities of DOCS K-5 led to improvements in child behavior, caregiver stress, and resulted in participant satisfaction with the program.

Results also indicated that caregivers with higher levels of depression were more likely to attend at least five sessions. It is possible that caregivers with greater symptoms of depression have less social capital than non-depressed peers (Cohen-Cline et al., 2018). Thus, this finding might imply that caregivers with greater reported symptoms of depression are those in most need of support, and therefore are more likely to attend sessions where social support is perceived to be more accessible. In addition, caregivers who identified as Hispanic or Latinx were more likely to report greater symptoms of depression than participants from other ethnic backgrounds (National Center for Health Statistics, 2021). These findings are consistent with the most recent Household Pulse Survey published by the Centers for Disease Control, which indicated that individuals identifying as Hispanic or Latinx reported similar or greater symptoms of depression compared to their non-Hispanic counterparts (National Center for Health Statistics, 2021). Unsurprisingly, children with more reported behavior concerns at pre-test were more likely to have higher scores at post-test (Kolko & Pardini, 2010). Prior researchers have found that features of oppositional defiant disorder (ODD) such as irritability predict continued symptoms of ODD following treatment (Kolko & Pardini, 2010). Finally, results suggested that caregivers who identified their child as male were more likely to report more intense scores on behavioral outcomes.

These results are especially meaningful as the scientific community advances towards widespread implementation of evidence-based practice. Telehealth platforms have expanded access to care for underserved families such as those living in rural areas and those with transportation or childcare obstacles, which are historical limitations associated with mental healthcare (Owens et al., 2002). Not to mention, provision of telehealth services has allowed families to participate while in their natural environment. With these continued efforts, a substantially larger number of providers and families can gain access to evidence-based behavioral strategies, which could alleviate long-term outcomes associated with disruptive behaviors.

The current study provides preliminary evidence to support the utility of a telehealth group-based parenting intervention; however, there are some notable limitations. First, the population characteristics are a significant limitation, even though there was high participation among Hispanic/Latinx participants (∼35%), most participants identified race as White (78%). These demographic characteristics limit the generalizability of the current findings to other racial groups. Second, the sample size might have precluded detection of significant relationships on measures such as caregiver reported depression. Although, uncontrollable events such as the COVID-19 pandemic also could have played a role. As discussed earlier, there are methodological considerations around the small sample as well. Namely, a greater sample size will reduce the need to rely on sibling records, which are problematic in regression modeling. As those results corresponded closely with our additional analysis that only included one child per household, we are confident that the evidence found for telehealth programs being as effective as in-person programs is strong. However, a more robust sample of participants would help to provide more concrete evidence in support of telehealth modalities for this program. Third, the study was limited by exclusive use of self-report outcome measures. Caregivers might have experienced an expectancy bias and thus reported more substantial improvements at post-test. In addition, the ECBI was only collected on a sub-section of the study participants limiting our ability to include modality of session in the regression modeling and effect sizes could not be calculated. Alternatively, caregivers might have felt pressured to provide socially desirable responses after becoming more involved and connected with the instructors and program coordinators. Finally, it is impossible to determine whether participant data were contaminated by other factors such as individualized parent-child therapy or child-focused therapy.

DOCS K-5 is a new BPT and merits further investigation of both the in-person and the telehealth adaptation (i-DOCS K-5). This pilot study is a starting point from which future studies can be designed to more rigorously evaluate the efficacy of this program. Future researchers might consider analyzing larger samples to increase power and confidence in the outcomes. In addition, future research could include randomization to in-person and telehealth platforms to establish greater internal validity. Importantly, further efforts to recruit a more racial, linguistic, and gender-inclusive population will advance generalizability of the current findings. Another avenue for future study could include use of more objective outcomes such as direct observation of caregiver skills and caregiver-child interactions by an independent evaluator. Finally, future researchers might consider evaluating other dimensions of treatment success such as frequency of homework completion as a predictor of caregiver-child outcomes.

Despite these limitations, this study offers preliminary support for the utility of telehealth BPTs. Early treatment and intervention for disruptive behaviors with BPTs is imperative, given the preponderance of challenging behaviors and the outcomes associated with them. The recent transition to telehealth modalities amidst the COVID-19 pandemic has produced fruitful research supporting widescale implementation of mental health services, thereby reducing barriers to care and advancing public health outcomes for children and families.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was funded by the Children's Board of Hillsborough County.

ORCID iDs

Heather Agazzi

Abraham Salinas-Miranda

Author biographies

Heather Agazzi, PhD, ABPP is a professor of pediatrics and psychiatry at the University of South Florida. She is the Chief of Child Development and the USF Health Psychology internship director. Her research interests include a) behavioral parent training programs for caregivers of children with diverse needs; b) Parent-Child Interaction Therapy for children with ASD; c) early trauma-informed interventions for infants and toddlers; d) early diagnosis and intervention for children with ASD. She is the lead author of the HOT DOCS and DOCS K-5 parent training programs.

Holland Hayford, M.A., is a doctoral candidate in school psychology at the University of South Florida. Her background includes providing behavioral and mental health interventions for individuals with autism spectrum disorders, emotional and behavioral disorders, and/or intellectual disabilities. She also has a background and interest in behavioral parent training, including group-based interventions, trauma-informed parenting, and Parent-Child Interaction Therapy. She is a certified PCIT therapist and a certified professional trainer in the programs Helping Our Toddlers, Developing Our Children’s Skills (HOT DOCS) and Developing Our Children’s Skills K-5 (DOCS K-5).

Nicholas Thomas is a research associate at the University of South Florida’s School of Public Health. He received his MPH and PhD from Tulane University’s School of Public Health and Tropical Medicine. Dr. Thomas’ research interests include social determinants of health, family and community violence prevention, disaster resilience, global health, and quantitative methods.

Sarah Dickinson, PhD is a licensed psychologist and assistant professor in the Department of Pediatrics at the University of South Florida. She has expertise in positive parenting practices and early childhood challenging behaviors and trauma. She has been trained in a number of evidence-based treatment protocols, including Parent-Child Interaction Therapy, Trauma-Focused Cognitive Behavioral Therapy, and Managing and Adapting Practices.

Cristina Ortiz, has a B.S. in Human Science, Major in Psychology, from Universidad del Este, Tampa, FL., and an Infant Toddler Developmental Specialist Training. Ms. Ortiz distinguish herself as a parent liaison and a teacher as she goes beyond expectations to assist caregivers in crisis. Ms. Ortiz implements positive behavior supports with caregivers and their young children from a range of socioeconomic and cultural backgrounds. She is the Project Coordinator for HOT DOCS & DOCS K-5 Programs and has been a part of the HOT DOCS team since April 2006 where she first participated as a parent. Her experiences as a mother, as well as her involvement in HOT DOCS, have touched the lives of many families with children with special needs. Her primary interest with HOT DOCS is reaching out to the Latino community and engaging them in early intervention services, connecting them to community resources and teaching them how to advocate for their children.

Abraham Salinas, MD, PhD, MACE has broad background in epidemiology, maternal and child health, family violence studied, and medicine. Currently, he is the Director of the Harrell Center for the Study of Family Violence, at the USF College of Public Health. He also serves as the Associate Director of the Center of Excellence (CoE) in Maternal and Child Health (MCH). His work has focused on the study of adverse childhood experiences and maternal and child health disparities. Methods expertise include community-based participatory research using mixed methods, program evaluation, epidemiological studies, structural equation models, mediation analyses, and meta-analytic models and systematic reviews. He teaches graduate level courses on Family & Community Violence and Child Health and Development.

References

Agazzi

Adams

Ferron

Shaffer-Hudkins

Salloum

(2019). Trauma-informed behavioral parenting for early intervention. Journal of Child and Family Studies, 28(8), 2172–2186. https://doi.org/10.1007/s10826-019-01435-3

Agazzi

Hayford

Thomas

Ortiz

Salinas-Miranda

(2021). A nonrandomized trial of a behavioral parent training intervention for parents with children with challenging behaviors: In-person versus internet-HOT DOCS. Clinical Child Psychology and psychiatry, 26(4), 1076–1088. https://doi.org/10.1177/13591045211027559

Agazzi

Salinas

Williams

Chiriboga

Ortiz

Armstrong

(2010). Adaptation of a behavioral parent-training curriculum for Hispanic caregivers: HOT DOCS Español. Infant Mental Health Journal, 31(2), 182–200. https://doi.org/10.1002/imhj.20251

Baker

Sanders

M. R.

Turner

K. M. T.

Morawska

(2017). A randomized controlled trial evaluating a low-intensity interactive online parenting intervention, triple p online brief, with parents of children with early onset conduct problems. Behaviour Research and Therapy, 91, 78–90. https://doi.org/10.1016/j.brat.2017.01.016

Baumel

Faber

(2018). Evaluating triple p online: A digital parent training program for child behavior problems. Cognitive and Behavioral Practice, 25(4), 538–543. https://doi.org/10.1016/j.cbpra.2017.10.001

Bearss

Burrell

T. L.

Stewart

Scahill

(2015). Parent training in autism spectrum disorder: What's in a name? Clinical Child and Family Psychology Review, 18(2), 170–182. https://doi.org/10.1007/s10567-015-0179-5

Boggs

S. R.

Eyberg

Reynolds

L. A.

(1990). Concurrent validity of the eyberg child behavior inventory. Journal of Clinical Child Psychology, 19(1), 75–78. https://doi.org/10.1207/s15374424jccp1901_9

Brestan

E. V.

Jacobs

J. R.

Rayfield

A. D.

Eyberg

S. M.

(1999). A consumer satisfaction measure for parent-child treatments and its relation to measures of child behavior change. Behavior Therapy, 30(1), 17–30. https://doi.org/10.1016/S0005-7894(99)80043-4

Carr

Dunlap

Horner

Koegel

Turnbull

Sailor

Anderson

J. L.

Albin

R. W.

Koegel

L. K.

Fox

(2002). Positive behavior support: Evolution of an applied science. Positive Behavior Interventions, 4(1), 4–16. https://doi.org/10.1177/109830070200400102

10.

Carr

Hartnett

Brosnan

Sharry

(2017). Parents plus systemic, solution-focused parent training programs: Description, review of the evidence base, and meta-analysis. Family Process, 56(3), 652–668. https://doi.org/10.1111/famp.12225

11.

Chacko

Jensen

S. A.

Lowry

L. S.

Cornwell

Chimklis

Chan

Lee

Pulgarin

(2016). Engagement in behavioral parent training: Review of the literature and implications for practice. Clinical Child and Family Psychology Review, 19(3), 204–215. https://doi.org/10.1007/s10567-016-0205-2

12.

Chorpita

B. F.

Daleiden

E. L.

Ebesutani

Young

Becker

K. D.

Nakamura

B. J.

Phillips

Ward

Lynch

Trent

Smith

R. L.

Okamura

Starace

(2011). Evidence‐based treatments for children and adolescents: An updated review of indicators of efficacy and effectiveness. Clinical Psychology: Science and Practice, 18(2), 154–172. https://doi.org/10.1111/j.1468-2850.2011.01247.x

13.

Cohen-Cline

Beresford

S. A.

Barrington

Matsueda

Wakefield

Duncan

G. E.

(2018). Associations between social capital and depression: A study of adult twins. Health Place, 50, 162–167. https://doi.org/10.1016/j.healthplace.2018.02.002

14.

Comer

J. S.

Furr

J. M.

Miguel

E. M.

Cooper-Vince

C. E.

Carpenter

A. L.

Elkins

R. M.

Kerns

C. E.

Cornacchio

Chou

Coxe

DeSerisy

Sanchez

A. L.

Golik

Martin

Myers

K. M.

Chase

(2017). Remotely delivering real-time parent training to the home: An initial randomized trial of Internet-delivered parent-child interaction therapy (I-PCIT). Journal of Consulting and Clinical Psychology, 85(9), 909–917. https://doi.org/10.1037/ccp0000230

15.

Dunlap

Horner

R. H.

(2006). The applied behavior analytic heritage of PBS: A dynamic model of action-oriented research. Journal of Positive Behavior Support, 8(1), 58–60. https://doi.org/10.1177/10983007060080010701

16.

Eyberg

S. M.

(2002). Therapy Attitude Inventory (TAI). In Schroeder

C. S.

Gordon

B. N.

(Eds.), Assessment and treatment of childhood problems. Guilford Press.

17.

Eyberg

S. M.

Pincus

(1999). Eyberg child behavior inventory & sutter-eyberg student behavior inventory-revised: Professional manual. PAR.

18.

Florean

I. S.

Dobrean

Păsărelu

C. R.

Georgescu

R. D.

Milea

(2020). The efficacy of internet-based parenting programs for children and adolescents with behavior problems: A meta-analysis of randomized clinical trials. Clin Child Fam Psychol Rev, 23(4), 510–528. https://doi.org/10.1007/s10567-020-00326-0

19.

Funderburk

B. W.

Eyberg

S. M.

Rich

B. A.

Behar

(2003). Further psychometric evaluation of the Eyberg and behavior rating scales for parents and teachers of preschoolers. Early Education and Development, 14(1), 67–82. https://doi.org/10.1207/s15566935eed1401_5

20.

Furlong

McGilloway

Bywater

Hutchings

Smith

S. M.

Donnelly

(2012). Behavioural and cognitive-behavioural group-based parenting programmes for early-onset conduct problems in children aged 3 to 12 years. The Cochrane database of systematic reviews, 15(2), CD008225. https://doi.org/10.1002/14651858.CD008225.pub2

21.

Ghandour

R. M.

Sherman

L. J.

Vladutiu

C. J.

Ali

M. M.

Lynch

S. E.

Bitsko

R. H.

Blumberg

S. J.

(2019). Prevalence and treatment of depression, anxiety, and conduct problems in US children. Journal of Pediatrics, 206, 256–267. https://doi.org/10.1016/j.jpeds.2018.09.021

22.

Goodman

(2001). Psychometric properties of the strengths and difficulties questionnaire. Journal of the American Acaddemy of Child & Adolescent Psychiatry, 40(11), 1337–1345. https://doi.org/10.1097/00004583-200111000-00015

23.

Hewitt

K. C.

Loring

D. W.

(2020). Emory university telehealth neuropsychology development and implementation in response to the COVID-19 pandemic. The Clinical Neuropsychologist, 34(7–8), 1352–1366. https://doi.org/10.1080/13854046.2020.1791960

24.

Hobbel

Drugli

M. B.

(2013). Symptom changes of oppositional defiant disorder after treatment with the incredible years program. Nordic Journal of Psychiatry, 67(2), 97–103. https://doi.org/10.3109/08039488.2012.685888

25.

Jones

D. J.

Forehand

Cuellar

Kincaid

Parent

Fenton

Goodrum

(2013). Harnessing innovative technologies to advance children's mental health: Behavioral parent training as an example. Clinical Psychology Review, 33(2), 241–252. https://doi.org/10.1016/j.cpr.2012.11.003

26.

Jones

D. J.

Forehand

Cuellar

Parent

Honeycutt

Khavjou

Gonzalez

Anton

Newey

G. A.

2014). Technology-enhanced program for child disruptive behavior disorders: Development and pilot randomized control trial. Journal of Clinical Child & Adolescent Psychology, 43(1), 88–101. https://doi.org/10.1080/15374416.2013.822308

27.

Kolko

D. J.

Pardini

D. A.

(2010). ODD dimensions, ADHD, and callous-unemotional traits as predictors of treatment response in children with disruptive behavior disorders. Journal of Abnormal Psychology, 119(4), 713–725. https://doi.org/10.1037/a0020910

28.

Kroenke

Spitzer

R. L.

Williams

J. B. W.

(2001). The PHQ-9. Journal of General Internal Medicine, 16(9), 606–613. https://doi.org/10.1046/j.1525-1497.2001.016009606.x

29.

Leijten

Raaijmakers

M. A.

de Castro

B. O.

Matthys

(2013). Does socioeconomic status matter? A meta-analysis on parent training effectiveness for disruptive child behavior. Journal of Clinical Child & Adolescent Psychology, 42(3), 384–392. https://doi.org/10.1080/15374416.2013.769169

30.

Loren

R. E. A.

Vaughn

A. J.

Langberg

J. M.

Cyran

J. E. M.

Proano-Raps

Smolyansky

B. H.

Tamm

Epstein

J. N.

(2015). Effects of an 8-session behavioral parent training group for parents of children with ADHD on child impairment and parenting confidence. Journal of Attention Disorders, 19(2), 158–166. https://doi.org/10.1177/1087054713484175

31.

Moore

C. G.

Carter

R. E.

Nietert

P. J.

Stewart

P. W.

(2011). Recommendations for planning pilot studies in clinical and translational research. Clinical and Translational Science, 4(5), 332–337. https://doi.org/10.1111/j.1752-8062.2011.00347.x

32.

National Center for Health Statistics . (2021). Anxiety and depression: Household pulse survey. CDC. https://www.cdc.gov/nchs/covid19/pulse/mental-health.htm

33.

Nieuwboer

C. C.

Fukkink

R. G.

Hermanns

J. M. A.

(2013a). Online programs as tools to improve parenting: A meta-analytic review. Children and Youth Services Review, 35(11), 1823–1829. https://doi.org/10.1016/j.childyouth.2013.08.008

34.

Nieuwboer

C. C.

Fukkink

R. G.

Hermanns

J. M. A.

(2013b). Peer and professional parenting support on the internet: A systematic review. Cyberpsychology Behavior and Social Networking, 16(7), 518–528. https://doi.org/10.1089/cyber.2012.0547

35.

Nobel

Hoekstra

P. J.

Agnes Brunnekreef

Messink-de Vries

D. E. H.

Fischer

Emmelkamp

P. M. G.

van den Hoofdakker

B. J.

(2020). Home-based parent training for school-aged children with attention-deficit/hyperactivity disorder and behavior problems with remaining impairing disruptive behaviors after routine treatment: A randomized controlled trial. European Child & Adolescent Psychiatry, 29(3), 395–408. https://doi.org/10.1007/s00787-019-01375-9

36.

Ogg

Shaffer-Hudkins

Childres

Feldman

Agazzi

Armstrong

(2014). Attendance and implementation of strategies in a behavioral parent-training program: Comparisons between English and Espanol programs. Infant Mental Health Journal, 35(6), 555–564. https://doi.org/10.1002/imhj.21470

37.

Owens

P. L.

Hoagwood

Horwitz

S. M.

Leaf

P. J.

Poduska

J. M.

Kellam

S. G.

Ialongo

N. S.

(2002). Barriers to children's mental health services. Journal of the American Academy of Child & Adolescent Psychiatry, 41(6), 731–738. https://doi.org/10.1097/00004583-200206000-00013

38.

Prime

Wade

Browne

D. T.

(2020). Risk and resilience in family well-being during the COVID-19 pandemic. American Psychologist, 75(5), 631–643. https://doi.org/10.1037/amp0000660

39.

Reese

R. J.

Slone

N. C.

Soares

Sprang

(2015). Using telepsychology to provide a group parenting program: A preliminary evaluation of effectiveness. Psychological Services, 12(3), 274–282. https://doi.org/10.1037/ser0000018

40.

Ros-DeMarize

Chung

Stewart

(2021). Pediatric behavioral telehealth in the age of COVID-19: Brief evidence review and practice considerations. Current Problems in Pediatric and Adolescent Health Care, 51(1), 100949. https://doi.org/10.1016/j.cppeds.2021.100949

41.

Salinas

Smith

J. C.

Armstrong

(2011). Engaging fathers in behavioral parent training: Listening to fathers' voices. Journal of Pediatric Nursing-Nursing Care of Children & Families, 26(4), 304–311. https://doi.org/10.1016/j.pedn.2010.01.008

42.

SAMHSA . (2018). Disaster Technical Assistance Center supplemental research bulletin: Behavioral health conditions in children and youth exposed to natural disasters. SAMHSA. https://www.samhsa.gov/sites/default/files/srb-childrenyouth-8-22-18.pdf

43.

Sanders

M. R.

Baker

Turner

K. M.

(2012). A randomized controlled trial evaluating the efficacy of Triple P Online with parents of children with early-onset conduct problems. Behaviour Research and Therapy, 50(11), 675–684. https://doi.org/10.1016/j.brat.2012.07.004

44.

Sayal

Tischler

Coope

Robotham

Ashworth

Day

Tylee

Simonoff

(2010). Parental help-seeking in primary care for child and adolescent mental health concerns: Qualitative study. British Journal of Psychiatry, 197(6), 476–481. https://doi.org/10.1192/bjp.bp.110.081448

45.

Silva

L. M.

Schalock

(2012). Autism parenting stress index: Initial psychometric evidence. Journal of Autism and Developmental Disorders, 42(4), 566–574. https://doi.org/10.1007/s10803-011-1274-1

46.

StataCorp (2019). Statistical software: Release Vol. 16. StataCorp LLC.

47.

Thongselratch

Leljten

Melendez-Torres

G. J.

(2020). Online parent programs for children’s behavioral problems: A meta-analytic review. European Child & Adolescent Psychiatry, 29(11), 1555–1568. https://doi.org/10.1007/s0078 7-020-01472 -0

48.

Titov

Dear

B. F.

McMillan

Anderson

Zou

Sunderland

(2011). Psychometric comparison of the PHQ-9 and BDI-II for measuring response during treatment of depression. Cognitive Behavioral Therapy, 40(2), 126–136. https://doi.org/10.1080/16506073.2010.550059

49.

Traube

D. E.

Hsiao

H. Y.

Rau

Hunt-O'Brien

L. X.

Islam

(2020). Advancing home based parenting programs through the use of telehealth technology. Journal of Child and Family Studies, 29(1), 44–53. https://doi.org/10.1007/s10826-019-01458-w

50.

Wainer

A. L.

Ingersoll

B. R.

(2015). Increasing access to an ASD imitation intervention via a telehealth parent training program. Journal of Autism and Developmental Disorders, 45(12), 3877–3890. https://doi.org/10.1007/s10803-014-2186-7

51.

Wainer

(1991). Adjusting for differential base rates: Lord's paradox again. Psychological Bulletin, 109(1), 147. https://doi.org/10.1037/0033-2909.109.1.147

52.

Walker

H. M.

Sprague

J. R.

(1999). The path to school failure, delinquency. and violence: Causal factors and some potential solutions. Intervention in School and Clinic, 35(2), 67–73. https://doi.org/10.1177/105345129903500201

53.

Webster-Stratton

Jamila Reid

Stoolmiller

(2008). Preventing conduct problems and improving school readiness: Evaluation of the incredible years teacher and child training programs in high-risk schools. Journal of Child Psychology & Psychiatry, 49(5), 471–488. https://doi.org/10.1111/j.1469-7610.2007.01861.x

54.

Webster-Stratton

Reid

M. J.

Hammond

(2004). Treating children with early-onset conduct problems: Intervention outcomes for parent, child, and teacher training. Journal of Clinical Child & Adolescent Psychology, 33(1), 105–124. https://doi.org/10.1207/S15374424JCCP3301_11

55.

Whitehead

A. L.

Julious

S. A.

Cooper

C. L.

Campbell

M. J.

(2016). Estimating the sample size for a pilot randomised trial to minimise the overall trial sample size for the external pilot and main trial for a continuous outcome variable. Statistical Methods in Medical Research, 25(3), 1057–1073. https://doi.org/10.1177/0962280215588241

56.

Wilder

(1950). The law of initial values. Psychosomatic Medicine, 12, 392.