Effectiveness of Psychotherapy in Adult ADHD: What Do Patients Think? Results of the COMPAS Study

Abstract

Objective: In the multicenter study Comparison of Methylphenidate and Psychotherapy in Adult ADHD (COMPAS), the efficacy of treatments has been primarily evaluated by observer-rated symptom change. Here, we additionally analyzed the patients’ subjective evaluation of therapy effects. Method: COMPAS compared ADHD-specific group therapy with unspecific clinical management with/without concomitant pharmacotherapy in a four-armed design. Evaluation through the patients’ retrospective perspective was performed after 1 year (post-treatment) and after another 1.5 years (follow-up). Results: In respect to patients’ subjective ratings, ADHD-specific group psychotherapy outperformed unspecific management post-treatment (z = 4.88, p < .0001) and at follow-up (z = 2.90, p = .004). Rank correlations with rater-based symptom change were small to moderate (post-treatment: r_s = 0.28, follow-up: r_s = 0.16). Conclusion: Therapy evaluation based on the patients’ perspective supports the concept of ADHD-specific group psychotherapy as a potentially useful therapy option in ADHD.

Keywords

adult ADHD psychotherapy follow-up mindfulness

Objective

ADHD is a mental disorder with the core symptoms of hyperactivity, impulsivity, and inattentiveness as described in Diagnostic and Statistical Manual of Mental Disorders–Fifth Edition (DSM-V; American Psychiatric Association, 2013), beginning in childhood. Meta-analyses show a prevalence of 3.4% in adults (Fayyad et al., 2007). Numerous negative consequences are associated (Shaw et al., 2012). The National Institute for Health and Care Excellence (NICE) guidelines (National Institute for Health and Care Excellence, 2008) recommend psychopharmacological treatment as first-line treatment for adults with ADHD with moderate or severe impairments. Psychotherapeutic interventions are recommended in cases with mild, residual symptoms, insufficient response to drug treatment, or if a patient decides against drug treatment after careful information. The guidelines do not recommend psychological treatments without medication as first-line treatment because of insufficient data. Later published reviews and meta-analyses found additional evidence for the efficacy of psychological treatments (Jensen, Amdisen, Jørgensen, & Arnfred, 2016; Young, Moghaddam, & Tickle, 2016). Especially treatments that are based on cognitive behavioral therapy (CBT) concepts have been shown to be effective (Vidal-Estrada, Bosch-Munso, Nogueira-Morais, Casas-Brugue, & Ramos-Quiroga, 2012). However, because of the small numbers of studies and small sample sizes of these studies, further research is needed (Jensen et al., 2016). Moreover, there is a lack of data on CBT without concomitant use of pharmacological treatment (Young et al., 2016). The large, multicenter Comparison of Methylphenidate and Psychotherapy in Adult ADHD Study (COMPAS) intended to provide additional evidence on the efficacy of psychotherapeutic approaches in the treatment of adult ADHD. It is the first and, to our knowledge, still the only randomized controlled multicenter trial that evaluated an ADHD-specific group therapy (GPT) program versus unspecific clinical management (CM) with or without concomitant stimulant medication. In contrast with expectations, the ADHD-specific group psychotherapy program did not outperform unspecific CM regarding the study’s primary end point, the reduction of ADHD symptoms rated by blinded observers. This result was surprising because former studies found specific cognitive behavioral programs to be more effective than unspecific conditions in the treatment of adults with ADHD (Hirvikoski et al., 2011; Safren et al., 2010; Solanto et al., 2010).

As method of treatment evaluation, the COMPAS study used the observer-rated Conners’ Adult ADHD Rating Scale–Observer Rated, Long Version (CAARS-O:L) as primary outcome (Christiansen, Hirsch, Abdel-Hamid, & Kis, 2014). Effectiveness was determined through comparisons of pre- and post-treatment measured symptom load. The distinctive feature of this method is the calculation of differences in symptom severity between two time points of measurement, usually before and after treatment (Bereiter, 1963; Michalak, Kosfelder, Meyer, & Schulte, 2003). Due to its strength with regard to objectivity, observer-rated pre–post comparisons are a standard form of therapy evaluation. Nevertheless, this method has some weaknesses, including effects of regression and accumulation of measuring errors, which can lead to bias in therapy evaluation (Stieglitz, Baumann, & Freyberger, 2001; Ülsmann, 2013). Therefore, the additional use of other techniques such as the retrospective evaluation of therapy success is recommended (Michalak et al., 2003). With the retrospective method, the therapy success is globally evaluated at one time point only, after the end of the treatment (Michalak et al., 2003).

Until now unpublished, the COMPAS study also included a therapy evaluation form to judge the effectiveness of the different treatments through the patient’s global retrospective perspective (Philipsen et al., 2010). Based on the data available from these patient questionnaires, the present research focused on the following objectives:

First, we sought to evaluate the treatment benefit of specific psychotherapy versus unspecific CM as measured by subjective, retrospective patient evaluation after 1 year of treatment. Because we assume that a different construct is measured (Michalak et al., 2003), subjective retrospective evaluation may offer deviating results compared with the rater-based, pre–post comparison. In support of this hypothesis, previous pilot studies, in which a subjective evaluation form was used, came to the conclusion that the ADHD-specific group therapy program, as used in COMPAS, is rated as helpful by patients (Hesslinger et al., 2002; Hirvikoski et al., 2011; Philipsen et al., 2007).

Furthermore, we were interested in the long-term effects of both psychological interventions, which were measured by the subjective therapy evaluation questionnaire 1.5 years after the end of treatment. Changes of therapy effect during the course of long-term follow-up seem to be possible in this study setting because the focus of the group psychotherapy program was to learn strategies to deal autonomously with ADHD-specific problems, ideally getting better over time. In contrast, conversations in the CM group could have had some short-term effect for patients by providing valuable psychological support regarding acute, individual daily life problems, but these effects may not be long lasting.

As defined by the four-armed design of the COMPAS study, the psychological interventions were evaluated in combination with either concomitant methylphenidate or placebo treatment.

In addition to the clinical implications of the results on subjective treatment effectiveness, we were also interested in the relationship of indirect versus retrospective measurements as methodological means.

A further research question focused on the identification of specific therapeutic factors of the group program, which are of particular importance for patients as judged by subjective evaluation. Concerning such factors, previous studies found slightly differing results, indicating that “the group” as well as “information and psychoeducation” (Hesslinger et al., 2002; Hirvikoski et al., 2011; Philipsen et al., 2007) may be central aspects. New information on this subject may help to create future programs that integrate relevant aspects in the most economical way.

Method

Study Design, Participants, and Treatment

The multicenter study COMPAS took place at seven German study sites from January 2007 to March 2013. It was funded by the German Federal Ministry of Research and Education (01GV0606, ISRCTN54096201). Included were males and females aged 18 to 60 years with an ADHD diagnosis according to Diagnostic and Statistical Manual of Mental Disorders–Fourth Edition (DSM-IV; American Psychiatric Association, 1994) criteria. Exclusion criteria included clinically significant abnormalities detected on physical examination, routine blood testing, electrocardiogram (ECG) or electroencephalogram (EEG), severe psychiatric conditions such as schizophrenia, and treatment with stimulants or ADHD-specific psychotherapy within the last 6 months prior to screening (Philipsen et al., 2010).

The time line of treatments and measurements over the course of the study is shown in Figure 1. Further details concerning the treatments and the structure of the study are presented in the study protocol (Philipsen et al., 2010).

Figure 1.

Study design.

Patients were randomized to four different treatment groups that combined psychological and psychopharmacological treatments in a 2 × 2 factorial design. The four treatment groups were group therapy (GPT) + methylphenidate hydrochloride (MPH), clinical management (CM) + MPH, GPT + Placebo, and CM + Placebo.

Treatments

Group therapy (GPT)

The GPT was conducted according to the manual of Hesslinger et al. (Hesslinger, Philipsen, & Richter, 2004), a program based on the principals of dialectical behavioral therapy (DBT; Linehan, 1993) and CBT. DBT was included in this program on the notion that ADHD shares some common clinical features with borderline personality disorder, particularly poor emotion and impulse control. This program covers detailed psychoeducation on symptoms, neurobiology, and comorbidities of ADHD as well as the practice of concrete strategies, including mindfulness training, self-organization skills, self-management (analysis of problem behavior and change), emotion regulation, impulse control, and stress management. After the principle of mindfulness training was introduced, each therapy unit started with a mindfulness exercise. Six to nine patients formed a closed therapy group. Each GPT session consisted of two 50 min units, divided by a 20 min break. The program started with an intense phase of 12 weeks with weekly session, during which all topics and skills were introduced. This was followed by 10 sessions, conducted every 4 weeks, focusing on repeating and more extensive practicing of strategies (Philipsen et al., 2010).

Clinical management (CM)

In contrast with GPT, psychiatric counseling within the CM condition was conducted in an individual setting. Counseling was performed in a nondirective fashion and explicitly did not include specific or structured behavioral interventions and homework. A CM session lasted 15 to 20 min. The number of sessions was the same as in the GPT condition with 12 weeks of weekly sessions, followed by 10 sessions conducted every 4 weeks.

Pharmacological treatment

The psychopharmacological treatment was double-blinded and consisted of treatment with MPH (sustained release, Medikinet retard®) or placebo as a control condition. The start dosage of MPH was 10 mg/day. Within the 52 weeks, the dosage was individually administered with a maximum dosage of 1.3 mg/kg.

Measurement of Treatment Outcomes

Treatment effectiveness was measured by both patient-rated questionnaires (patient-rated ADHD symptoms, CAARS-S:L; evaluation of GPT and CM), as well as observer-rated interviews (CAARS-O:L), and Clinical Global Impression of Improvement (CGI-I) and Clinical Global Assessment of Effectiveness (CGA-E) Scales.

Observer-based ratings were performed by qualified raters. To maximize objectivity and inter-rater reliability, all raters underwent a specific training on the methodology of CAARS interviewing and rating. Training success was cross-checked by a second-rater assessment of standardized videotaped interview sessions. All raters were blinded and were not involved in the study apart from their function as interviewers/raters.

Measurements of pre–post comparisons (CAARS-O:L, CAARS-S:L) were performed at baseline, post-treatment (week 52), and at follow-up (1.5 years after treatment completion). All other instruments concerned retrospective evaluation and were, thus, performed post-treatment and at follow-up only (Figure 1).

All constructs of effectiveness measurement are described in detail below. Results on the Beck Depression Inventory (BDI) and Symptom Checklist 90 (SCL-90) as also assessed in COMPAS were not included in the present analysis and have been reported previously (Philipsen et al., 2015).

Instruments of Treatment Evaluation

Patient-rated retrospective effectiveness

The effectiveness through the patients’ perspective was evaluated by a single-item solution, asking for a global verdict on the therapy effectiveness, which was specifically designed for the COMPAS study. This question was based on the principles of the Clinical Global Assessment–Efficacy (CGA-E, see below) but was specified to the evaluation of ADHD symptoms rather than undefined “efficacy.” Patients were asked, “How do you judge the effectiveness of the whole treatment in respect to your AD(H)D-symptoms?” (post-treatment measurement) and “How do you judge the effectiveness of the whole study treatment in respect to your AD(H)D-symptoms until today’s date?” (follow-up measurement). Answers were made on a 5-point Likert-type scale ranging from 0 = not effective to 4 = very effective. For easier interpretation of descriptive results, we summarized answers ranging from moderately effective = 2 to very effective = 4 under the label “effective.” The data were collected after the end of the treatment (52 weeks after the baseline measurement) and at follow-up, 2.5 years after baseline measurement (see Figure 1).

Pre–post comparisons (CAARS-O:L and CAARS-S:L)

For treatment evaluation based on pre–post comparisons, ADHD severity was measured using the CAARS-O:L, and treatment effectiveness was determined by calculating the pre–post differences in symptom load between baseline and treatment completion/follow-up.

The CAARS measures presence and severity of ADHD symptoms. It is a multi-informant assessment instrument with an observer- and subjective-rated version (Conners, Ehrhard, & Sparrow, 1999). To determine the indirect effectiveness, we used both forms in the long version: the CAARS observer-rated, long version (CAARS-O:L) and the CAARS in the subjective-rated, long version (CAARS-S:L). The CAARS-S:L and the CAARS-O:L each consist of 66 items and nine subscales. The subscales include 4 factor-derived scales, three scales with symptoms relevant to DMS-IV, an Inconsistency Index and an ADHD-Index. The ADHD-Index consists of 12 items that separate best between patients with high and low probability of an ADHD diagnosis (Christiansen et al., 2014). A higher score indicates a higher probability of an ADHD diagnosis.

In the COMPAS study, the difference of the ADHD-Index of the CAARS-O:L between baseline and end of intensive training was defined as the primary outcome. Differences between baseline and follow-up on the CAARS-O:L ADHD-Index and differences on the CAARS-S:L ADHD-Index were used as secondary outcomes. Primary and secondary outcomes have been published in Philipsen et al. (2015).

For data analysis, the pre–post differences between Week 0 and 52 (post) and between Week 0 and 130 (follow-up) were calculated. Positive differences indicate a symptom reduction.

Observer-rated, retrospective effectiveness

To assess effectiveness retrospectively and rater-based, the CGI-I Scale and the CGA-E Scale were used. The CGI-I provides a brief clinical evaluation of the improvement during therapy. It is a commonly used, practical instrument (Busner & Targum, 2007) that is based on a single-item rater evaluation of change between the beginning and the end of treatment on a 7-point Likert-type scale from 1 = very much improved to 7 = very much worse. The CGA-E Scale is also a single-item instrument. On a 4-point Likert-type scale, a rater assesses the effectiveness of the treatment from 1 = minor to 4 = very good.

Specific evaluation of the GPT

To assess the specific effect of distinct factors of the GPT program, a questionnaire was constructed, because no standardized, validated questionnaire was available. Factors considered being potentially helpful aspects of GPT were “information,” “group members,” and “group therapist.” The factor helpfulness of information was operationalized through four questions concerning the helpfulness of information about neurobiology, ADHD-specific medication (not only stimulants), comorbid illnesses, and addiction. Helpfulness of group members and the group therapist was operationalized through questions concerning the helpfulness in general, in respect to motivation and in respect to executing skills. Answers were made on a 5-point Likert-type scale ranging from 0 = not helpful to 4 = very helpful. Assessments between 2 = moderate helpful and 4 = very helpful were integrated and reported as helpful.

Statistical Methods

Descriptive analyses of data were used to determine the direct, subjective effectiveness. Subsequent to the descriptive analyses of subjective effectiveness, a stratified Wilcoxon rank sum test was performed to detect differences between GPT and CM (stratification for largest study site versus all other sites). A Kruskal–Wallis test was used to investigate differences between the four arms (GPT + MPH, CM + MPH, GPT + Placebo, and CM + Placebo), and followed by stratified Wilcoxon tests to localize differences in two-group comparisons. The two-sided significance level was set at .05. No adjustment for multiple testing was made. Spearman rank correlations r_s were calculated to determine the stability of the retrospective effectiveness evaluation between post-treatment and follow-up time points (in general and for all treatment groups separately) and to determine the relationship between pre–post and retrospectively measured effectiveness. To interpret the Spearman coefficients, the conventions of Cohen for Pearson correlation (Cohen, 1988) are used. Following them, a correlation coefficient of .10 is thought to represent small association, a correlation coefficient of .30 is considered as moderate, and a correlation coefficient of .50 or larger as a large correlation. To evaluate helpful aspects of GPT, descriptive analyses were used. All analyses were performed using SAS version 9.2 statistical software (SAS Institute, Inc.).

Results

Sample

For detailed information concerning the general characteristics of the study sample (including recruitment process and participant characteristics), see Philipsen et al. (2014). At the end of the recruitment process, 433 patients were randomized to one of the four treatment conditions. At Week 52 (post-treatment), 323 patients completed the question concerning the subjective therapy evaluation. At follow-up 2.5 years after baseline, the data of 238 patients concerning the subjective therapy evaluation were available.

Subjective Retrospective Effectiveness

Figure 2 illustrates the descriptive results of the subjective retrospective effectiveness evaluation 52 weeks after treatment start (post). The overall treatment was judged as effective (“moderately effective” or better on the 5-point Likert-type scale) by 80% of 82 patients who had received the combination of group therapy and methylphenidate (GPT + MPH) and by 64% of 77 patients who had received group therapy plus placebo (GPT + placebo). In contrast, only 59% of the 87 patients in the clinical management plus methylphenidate condition (CM + MPH) and 34% of 77 patients in the clinical management plus placebo condition (CM + placebo) found the therapy to be effective.

Figure 2.

Subjective retrospective effectiveness (post-treatment).

Overall, the four groups differed significantly concerning their effectiveness evaluation post-treatment (χ² = 40.25, df = 3, p < .0001). Table 1 shows the results of the two-group comparisons of GPT and CM, as well as GPT and CM in combination with either MPH or placebo. Independently of the pharmacological treatment, GPT was rated as significantly more effective compared with CM (z = 4.88, p < .0001). Differences between the effectiveness evaluation in the GPT and CM groups are smaller in the MPH-controlled condition than in the placebo condition, but still significant at an α level of 5%.

Table 1.

Results of the Wilcoxon Two-Group Comparisons Concerning Effectiveness (Post).

Compared groups	N	z	p
GPT versus CM	323	4.88	<.0001
GPT/MPH versus CM/MPH	169	2.00	.046
GPT/placebo versus CM/placebo	154	4.89	<.0001
GPT/placebo versus CM/MPH	164	1.18	.24

Note. GPT = group therapy; CM = clinical management; MPH = methylphenidate hydrochloride.

As shown in Figure 2, the differences in subjective effectiveness between GPT and CM are larger in the placebo-controlled conditions (z = 4.89, p < .0001) than in the MPH conditions (z = 2.00, p = .046). Furthermore, the effectiveness evaluation between the GPT + placebo condition and the CM + MPH condition did not differ significantly post-treatment (z = 1.18, p = .24).

At follow-up, the distribution of the retrospective effectiveness evaluation between the four treatment groups remained stable (see Figure 3). A total of 78% of the 60 patients in the GPT + MPH condition judged their treatment as effective. In contrast, only 58% of the 65 patients who had received CM in combination with MPH judged their treatment as effective. Concerning the follow-up data, the Kruskal–Wallis test revealed a significant difference between at least two of the four treatment groups concerning their efficacy evaluation (χ² = 12.58, df = 3, p = .006). Overall comparisons showed statistically significant differences between the psychological treatments (z = 2.90, p = 0.004). When separated by concomitant pharmacotherapy, differences between GPT and CM were significant when combined with placebo (z = 2.59, p = 0.010), whereas differences between treatments in combination with MPH were not significant (z = 1.45, p = .15; see Table 2). Congruent to the post-treatment results, the effectiveness evaluation of the GPT + placebo and the CM + MPH group did not differ significantly at follow-up (z = 1.46, p = .14).

Figure 3.

Subjective retrospective effectiveness (follow-up).

Table 2.

Results of the Wilcoxon Two-Group Comparisons Concerning Effectiveness (Follow-Up).

Compared groups	N	z	p
GPT versus CM	238	2.90	.004
GPT/MPH versus CM/MPH	125	1.45	.14
GPT/placebo versus CM/placebo	113	2.59	.010
GPT/MPH versus CM/placebo	127	1.46	.14

Note. GPT = group therapy; CM = clinical management; MPH = methylphenidate hydrochloride.

Stability of the Subjective Retrospective Effectiveness Measurement

Overall, there is a large correlation (r_s = .60) between the effectiveness evaluation post-treatment and at follow-up. As Table 3 shows, on group level, all correlations are moderate (r_s = .43 in the CM + MPH condition) or large (r_s ≥ .50). Overall, there are higher correlations in the placebo-controlled conditions than in conditions with concomitant MPH treatment (see Table 3).

Table 3.

Correlation Between Effectiveness Evaluation Post-Treatment and at Follow-Up.

		All four groups	GPT/MPH	GPT/placebo	CM/MPH	CM/placebo
Correlation of direct subjective effectiveness post-treatment and follow-up	N	213	55	54	57	47
	r _s	.60	.50	.65	.43	.62
	p	<.0001	<.0001	<.0001	.0009	<.0001

Note. GPT = group therapy; MPH = methylphenidate hydrochloride; CM = clinical management.

Correlation Between Indirect and Retrospective Effectiveness Evaluation

The correlation between the subjective effectiveness evaluation and the difference in the ADHD- Index of the CAARS-O:L between week 0 and 52 was only small to moderate (r_s = .28, p < .0001, N = 315; see Table 4). In the follow-up, the correlation is even lower (r_s = .16, N = 235, p = .017). As expected, the correlations are positive, indicating a positive relation between symptom improvement and subjective effectiveness evaluation.

Table 4.

Correlation Between Effectiveness Evaluation (Post-Treatment, Follow-Up) and Differences on CAARS-O:L, Differences on CAARS-S:L, CGI Change, and CGA Effectiveness.

		Difference CAARS-O:L	Difference CAARS-S:L	CGI-I	CGA-E
Effectiveness (post-treatment, 0-52)	N	315	316	226	226
	r _s	.28	.41	−.46	.66
	p	<.0001	<.0001	<.0001	<.0001
Effectiveness (follow-up, 0-130)	N	235	232	238	238
	r _s	.16	.20	−.49	.67
	p	.017	.002	<.0001	<.0001

Note. CAARS-O:L = Conners’ Adult ADHD Rating Scale–Observer Rated, Long Version; CAARS-S:L = Conners’ Adult ADHD Rating Scale–Subjective Rated, Long Version; CGI = Clinical Global Impression; CGA = Clinical Global Assessment; CGI-I = CGI of Improvement; CGA-E = CGA of Effectiveness.

As shown in Table 4, the correlation between the subjective effectiveness evaluation and the patient-rated change of CAARS (CAARS-S:L) at week 52 was of moderate strength with r_s = .41. Like the correlation between differences in the CAARS-O:L and subjective effectiveness, the correlation of differences on the CAARS-S:L and effectiveness measured is lower in the follow-up than post-treatment (r_s = .20, p = .002).

Between subjective effectiveness and the CGI-I score measured at Week 52 (r_s = −.46) and at follow-up (r_s = −.49), a nearly large negative correlation was found, indicating a positive correlation concerning the clinical well-being of patients. The correlation between subjective effectiveness and CGA-E is even higher (see Table 4).

Helpfulness of Different Aspects of the GPT, Assessed Post-Treatment

All potentially relevant aspects of the GPT (information, group members, group therapist) were judged with positive ratings (moderately helpful to very helpful) by the vast majority of patients.

Descriptive data analysis shows that the group therapist is received as the most helpful factor by patients. In all, 92% of 82 patients in the MPH-controlled group and 90% of 73 patients in the placebo-controlled condition found the group therapist in general to be helpful.

The positive evaluation of the therapist concerned motivational aspects as well as executing skills: 85% in the GPT + MPH group and 82% in the GPT + placebo group found the group therapist helpful for motivation and 83% in the GPT + MPH group and 81% in the GPT + placebo group found the group therapist helpful for the execution of skills.

The aspect “group members in general” was rated as helpful by 79% of patients across both pharmacological treatment groups. Information concerning neurobiology, medication, comorbid illnesses, and addiction were judged as helpful by between 69% and 79% of patients. Information about comorbid illnesses was the most helpful informational aspect. A total of 79% in the GPT + MPH condition and 77% in the GPT + placebo condition found it to be helpful. The two GPT groups differed slightly in their judgment of the helpfulness of medication and addiction. Only 69% in the GPT + MPH group found information concerning medication and 71% information concerning addiction helpful, whereas 77% and 76% in the GPT + placebo group found these aspects helpful. Group members were judged as mostly helpful in general. In total, 79% in the GPT + MPH and 80% in the GPT + placebo condition found them helpful. In respect to motivation, 76% in both treatment groups found group members to be helpful.

Discussion

To further investigate the effect of specific psychological treatment on adults suffering from ADHD, the large multicenter study COMPAS compared a DBT-based, ADHD-specific treatment program with supportive counseling as an unspecific form of clinical management.

The ADHD-specific program by Hesslinger et al. (2004) has previously been shown to be effective in reducing symptoms of ADHD (Hesslinger et al., 2002; Philipsen et al., 2007). Against expectations, the COMPAS study did not show a superiority of specific GPT compared with supportive counseling with regard to observer-rated ADHD symptom reduction (Philipsen et al., 2015).

However, the analysis presented here of the subjective, retrospective evaluation by patients showed that at the end of treatment as well as 1.5 years after the end of treatment, significantly more patients who had received the specific group psychotherapy found their treatment to be effective, compared with patients treated with unspecific counseling.

While measurement of retrospective subjective effectiveness was not the primary end point of the study and the results need to be interpreted with care, they indicate that, from the patients’ perspective, the type and specificity of psychological treatment may play a significant role. In the long-term follow-up, the type of psychological treatment only had a significant impact if patients had not received concomitant pharmacological treatment. This may be explained by the well-known high effectiveness of MPH treatment, masking the independent effects of psychotherapy. Because of the larger sample size in the evaluation just after treatment compared with follow-up, the moderate differences in the subjective effectiveness evaluation between the psychological treatments with concomitant pharmacological treatment became statistically significant.

Regarding the subjective retrospective effectiveness, there was no significant difference between ADHD-specific group therapy in combination with placebo compared with clinical management in combination with MPH, however, this study was not designed to show statistical non-inferiority. Still, our results suggest that psychotherapy may offer a beneficial treatment option especially to those patients who do not receive pharmacological treatment, for example, for medical or personal reasons.

In addition to the therapy evaluation through the patients’ global retrospect perspective, we were also interested in the evaluation of the used measure from a methodological perspective.

The evaluation of subjective effectiveness was highly stable, even 1.5 years after the end of treatment. This result (overall correlation of r_s = .60) can also be interpreted as an indication for the retest reliability of the used measure of direct subjective effectiveness. Interestingly, the stability of the subjective effectiveness evaluation was higher when combined with a placebo. An explanation might be a potential discontinuation of pharmacological treatment after the end of study treatment. Results concerning the pharmacological treatment between the end of study treatment and follow-up assessment in COMPAS will be published separately.

As expected, we found only a small to moderate relation between the global retrospective effectiveness evaluation and the indirect, pre–post measured symptom change through observer- or self-rating. This result is in line with a previous study that compared the relationship between retrospective and pre–post measured forms of treatment evaluation. It implies that the direct form of effectiveness evaluation is a complementary construct in the evaluation of psychotherapeutic effects with prognostic validity (Michalak et al., 2003).

In line with this interpretation, we found a strong association between the retrospective subjective effectiveness measure and the retrospective observer-based evaluation of effectiveness (CGI-I, CGA-E). This result indicates that the defining factor for the global post-treatment measured form of therapy evaluation is the retrospective way of therapy assessment, rather than the subjectiveness of the patient-rated measurement.

The weak correlation of the subjective retrospective measurement with pre–post measured forms and the large correlation with rater-based, retrospective forms also supports the construct validity of the used instrument. Nonetheless, further validation of the used one-item solution of therapy effectiveness is needed, especially to delimit it from other retrospect measured constructs, such as satisfaction. The latter may be especially prone to influences of social, group-related factors, such as new friends and mutual support, and this may have played a role in the individual patient assessment of global effectiveness. However, patients were explicitly asked to rate therapy effectiveness in respect to their ADHD symptoms, to minimize such bias by unspecific factors of patient contentment.

Independently of the exact construct measured here, our results show that, subjectively, patients felt to have profited more strongly from the ADHD-specific group program than from the individual CM. A main burdensome feature of ADHD is the interference with patients’ functioning in daily private and professional life, often causing secondary psychiatric problems due to constant negative feedback. Thus, in ADHD therapy, measures that improve the overall patient well-being should be regarded as useful per se, even if objectively measured features of ADHD have not improved substantially.

Concerning the utility of the retrospective effectiveness evaluation, Michalak et al. (2003) found that the retrospective subjective evaluation of treatment success predicted future medical and psychological treatment better than the symptom change throughout therapy. Future studies should examine whether this relation can also be found for the subjective global effectiveness evaluation in the treatment of adults with ADHD.

It is unclear which aspect of improvement patients focus on when evaluating the effectiveness of the treatment. Potentially, they primarily focus on only a few particularly disruptive symptoms of ADHD, which have been reduced throughout treatment, and this may to some degree explain the differences between pre–post (indirect) and retrospective measurement. Because the CAARS ADHD index, as used here, consists of a total of 12 items, the weight of each single item on the overall score is relatively small. Still, improvement of single ADHD-related problems, such as better time management or improved impulse regulation, may have a large impact on an individual patient’s functional outcome in daily life. Furthermore, “soft” factors such as the self-perception of an individual in relation to their diagnosis of ADHD or a general mind-set of self-efficacy by having strategies to cope with ADHD symptoms, are not covered by CAARS, yet may mean a profound difference to some patients. Our results suggest that the clinical evaluation of quantifiable changes in ADHD symptoms, as measured by CAARS, may not be optimally suited to assess a patient’s individual impairments in daily activities from an integral perspective. Further research is needed to investigate the relative amount of burden of specific ADHD symptoms and phenomena on the subjective patient outcome in work-related and private functioning as well as overall psychological well-being.

A limitation of this study is the evaluation of patient-rated retrospective efficacy based on a single question only. While this corresponds to the principle of a “global” retrospective evaluation, the vulnerability of this method to a placebo effect may be high. In our study design, the CM condition served as a placebo condition to control this effect, however, it has to be noted that the CM condition differed from the GPT with regard to the amount of time spent in the therapy setting. As the time spent in the GPT setting was considerably greater than in the CM condition, this may have influenced efficacy ratings of patients. For example, a closer relationship to the therapist is favored in a time-extensive group setting, and this may be displayed in the rating of the group therapist as the most helpful aspect of GPT. Yet, while this line of argumentation implies the possibility that the effects of GPT may be relatively unspecific, it does not take away the fact that patients felt to have benefited from the program. Also, from a socioeconomic viewpoint, GPT seems a very good way to maximize the time spent in a therapy setting at reasonable costs.

In line with the nature of the disorder, the number of dropouts after randomization was high. Patient ratings were available for roughly three quarters (323/433) and half (238/433) of the patients post-treatment and at follow-up, respectively. This may have biased the ratings post-treatment and at follow-up toward more positive ratings, because it can be assumed that patients who did not feel the study treatment to be effective were more likely to drop out. In line with this argumentation, the highest number of dropouts occurred in the CM + placebo group. Thus, bias by dropouts may lead to overestimation of the efficacy of therapies in all groups but does not jeopardize the finding of a superiority of GPT versus CM on the patients’ ratings.

To integrate relevant aspects of group psychotherapy in future programs, we evaluated different aspects of the ADHD-specific group psychotherapy. Deviating from Hirvikoski et al. (2011) and Hesslinger et al. (2002) who found “the group” and Philipsen et al. (2007) who found “information” and “psychoeducation” to be most helpful, in the COMPAS study, the group therapist was perceived as the most helpful aspect. Group members and given information were regarded as similarly helpful. As COMPAS was a multicenter study, substantial bias resulting from individual, particularly effective group therapists is unlikely, still, the individuality of therapists is a factor difficult to control in comparisons between studies. The diverging results from different studies using the same therapy manual might suggest that several factors are of importance, also depending on different patient collectives, and supports the continuation and further development of programs covering a range of aspects, including psychoeducation, within a group setting under experienced guidance. Here, future research focusing on the advancement of therapy programs is needed.

Due to the limitations discussed above, the results presented here need to be interpreted with care, and further research is needed to further clarify the benefits of psychotherapeutic approaches in adult ADHD therapy and their potential place in routine clinical care. Because a major difficulty of interpretation in the COMPAS study was the difference between observer-based pre–post comparisons and the global, retrospective efficacy measures, future studies should be designed with particular care regarding the instruments of efficacy measurement and their ability to integrate patient-centered concerns while minimizing the risk of bias by clinically irrelevant factors.

Altogether, our results support the view of ADHD-specific group therapy as a potentially promising approach in the treatment of adult ADHD, yet, further scientific evidence is needed for clearer implications on the role of psychotherapy in routine clinical practice.

Footnotes

Acknowledgements

The Comparison of Methylphenidate and Psychotherapy in Adult ADHD Study (COMPAS) Consortium includes the following members: Michael Colla, MD, Laura Gentschow, Paula Kunze, and Daina Langner, PhD, Department of Psychiatry and Psychotherapy, Campus Benjamin Franklin, Charité–University Medicine Berlin, Berlin, Germany; Bernhard Kis, MD, Mona Abdel-Hamid, PhD, Viola Heinrich, Markus Krämer, MD, and Jennifer Uekermann, PhD, Department of Psychiatry and Psychotherapy, LVR-Hospital Essen, Faculty of Medicine, University of Duisburg-Essen, Duisburg and Essen, Germany; Alexandra Philipsen, MD (coordinating investigator), Erika Graf, PhD, Swantje Matthies, MD, Marc Loewer, MD, Patricia Borel, Imke Jansen, Steffi Bonfico, Manuel Jooßens, Chiharu Sadohara, Manfred Weber, Melanie Kamp, Tatja Dopatka, Evgeniy Perlov, MD, and Harald Richter, PhD, Department of Psychiatry and Psychotherapy, University Medical Center Freiburg, University of Freiburg, Freiburg, Germany; Michael Roesler, MD, Wolfgang Retz, MD, Petra Retz-Junginger, PhD, Konstanze Roemer, Birgit Leipnitz, MD, Sabine Doyran, MD, and Monika Schulte-Altedorneburg, MD, Institute for Forensic Psychology and Psychiatry, Saarland University Medical Center and Saarland University Faculty of Medicine, Homburg/Saar, Germany; Christine Carl, PhD, and Clemens Keutler, MD, Department of Child and Adolescent Psychiatry and Psychotherapy, St. Elisabethen Krankenhaus, Lörrach, Germany; Michael Huss, MD, Catherine Kornmann, Arne Bürger, Galina Chervenkova, and Patricia Meinhardt, Department of Child and Adolescent Psychiatry and Psychotherapy, University Medicine Mainz, Mainz, Germany; Esther Sobanski, MD, Barbara Alm, MD, Martina Schumacher-Stien, Simon Bukow, MD, and Sotiria Argiriou-Martin, Department of Psychiatry and Psychotherapy, Central Institute of Mental Health, Clinical Faculty Mannheim, University of Heidelberg, Mannheim, Germany; Christian Jacob, MD, Thomas Jans, PhD, Silke Groß-Lesch, MD, Monika Heine, MD, Andrea Boreatti-Hümmer, MD, Julia Heupel, Susanne Reichert, Sabine Müller, Susanne Kreiker, Alexandra Gessner, Annette Conzelmann, and Christina Bähne, Department of Psychiatry, Psychosomatics and Psychotherapy and Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, University Hospital of Würzburg, Würzburg, Germany; and Rainer Bredenkamp (director), Gabriele Ihorst, PhD, and Erika Graf, PhD, Clinical Trials Unit, University Medical Center, University of Freiburg, Freiburg, Germany. The independent data monitoring committee included Helmut Remschmidt, MD, Department of Child and Adolescent Psychiatry, Philipps University, Marburg, Germany; Gernot Wassmer, PhD, Institute for Medical Statistics, Informatics, and Epidemiology, University of Cologne, Cologne, Germany; and Norbert Wodarz, MD, Department of Psychiatry, University. We thank Angelika Gerlach, Clinical Trials Unit, University Medical Center–University of Freiburg, Freiburg, Germany, who contributed to statistical data analysis.

Authors’ Note

All data are accessible upon request by contacting the corresponding author.

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Alexandra Lam declares that she received travel grants within the last year from MEDICE Arzneimittel Pütter GmbH and Co. KG. Swantje Matthies received a speakers’ fee from Jansen-Cilag and was involved in clinical trials conducted by Janssen-Cilag and Lilly as a study physician in the years 2007–2009. She has no conflicts of interest since that time. Esther Sobanski has served on advisory boards of MEDICE Arzneimittel Pütter GmbH and Co. KG, Eli Lilly and Co, and Shire. Michael Rösler has, during the last 3 years, been part of advisory boards and a speaker for Medice Arzneimittel, Novartis, Shire, and Lilly. He has performed phase 3 studies for Medice. Wolfgang Retz has served on the advisory board of MEDICE within the last 3 years and has received funding for clinical trials by Novartis and royalties for publications concerning psychosocial treatment of ADHD from Elsevier, Hogrefe, and Kohlhammer publishers. Christian Jacob has served on the advisory board of MEDICE Arzneimittel Pütter GmbH until 2015. Michael Colla has served on advisory boards, received speaker’s honoraria, or performed phase 3 studies with Shire, Eli Lilly and Co, and Novartis. Michael Huss has received personal fees, non-financial support or grants within the last 3 years by Shire, MEDICE Arzneimittel Pütter GmbH and Co. KG, Novartis, Eli Lilly, Engelhard Arzneimittel, and Actelion. He holds the patents DE 10221839 B4 and US 20050131292 A1 on doppler radar. Bernhard Kis is an advisory board member of Medice and Servier and has received speaking honoraria by MEDICE Arzneimittel Pütter GmbH and Co KG, Eli Lilly and Co, Germany, and Servier. Ludger Tebartz van Elst has held payed talks on different medical issues indirectly paid for by different institutions and pharmaceutical companies. Alexandra Philipsen has served on advisory boards, given lectures, performed phase 3 studies, or received travel grants within the last 3 years from Eli Lilly and Co, Janssen-Cilag, MEDICE Arzneimittel Pütter GmbH and Co KG, Novartis, and Shire; and has authored books and articles on psychotherapy published by Elsevier, Hogrefe, Schattauer, Kohlhammer, Karger, Springer, and Oxford Press. All other authors declare that they have no conflicts of interest.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The trial was funded by the German Federal Ministry of Research and Education (01GV0606) and is part of the German network for the treatment of ADHD in children and adults (ADHD-NET).

Author Biographies

Vivien Groß, Dipl-Psych, is a psychotherapist and a PhD student in the Psychiatry Department at the medical campus of the University of Oldenburg. She is especially interested in ADHD and the evaluation of its treatment.

Caroline Lücke, MD, is a postdoctoral researcher in the Psychiatry Department at the medical campus of the University of Oldenburg. Her research interests include ADHD, psychiatric epidemiology, and transcranial magnet stimulation.

Erika Graf, Dipl-Stat, is a biostatistician working as a postdoctoral researcher at the Institute for Medical Biometry and Statistics, the Clinical Trials Unit, and the Medical Faculty of the Medical Center–University of Freiburg.

Alexandra P. Lam, MD, is a resident and researcher in the Department of Psychiatry and Psychotherapy and the Department of Biological Psychology of the European Medical School at the University of Oldenburg. Her research focuses on adults with ADHD, especially on the neurobiology and treatment of ADHD.

Swantje Matthies, MD, works as a certified psychiatrist, senior physician at the specialized treatment unit for borderline personality, and research associate in the Department of Psychiatry and Psychotherapy, University Medical Centre Freiburg, Germany. Her research areas include emotion regulation in ADHD and borderline personality disorder, as well as psychotherapeutic interventions in adult ADHD.

Patricia Borel, Dipl-Psych, is a psychotherapist in her own psychotherapy practice at Müllheim and was a researcher in the Psychiatry Department at the medical campus of the University of Freiburg. She is especially interested in ADHD and the evaluation of its treatments.

Esther Sobanski, MD, is professor of child and adolescent psychiatry, deputy director of the Clinics of Child and Adolescent Psychiatry and Psychotherapy University Mainz and head of the Scientific Working Group Adult ADHD Central Institute of Mental Health (CIMH) Mannheim. Her research topics are clinical and neurobiological aspects of ADHD over the lifespan, as well as trauma and psychotherapy.

Michael Rösler, MD, is professor of forensic psychiatry at the University of the Saarland. He has been engaged in ADHD research for the past 25 years.

Wolfgang Retz, MD, is professor and head of forensic psychiatry and psychotherapy in the Department of Psychiatry and Psychotherapy at the University Medical Center Mainz. His research focus is the neurobiology, diagnosis, and treatment of disruptive behavior disorders, including ADHD.

Christian Jacob, MD, is head of the Department of Psychiatry and Psychotherapy at the medius clinic Kirchheim. His scientific interests focus on neurobiological foundation of adult ADHD and personality disorders.

Michael Colla, MD, is a senior physician at the Clinic and Polyclinic for Psychiatry and Psychotherapy, University of Rostock, Germany. His research focuses on diagnostics, treatment, and neuroimaging correlates of adult ADHD.

Michael Huss, MD, is professor for Child and Adolescent Psychiatry and head of the department at the Johannes Gutenberg-University Mainz, Germany. He also graduated as clinical psychologist working in several research groups at the University of Illinois at Chicago, at the Charité Berlin, and the Gutenberg-University in Mainz, Germany. His research interests are in ADHD, substance use disorders, and depression.

Thomas Jans, PhD, is senior clinical psychologist in the Department of Child and Adolescent Psychiatry, Psychosomatics, and Psychotherapy at the University Hospital of Würzburg, Germany. Research interests focus on treatment of ADHD across the lifespan and on the long-term course of child psychiatric disorders.

Bernhard Kis, MD, is senior physician and heads the Adult ADHD outpatient unit in the Department of Psychiatry and Psychotherapy at the University Medical Center, Göttingen. His research focuses on diagnostic tools and cognition regarding ADHD in adult patients.

Mona Abdel-Hamid, PhD, is a clinical psychologist and has been holding positions as a research assistant and lecturer in the Faculties of Psychology and Medicine of the universities of Bochum, Duisburg-Essen, and Göttingen, Germany. Her primary research interests comprise ADHD and mental health of the deaf.

Ludger Tebartz van Elst, MD, is professor for psychiatry and psychotherapy and deputy director of the Department for Psychiatry and Psychotherapy of the University Hospital, Freiburg. His research focus is on neurodevelopmental disorders and schizophrenia in adults.

Alexandra Philipsen, MD, is professor and the head of the Department of Psychiatry and Psychotherapy at the medical campus of the University of Oldenburg. Her research focuses on the neurobiological mechanism and treatment of adult ADHD.

References

American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorders (4th ed.). Washington, DC: Author

American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Arlington, VA: American Psychiatric Publishing.

Bereiter

(1963). Some persisting dilemmas in the measurement of change. In Harris

C. W.

(Ed.). Problems in Measuring Change. Madison: The University of Wisconsin Press. 3-20.

Busner

Targum

S. D.

(2007). The clinical global impressions scale: Applying a research tool in clinical practice. Psychiatry (Edgmont), 4(7), 28-37.

Christiansen

Hirsch

Abdel-Hamid

Kis

(2014). CAARS. Conners Skalen zu Aufmerksamkeit und Verhalten für Erwachsene. Deutschsprachige Adaptation der Conners’ Adult ADHD Rating Scales (CAARS) von C. Keith Conners, Drew Erhardt und Elizabeth Sparrow [Testbox mit Manual, jeweils 10 Fragebögen Langversion Selbst- und Fremdbeurteilung, jeweils 10 Fragebögen Kurzversion Selbst- und Fremdbeurteilung, jeweils 10 Fragebögen Screeningversion Selbst- und Fremdbeurteilung und den entsprechenden Auswertungs- und Profilbögen]. Bern, Switzerland: Huber.

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.) Hillsdale: Erlbaum.

Conners

C. K.

Ehrhard

Sparrow

(1999). CAARS adult ADHD rating scales. New York, NY: Multi Health Systems.

Fayyad

De Graaf

Kessler

Alonso

Angermeyer

Demyttenaere

. . . Jin

(2007). Cross-national prevalence and correlates of adult attention-deficit hyperactivity disorder. British Journal of Psychiatry, 190, 402-409. doi:10.1192/bjp.bp.106.034389

Hesslinger

Philipsen

Richter

(2004). Psychotherapie der ADHS im Erwachsenenalter: Ein Arbeitsbuch [Psychotherapy of ADHD in adults: A manual]. Göttingen, Germany: Hogrefe.

10.

Hesslinger

Tebartz van Elst

Nyberg

Dykierek

Richter

Berner

. . . Ebert

(2002). Psychotherapy of attention deficit hyperactivity disorder in adults. European Archives of Psychiatry and Clinical Neuroscience, 252, 177-184. doi:10.1007/s00406-002-0379-0

11.

Hirvikoski

Waaler

Alfredsson

Pihlgren

Holmstrom

Johnson

. . . Nordstrom

A. L.

(2011). Reduced ADHD symptoms in adults with ADHD after structured skills training group: Results from a randomized controlled trial. Behaviour Research and Therapy, 49, 175-185. doi:10.1016/j.brat.2011.01.001

12.

Jensen

C. M.

Amdisen

B. L.

Jørgensen

K. J.

Arnfred

S. M. H.

(2016). Cognitive behavioural therapy for ADHD in adults: Systematic review and meta-analyses. ADHD Attention Deficit and Hyperactivity Disorders, 8, 3-11. doi:10.1007/s12402-016-0188-3

13.

Linehan

(1993). Cognitive-behavioral treatment of borderline personality disorder. New York, NY: The Guilford Press.

14.

Michalak

Kosfelder

Meyer

Schulte

(2003). Measuring therapy outcome—Pre-post effect sizes and retrospective measurement. Zeitschrift Für Klinische Psychologie Und Psychotherapie, 32, 94-103.

15.

National Institute for Health and Care Excellence. (2008). Attention deficit hyperactivity disorder: Diagnosis and management of ADHD in children, young people and adults (National Clinical Practice Guideline No. 72). Retrieved from https://www.nice.org.uk/guidance/cg72

16.

Philipsen

Graf

Jans

Matthies

Borel

Colla

. . . Berger

(2014). A randomized controlled multicenter trial on the multimodal treatment of adult attention-deficit hyperactivity disorder: Enrollment and characteristics of the study sample. Attention Deficit and Hyperactivity Disorders, 6, 35-47. doi:10.1007/s12402-013-0120-z

17.

Philipsen

Graf

Tebartz van Elst

Jans

Warnke

Hesslinger

. . . Berger

(2010). Evaluation of the efficacy and effectiveness of a structured disorder tailored psychotherapy in ADHD in adults: Study protocol of a randomized controlled multicentre trial. Attention Deficit and Hyperactivity Disorders, 2, 203-212. doi:10.1007/s12402-010-0046-7

18.

Philipsen

Jans

Graf

Matthies

Borel

Colla

. . . van Elst

L. T.

(2015). Effects of group psychotherapy, individual counseling, methylphenidate, and placebo in the treatment of adult attention-deficit/hyperactivity disorder a randomized clinical trial. JAMA Psychiatry, 72, 1199-1210.

19.

Philipsen

Richter

Peters

Alm

Sobanski

Colla

. . . Hesslinger

(2007). Structured group psychotherapy in adults with attention deficit hyperactivity disorder. Journal of Nervous and Mental Disease, 195, 1013-1019.

20.

Safren

S. A.

Sprich

Mimiaga

M. J.

Surman

Knouse

Groves

Otto

M. W.

(2010). Cognitive behavioral therapy vs relaxation with educational support for medication-treated adults with ADHD and persistent symptoms: A randomized controlled trial. Journal of the American Medical Association, 304, 875-880. doi:10.1001/jama.2010.1192

21.

Shaw

Hodgkins

Caci

Young

Kahle

Woods

A. G.

. . . Arnold

L. E.

(2012). A systematic review and analysis of long-term outcomes in attention deficit hyperactivity disorder: Effects of treatment and non-treatment. BMC Medicine, 10(1), Article 99.

22.

Solanto

M. V.

Marks

D. J.

Wasserstein

Mitchell

Abikoff

Alvir

J. M.

. . . Kofman

M. D.

(2010). Efficacy of meta-cognitive therapy for adult ADHD. The American Journal of Psychiatry, 167, 958-968. doi:10.1176/appi.ajp.2009.09081123

23.

Stieglitz

R.-D.

Baumann

Freyberger

H. J.

(2001). Psychodiagnostik in Klinischer Psychologie, Psychiatrie, Psychotherapie [Psychodiagnostics in clinical psychology, psychiatry and psychotherapy]. Stuttgart, Germany: Georg Thieme Verlag.

24.

Ülsmann

(2013). Psychotherapieerfolg: Eine Frage der Operationalisierung? Konvergenzen und Divergenzen von Psychotherapieerfolgsmaßen [Success of psychotherapy: A question of operationalization? Convergences and divergences of success measures in psychotherapy] (Doctoral thesis). Ruhr-Universität Bochum, Germany.

25.

Vidal-Estrada

Bosch-Munso

Nogueira-Morais

Casas-Brugue

Ramos-Quiroga

J. A.

(2012). Psychological treatment of attention deficit hyperactivity disorder in adults: A systematic review. Actas Espanolas De Psiquiatria, 40, 147-154.

26.

Young

Moghaddam

Tickle

(2016). The efficacy of cognitive behavioral therapy for adults with ADHD: A systematic review and meta-analysis of randomized controlled trials. Journal of Attention Disorders. Advance online publication. doi:10.1177/1087054716664413

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB