When Research Meets Reality–-Lessons Learned from a Pragmatic Multisite Group-Randomized Clinical Trial on Psychosocial Interventions in the Psychiatric and Addiction Field

The Randomized Controlled trial

One main feature of the RCT is that it provides strong internal validity through strict inclusion and strict exclusion criteria. This strong internal validity, however, comes at the expense of poorer external validity which means reduced generalizability of the results to the heterogeneous groups of patients and settings in everyday clinical practice.^6,9,10

Another prerequisite for establishing strong internal validity is a strict manualized treatment conducted by highly trained and specialized therapists. In everyday clinical practice, however, clinicians are facing a wider range of disorders which requires a more eclectic approach tailored to meet the differential needs of the individual patient. Additionally, the treatment is typically provided by health-care professionals with different experiences and training.^9–11 It has often proved difficult to implement manualized treatment approaches as this presupposes basic skills, time consuming training and highly motivated therapists. ¹²

Another main feature of the RCT design is that it intends to establish equal groups in regard to known and unknown confounding factors by randomization. However, this presupposes a sample size that is large enough compared to the number of variables, ⁴ which might not be the case in many randomized clinical trials as difficulties in recruitment are more the rule than the exception.^13–15 It is estimated that 19% of all worldwide pharmaceutical RCTs fail to enroll any patients, and that 90% fail to enroll the target number of patients within the target amount of time. ¹⁶ There is no reason to believe this would be different for studies in the psychiatric and addiction fields as these studies include patients that are normally excluded from most RCTs.^6–8

Another feature of rigorous RCTs is blinding of patients, therapists and researchers to treatment allocation. However, in studies with complex interventions blinding is often difficult or even impossible to accomplish. This means that patients might withdraw from the study when allocated to a less preferred treatment, and by this pose a threat to the between-group equivalence.^1,17 Additionally, in clinical practice the choice of treatment is usually a result of collaboration between the patient and the therapist. ⁹

The Pragmatic RCT

These discrepancies between the “ideal research” and the “real-world” settings have motivated a distinction between efficacy and effectiveness studies along with a search for complementary research methods. Efficacy studies investigate the effect of specific treatment factors under ideal conditions, like strict RCTs with a strong emphasis on internal validity. Effectiveness studies, on the other hand, with a stronger emphasis on external validity, investigate the effect of treatments under “real-world” conditions.^18,19 When it comes to complementary research methods, these often involve modifications of the traditional methods. All such modifications, however, create new challenges.

One of these complementary research methods is the pragmatic RCT. ⁹ The key aim of the pragmatic RCT is to reflect the heterogeneity of patients in clinical practice by using broader inclusion criteria and keeping exclusion criteria to a minimum, thereby enhancing the generalizability of the RCT to patients and settings in everyday clinical practice. Consequently, the pragmatic RCT can be regarded as lying in between efficacy- and effectiveness studies. The pragmatic RCT is also more flexible in defining the intervention and tends not to use a treatment manual. Usually the concern of the traditional RCT is to evaluate specific active ingredients in the treatments provided. The pragmatic RCT, investigating more complex interventions, is more interested in examining each treatment as an entity or a “black box”. ⁹ Such interventions might be impossible or even unethical to compare with placebo. An alternative would be to compare the experimental treatment to “treatment as usual” (TAU). This means that the patients in the control group receive the treatment they would normally have received. However, TAU is a difficult term to define because it depends greatly on the preference, skills, knowledge and resources of the therapists delivering it. ⁹

Multi-Center trials

A well-known challenge in conducting trials evaluating complex psychosocial interventions is that the expected effect-sizes tend to be small, especially when the control condition is TAU or another active treatment which will also constitute a treatment effect. Frequently, such studies require large sample sizes to ensure sufficient statistical power. Additionally, inclusion, treatment adherence and follow-up is often challenging when studying patients with multiple disorders. This renders studies vulnerable to type 2 statistical error. One way of meeting this challenge is to use a multi-center design. If no single center has a sufficient case load, trials including patients from several centers might provide an adequate study population. In multi-center studies, the randomization of participants might be done on an individual or center-level, ie, the individually randomized group treatment trial (IRGT) or the group-randomized clinical trial (GRCT). ²⁰ Individual randomization is usually applied when the intention is to study effects at the individual level. The group-randomized clinical trial is useful in examining the context in which the therapy is given or the differences of interventions in specific groups (schools, hospitals, etc.). ²⁰ Additionally, GRCTs may reduce the risk of contamination between treatment conditions. In treatment research on psychosocial methods it is difficult to hide or mask the different methods that are to be randomized. This creates a risk of contamination of knowledge between the therapists and patients in the intervention and control groups. In the GRCT the different centers are allocated to the different treatment conditions as a whole. All the therapists in one center will therefore belong to the same treatment condition, minimizing the risk of contamination. ²¹

However, multi-center trials represent organizational, administrative, economic, treatment fidelity and statistical challenges. In order to secure inclusion, fidelity, protocol adherence and data quality, it is essential to have a trial administrator at each center managing and monitoring the trial.^1,22 Additionally, one has to consider clustering of participants in both types of multi-center trials. In the IRGT clustering may develop during the course of the trial, whilst one has to consider clustering from the start of the trial in the GRCT. This clustering threatens the statistical assumption of independent observations and should be accounted for by hierarchical statistical analyses.^23–25 Further, such intra-class correlations may reduce statistical power and should be compensated for in protocol sample size calculations.^26–28

Center recruitment

Considerable time was spent on recruiting CMHCs into the trial. Thirty five CMHCs from 3 out of 5 Regional Health Trusts were invited. More information was provided through meetings at the CMHCs that responded positively to the invitation. Most of the CMHCs were positive with respect to gaining knowledge about the effects of the intervention, but reluctant in regard to the workload demanded by such studies. In the end, 10 CMHCs agreed to participate in the study. However, one center withdrew just before randomization, leaving us with a total of 9 centers. The recruitment process significantly delayed the initiation of the project and put the trial behind schedule from the beginning.

The contracts between the participating CMHCs and the project management would have to be worded differently between the experimental and the control group. We therefore decided to make the written agreements with each participating center after treatment allocation. This turned out to be a mistake as one of the centers allocated to the control group withdrew 2 months into the project. They stated this was due to workload and staffing problems. During the course of the trial, another center in the control group did not manage to include any patients. As failing to include the target number of patients would mean risking type two statistical error, several attempts were made to include additional centers to the control group even though it would have compromised the RCT design. Towards the end of the study period, we managed to include two more centers which, however, did not manage to include any patients. This left us with 5 centers in the Intervention Group (IG) and two centers in the Control Group (CG) (Fig. 1).

OPEN IN VIEWER

Figure 1

Flowchart of CMHCs on centre level.

Patient recruitment

A total of 5323 patients were referred to the CMHCs during the inclusion period (Fig. 2). Only 35% of these patients were actually screened with the AUDIT and the DUDIT (CG: 50%, IG: 27%) with a range from 17%–64% between CMHCs (data not shown). Of these 1867 patients, 88 (5%) handed in a blank form (CG: 7%, IG: 3%). All of the blank forms from the CG came from the CMHC that did not manage to include any patients and constitutes 20% of the screened forms from that CMHC.

OPEN IN VIEWER

Figure 2

Flowchart of patients in the intervention group and the control group.

Eighteen per cent of the screened patients scored above the cut-off level of the AUDIT and/or the DUDIT (CG: 10%, IG: 27%), ranging from 5%–34% between centers. Only 31% of those that scored above the cut-off level were actually referred to the local trial administrator for the baseline evaluation (CG: 33%, IG: 30%), ranging from 0%–50% between centers. The main reason for not being referred was “no oral consent to participate in the study” (CG: 32%, IG: 59%). Another reason was that the clinicians deemed the patient's motivation for substance use treatment as being too low (IG: 0.04%) or that the patient was not in need of such treatment (CG: 14%). The latter reason was only stated by the CMHC that did not manage to include any patients.

In the end, only 81 patients were included in the study. This was 4% of the total number of screened patients and 24% of the patients that scored above the cut-off level of the AUDIT and/or the DUDIT (CG: 2%, 25%, IG: 6%, 23%), ranging from 0%–7% and 15%–40% respectively, between centers. After inclusion, 2 patients in the CG and 3 patients in the IG withdrew their consent, leaving us with 21 patients in the CG and 55 patients in the IG. This means that the number of participants available for the intention to treat analyses were 76, ranging from 6–16 participants between centers.

After inclusion, two patients in the IG never met for treatment, 11 patients received less than 5 sessions (CG: 3, IG: 8) and 14 patients never returned for follow-up interviews (CG: 3, IG: 11) (Data not shown). This left us with 56 completers, defined as receiving 5 sessions or more and meeting for at least one follow-up interview (CG: 17, IG: 39), ranging from 4–14 between centers. Only 48 patients had at least 5 sessions and met for both follow-up interviews (CG: 15, IG: 33) (data not shown). There was no significant difference in the number of completers between the IG and the CG (χ ² = 0, 80; df = 1; P = 0, 374).

There were three main differences in the referral policy of patients with SUD across centers. At one extreme, the centers referred all patients with SUD to the addiction outpatient unit except patients with co-occurring schizophrenic spectrum disorders. Other CMHCs treated patients with co-occurring SUD at the psychiatric outpatient clinic as long as the SUD was not severe (harmful use and abuse only). At the other extreme, one CMHC treated all patients with mental illness at the psychiatric outpatient clinic regardless of co-morbid SUD. We found that the CMHC that failed to include any patients was amongst the four CMHCs that referred all patients with co-morbid SUD to the addiction outpatient unit. The one CMHC that did not refer patients with SUD elsewhere was the only CMHC that managed to include the target number of patients during the initial inclusion period. Otherwise, we did not find any relationship between the center referral policy and the number of people that scored above the threshold level of the screening instruments (data not shown).

The inclusion period proved to be insufficient. After the planned inclusion period of 9 months only one CMHC had managed to recruit the target number of patients and only 55 patients were included in total (CG: 14, IG: 41). The remaining CMHCs and their coordinators were willing to continue recruiting for another 3 months. Nevertheless, after those 3 months only 74 patients were included (CG: 18, IG: 56) (data not shown). The remaining CMHCs were asked to continue recruiting until the target number of patients was included or until the end of the year (another 9 months). All CMHCs from the CG and one CMHC from the IG accepted this. Still, patient enrolment was slow and one CMHC (CG) did not manage to recruit any patients at all by the end of the additional year.

In the CMHC that managed to include the target number of patients during the initial recruiting period, another problem arose. They had so many eligible patients that it exceeded the treatment capacity of the clinicians that were trained in the experimental intervention. They therefore stopped recruiting when the minimum number of required patients was included, 2 months ahead of schedule. To address this problem we chose to train two additional therapists at that clinic.

Follow-up interviews

The next challenge was to get patients to return for the follow-up assessments. One CMHC (IG) managed to complete follow-up interviews for all patients at 6 and 12 months. At the other extreme, one CMHC (IG) lost 6 out of 14 patients at both follow-up sessions. The other CMHCs lost 1–2 patients each from both follow-up sessions (data not shown). In order to reduce drop-out from the follow-up assessments, the patients were offered €25 for each follow-up session as a compensation to cover their travel expenses. The Regional Medical Ethics Committee did not accept a higher monetary incentive, to prevent it from becoming an incentive for participation on its own.

Recruiting centers

The first time-consuming challenge was recruiting centers. Thirty five CMHCs were asked to participate and only 9 accepted. The main reasons for declining participation were challenges in the clinics with respect to time and financial demands. Other reasons were shortage of specialists, lack of a local supportive infrastructure, inadequate research training, and data collection challenges. It seems that many clinics value research and want to contribute to it, but few have the resources needed to follow it through. This was clearly demonstrated when one of the participating CMHCs withdrew 2 months into the project.

Researchers have a formidable challenge in both motivating centers to contribute to research and to keep the burden of participation at an acceptable level. If the burden of participation is too high, the study will run the risk of collecting incomplete and missing data. To accomplish good motivation, adherence and implementation of research methods in the clinic, it is important to involve the clinics in the project at an early stage and to let them contribute actively in the planning and execution phases of the project.^36,37 This will both enhance the centers’ feeling of “ownership” towards the study, and also contribute to a design that is in accordance with the clinics’ needs and workload. However, making contracts with the clinics at an early stage might be premature due to high application refusal rates.

Some of the clinic leaders stated that they would not want to participate in the study if they were allocated to the control group, as it would not be worth the effort for them. This highlights further the importance of including the centers at an early stage of the study planning process and to use strong acknowledgement and reinforcement strategies. Strong reinforcements are essential to make the effort of participation worthwhile regardless of allocation. One reinforcement strategy could be to give the co-workers at the centers a role that fulfills the Vancouver requirements for co-authorship. ³⁶ In our study the control group centers were promised ownership of all data from their own center along with treatment manuals and teaching in the experimental intervention after the study.

The local trial administrators have an important role in the study and it is essential to select appropriate people for these positions. However, we did not have the luxury of selecting between several candidates for the job. At most CMHCs it was difficult to find even one single interested person. Thus, the local trial administrators were a mix of people with varying education, training, interests, motivation and time available for the trial. Furthermore, the more centers that are involved in a multi-site trial and the larger the geographical area they are spread over, the greater the responsibility and project work-load for the local trial administrators. This could create differences in how the data is collected between centers.

Recruiting patients

The therapists at the CMHCs were supposed to screen all new referrals with the AUDIT and the DUDIT. However, only 35% were actually screened. According to the local trial administrators this was due to several factors. Some therapists felt the screening was a new burden on their already busy schedule, some felt insecure addressing the issue of alcohol and drug use and that it was particularly awkward to introduce this subject during the first session and some were concerned it would threaten the therapeutic alliance to address such a delicate issue. In one center the leader had resigned shortly after the project started. His successor had little interest in the project and did not encourage or support the staff to continue screening. Most local trial administrators, however, commented that they found the screening relevant and some of the clinics even wanted to keep it as part of their routine after the project was over.

Our understanding is that these problems were insufficiently considered. Therapists are often uncomfortable with addressing the issue of alcohol and drug use with the patient and need extra coaching.^38–42 Unless the importance of screening is strongly emphasized by the local trial administrator and/or the research group and followed by strong support from the clinic management, such new routines are easily forgotten. Most clinical research projects require the establishment of new routines to gather the required data for the project. Implementation of new routines and treatment methods has long shown to be a challenge in most settings and require formidable resources on many levels.^43,44 Ideally, such routines should be well established before the start of the study. ³⁶

An unforeseen problem was the very low prevalence of patients with problematic alcohol and drug use identified by the AUDIT and the DUDIT in some of the CMHCs. From previous prevalence findings of SUD in psychiatric outpatient clinics, one would expect this number to be higher.^45,46 This might be explained by different referral policies between centers. During the planning phase of the project new national mental health services guidelines were published. These guidelines were interpreted differently by the centers. One interpretation was to establish separate addiction outpatient units as part of the CMHC, rendering the psychiatric outpatient clinic of the CMHC almost deplete of patients comorbid of mental and SUD.

The therapists were asked to refer all patients that scored above the cut-off level of the AUDIT and the DUDIT to the local trial administrator for baseline evaluations. However, only about a third were actually referred. The main reason for this was that the patients did not want to participate in the study. In the Intervention Group, participation in the study could imply a change of therapist to one that had learned the intervention. This, however, does not explain why the patients in the Control Group refused to participate. This could be due to how the therapists presented the project to the patients, perhaps reflecting their own attitudes towards the project.^44,47 At one CMHC the clinicians deemed 13 patients as not needing treatment although they scored above the threshold level of the screening instruments. Considering this was the CMHC that did not manage to include any patients, this could mean that the project was not well enough implemented in that clinic. Several studies have stated that a prerequisite for implementing a trial into a clinical setting is a supportive clinic management that motivates the clinicians to participate.^43,44

We soon realized that inclusion was happening too slowly, so we contacted the CMHCs and urged their management to make sure that the psychiatric outpatient clinics would take responsibility for their share of patients with co-occurring SUD instead of referring them elsewhere. This created several dilemmas. In the Control Group it would be unethical to keep the patients with co-occurring SUD at the psychiatric outpatient clinics if they did not feel they had the competency to treat them there. On the other hand, different policies in handling referrals in regard to co-occurring SUD between the IG and the CG would bias our sample. Additionally, our Intention to Treat (ITT) design would keep the patient data in the group they were allocated to even though the patients were referred to another treatment facility (eg, addiction treatment) after inclusion in the study. This means that if the CG would refer their patients to an addiction treatment facility, it would make it difficult to conclude whether there were any real differences in the interventions given between the IG and the CG.

To address the slow inclusion rate, we extended the inclusion period by one year. Extending the inclusion period is problematic in several ways. It exceeds the original funding and increases the risk of having to deal with therapist turn-over.^48,49 During the first year of the project, we had to train 4 additional therapists. We experienced that the extended inclusion period did not increase inclusion numbers to a great extent. During the initial inclusion period we included about 37% of the calculated sample size needed. After the first 3 months of the extended inclusion period we had included 49% of the calculated sample size, and we ended up with only 51% (ITT) of the calculated sample size after an additional year of recruiting. This means that the cost of extending the inclusion period is greater than the benefit.

Follow-up assessments

Not unexpectedly, follow-up proved challenging. As the first CMHC to finish including patients experienced great difficulties in reaching the patients for follow-up assessments, we applied for additional funding to pay the participants a symbolic amount for their participation in the study. This necessitated additional approval from the Regional Medical Ethics Committee. We are unsure if this payment had an effect on follow-up rates. The additional strategies employed seem more important. Many of the local trial administrators used a lot of creativity in reaching the patients for their follow-up assessments, such as interviewing patients that had moved to other parts of the country by phone and mail and making appointments for interviews at the patient's workplace when the patient was unable to visit the clinic. Our experience is that the local trial administrators who put more effort into the task than was expected of them and who where the most creative, managed to reach the most patients for their follow-up assessments. Our interpretation is that this type of research puts heavy demands on the participating centers and local trial administrators, and that these demands and costs are often underestimated. The result might be under-funding and the use of inadequate strategies which could easily endanger multi-site studies. Securing adherence is a complex issue that requires multiple and comprehensive strategies at different stages of the trial.^50,51 This requires the researchers to be directly involved and integrated into the research work of each center, thereby improving data quality and adherence.^43,44

Ethical considerations

Commonly, clinical trials encounter problems in recruitment, follow-up and adherence which may reduce the studies’ probability of demonstrating an effect (type 2 statistical error). This is ethically problematic because one risks wasting financial and human resources. The Medical Ethics Committees therefore make an assessment of the realism in each project prior to approval. When encountering such challenges during the course of the trial the researchers are faced with the choice of aborting the study or reinforcing the measures. To be able to make these choices at the appropriate time the researchers need to monitor the project continuously and to have created strategies on how to handle such difficulties in advance. In our case, we chose to reinforce the measures by training additional therapists, extending the inclusion period and providing monetary incentives. The latter two reinforcement strategies necessitated additional approval from the Regional Ethics Committee. In either case, whether the researchers choose to abort the study or to reinforce the measures, the researchers have an ethical obligation to report these challenges in a way that prevents other researchers from encountering the same problems.

Summing up

Studies of complex interventions in unselected clinical populations are essential in the development of evidence based treatments in the psychiatric and addiction field. The methodological problems are considerable but it is possible to overcome the challenges by thorough planning and addressing the obstacles at an early stage.