Abstract
Monitoring outcomes in psychotherapy is important for advancing research into empirically supported treatments, collecting data to inform practice-based evidence and for monitoring patient treatment responses during routine care. We implemented two routine outcome monitoring measures in a community psychology clinic in South Africa, namely, the Outcome Rating Scale and Kessler Psychological Distress Scale. In this article, we adopt a case study methodology to describe and critically reflect on the implementation process and evaluate the utility of the outcome data collected during the first year of implementation. Our findings suggest that the measures have some utility but that there were significant problems with implementation. We found that Outcome Rating Scale and Kessler Psychological Distress Scale scores correlated in the expected direction; however, Outcome Rating Scale intake scores for the present population were at odds with the measure’s norms. The findings highlight the care that needs to be taken to change routine clinical practice when outcome measures are implemented in local settings and the need for research to validate the use of these ROM measures in South African clinical settings.
Keywords
Measuring outcomes in psychotherapy is important in order to advance research into empirically supported treatments, inform evidence-based practice, amass practice-based evidence, and monitor patient treatment responses during routine care. In spite of the potential advantages of using outcome measures to these ends, there is scant data on the use of psychotherapy outcome measures in South Africa (SA). Internationally, scholars have argued that many mental health professionals are sceptical of the utility of such measures and are cautious about applying empirical data to individual cases (Crits-Christoph, Gibbons, & Mukherjee, 2013). Literature also suggests that many psychotherapists are not substantially influenced by empirical findings when they construct their case formulations and implement treatment plans (Cohen, Sargent, & Sechrest, 1986; Morrow-Bradley & Elliott, 1986).
In SA, there is little data regarding the use of routine outcome monitoring (ROM) in clinical practice. Within this context, we implemented, and evaluated the use of, two ROM measures in a community psychology clinic in the Western Cape, SA. The Outcome Rating Scale (ORS) and Kessler Psychological Distress Scale (K10) were implemented in routine clinical care during 2014. In this article, we employ a case study methodology to describe and critically reflect on the process of implementing a ROM system and to report the psychometric performance of these measures in this context. In critically describing and assessing the implementation of psychotherapy outcome measures in routine practice, we provide (to the best of our knowledge) the first account of implementing these scales in SA.
The utility of ROM systems
There are a number of reasons to explore the use of ROM measures in the practice of psychotherapy. Measuring psychotherapy outcomes can help to advance evidence-based practice, collect practice-based evidence, and improve clinician responsiveness by detecting patients at risk of treatment failure. Nonetheless, the implementation of ROM systems is not without challenges and clinicians have been slow to adopt the routine use of psychotherapy outcome measures.
Gathering evidence on the efficacy of psychotherapeutic interventions has been a priority in clinical research ever since the mid-century reviews of Eysenck (1952) and Levitt (1957) questioned the effectiveness of psychotherapy. Subsequently, a substantial body of research has attempted to answer questions about which psychotherapy interventions work for whom and the variables responsible for therapeutic change (Lambert & Ogles, 2004). This has spurred a drive for the implementation of therapies informed by empirical evidence and given rise to the Evidence-Based Practice (EBP) movement (APA Presidential Task Force on Evidence-Based Practice, 2006; Institute of Medicine, 2001; Sackett, Rosenberg, Gray, Haynes, & Richardson, 1996). Clinicians have, however, questioned the relevance and utility of using empirical evidence and empirical data to direct the care of individual patients (Kazdin, 2008); clinicians have argued that clinical intuition and qualitative descriptions of patients’ progress in therapy are more useful than standardised quantitative measures and that the kinds of gains made in psychotherapy cannot be easily measured or reduced to a simple index. This has placed psychotherapy practitioners and psychotherapy researchers at odds with one another; psychotherapists are doubtful about the relevance of quantifying therapeutic change, and researchers are worried about practicing while failing to quantify therapeutic change (Kazdin, 2008).
The scientist–practitioner divide appears to be marked in SA. Kagee and Lund (2012) have argued that the international movement for evidence-based practice has ‘had little impact among South African clinical and counselling psychologists’ (p. 103). This is surprising, not least because of the resource constraints in SA and the need for psychotherapists to maximise the benefits of scarce mental health care resources (Lund et al., 2010; Lund & Flisher, 2009).
Kazdin (2008) argued that ROM is the key to bridging the scientist–practitioner gap and to making research more relevant to clinical practice. These tools provide an empirical, cost-effective, and practice-centred means of uniting the ends of clinical work with the goals of research by producing practice-based evidence. Practice-based evidence is an approach to generating scientific knowledge about psychotherapy which engages practitioners in the research process. This bottom-up approach starts from the context in which therapists are working and addresses questions of interest to practitioners. One of the central components of generating practice-based evidence is the adoption and implementation of an outcome monitoring system such as ROM.
The use of ROM in psychotherapy not only allows practitioners and researchers to collect data within a naturalistic setting (Margison et al., 2000) but also allows psychotherapists to monitor the progress of each client (Charman, 2004). ROM data can be mapped as an individualised recovery curve which can then be compared with normed curves to establish whether clinically significant change has been achieved (Charman, 2004). Data obtained from ROM may enable clinicians to assess the extent to which their clients are responding to treatment, to improved functioning or symptom remission (Charman, 2004). These data may be useful in guiding clinical decision making tailored to individual client needs (Charman, 2004). Receiving routine feedback about individual patient treatment outcomes can enhance practitioners’ treatment responsiveness and planning, particularly for patients who may be on course for treatment failure.
If we hope to amass evidence of what kinds of psychotherapies work in local contexts, develop culturally informed indigenous psychotherapies, and allocate mental health resources rationally, we will need to engage critically with issues pertaining to evidence and measuring psychotherapy outcomes. One manner in which we can begin this process is through implementing ROM systems. Despite the apparent usefulness of psychotherapy outcome measures and clinicians stating that they have an interest in reliable outcome information, a number of surveys evaluating the implementation of ROM have shown poor implementation of these measures in practice (Boswell, Kraus, Miller, & Lambert, 2013; Miller, Hubble, Chow, & Seidel, 2013). Research is needed to try and understand the reasons for the poor uptake of these measures in routine practice. This need is especially relevant in SA, where there is a paucity of studies on the implementation of ROM in local clinical practice settings (Young & Edwards, 2013).
Measures used to monitor psychotherapy outcomes
There are numerous scales available for assessing treatment outcomes in psychotherapy, although no core battery is widely accepted (Crits-Christoph et al., 2013). Table 1 provides a summary of some commonly utilised ROM measures. There are a number of factors to consider when selecting a ROM system or scale; consideration needs to be given to the validity and reliability of the instruments, their clinical utility, and their sensitivity to change. To this list of generic concerns, we might add that there are particular considerations for local practitioners when selecting ROM measures, including issues of cultural appropriateness, cost, and the availability of validated translations of the instruments into indigenous languages.
Examples of psychotherapy treatment measurement systems and questionnaires.
All of the ROM instruments listed in Table 1 rely on client self-report measures. Thambirajah (2005) points out that this is potentially problematic as self-report measures rely on three implicit assumptions. First, self-report measures assume that clients are reliable observers of their own behaviour and have sufficient insight and self-knowledge about the attribute being assessed. Second, they presume that clients’ feelings about themselves are stable irrespective of the situation. Finally, they assume that individuals are willing to share truthfully their perceptions about themselves.
Study setting
Given the potential utility of ROM measures, we were interested in implementing a system to monitor the treatment outcomes of patients receiving care in an outpatient community psychology clinic in the Western Cape province of SA (hereafter referred to as the clinic). The clinic provides a range of free psychological assessment and individual, group and family psychotherapeutic services to members of the community who would not otherwise be able to afford care. The clinic also serves as a training site for 10 clinical psychology master’s students who are enrolled in the first year of their professional training at a local university. These student psychologists practice under the close supervision of registered psychologists. Clients who access care at the clinic typically experience problems related to depression, anxiety, exposure to trauma, family conflict, violence, substance abuse, scholastic difficulties, and interpersonal problems.
During 2014, a total of 100 patients received care at the clinic and 779 sessions of psychotherapy were conducted. In general, clients’ attendance rate is erratic and there is a high rate of patient attrition. In part, this is the result of barriers to care such as the transport difficulties patients experience to reach the clinic.
Method
A case study methodology was employed to (1) describe and critically evaluate the implementation of a ROM system in a community psychology clinic and (2) explore the psychometric properties of the ROM measures in this clinical context.
Participants
Qualitative data were collected in the form of field notes via participant observation during 2014 by one of the authors (J.B.), who was at the time responsible for the implementation of the ROM system and for overseeing the running of the clinic. These qualitative data served as field notes which were a record of the implementation process.
Quantitative data about the performance of the ROM measures and the implementation of the ROM system were collected from a consecutive sample of all clients 14 years and older that sought individual psychotherapy services at the clinic during 2014 (n = 44). The sample comprised more females (n = 35) than males (n = 9). The majority of the sample consisted of White (n = 20) and coloured (n = 19) clients, with four Black clients and one Indian client. The ages of clients in this sample ranged from 14 to 65 years, with a mean of 30 years (SD = 11.7).
Instruments
The following instruments were used to monitor psychotherapy outcomes.
ORS
The ORS provides an indication of clients’ subjective sense of well-being in their personal, interpersonal, social, and general life functioning (International Centre for Clinical Excellence FIT Manuals Development Team, 2011a). The ORS is a brief visual analogue version of the OQ-45.2 (Outcome Questionnaire) scale. The rationale behind the ORS’s development was the protracted length of time it took for clients to complete the full OQ-45.2 measure in practice (Miller, Duncan, Brown, Sparks, & Claud, 2003). The ORS items consist of four different lines, each representing a different domain of functioning: individual, interpersonal, social, and overall well-being (Miller & Bargmann, 2012). The ORS is suitable for individuals above the age of 13 years. Past evaluations of the ORS’s concurrent validity yielded correlations that range between −.53 and −.75 for the ORS and OQ-45.2 scale (International Centre for Clinical Excellence FIT Manuals Development Team, 2011b). This measure has also shown very good specificity between clinical and non-clinical cases. Research has found a significant difference in the pre-test scores of clinical and non-clinical samples at p < .0001 level (Miller et al., 2003). With regard to reliability, the ORS has displayed internal consistencies that range between Cronbach’s alphas of .87 and .96 (Bringhurst, Watson, Miller, & Duncan, 2006; Campbell & Hemsley, 2009; Hafkenscheid, Duncan, & Miller, 2010; Miller et al., 2003).
K10
The K10 indicates the severity of clients’ distress, specifically related to depression and anxiety symptoms (Kessler et al., 2002). The K10 was initially designed to be used as a screening tool for non-specific distress in the National Health Interview Survey in the United States of America (Andrews & Slade, 2001). The K10 is designed for measuring previous 1-month levels of distress (Andrews & Slade, 2001). More specifically, the K10 has been demonstrated to capture variability in two domains: depression and anxiety (Brooks, Beard, & Steel, 2006). The K10 has shown good criterion validity among HIV+ and pregnant women populations in SA (Spies et al., 2009a; Spies et al., 2009b). Using the SCID (Structured Clinical Interview for the DSM) as gold standard, the K10 was able to identify true cases and non-cases of major depression 73% and 77% of the time (Spies et al., 2009a; Spies et al., 2009b). Furthermore, the K10 displayed moderate criterion validity for a sample representative of the general SA population in the South African Stress and Health (SASH) study, using the Composite International Diagnostic Interview as gold standard (Andersen et al., 2011). In the SASH study, the K10 was able to correctly identify major depression 63.9% of the time, any anxiety disorder 65.3% of the time, and generalised anxiety disorder 69.3% of the time (Andersen et al., 2011). With regard to reliability, the K10 has displayed good overall internal consistencies in SA studies. Cronbach’s alphas in these studies ranged between .85 and .92 (Kuo & Operario, 2011; Louw et al., 2012; Peltzer et al., 2012; Pengpid, Peltzer, & Heever, 2011). The K10 is specified to measure previous 1-month levels of distress and was thus administered every fourth week (Andrews & Slade, 2001).
Procedure
During 2014, one of the authors (J.B.) kept field notes documenting the implementation process of the ROM system. In June 2015, data were retrospectively extracted by one of the authors (A.S.) from the folders of all patients 14 years and older who accessed psychological care at the clinic in 2014. Data were checked and verified. Quality checks were also performed to identify inconsistencies and inaccuracies in the data collection. The following data were collected from patient folders:
Demographic data. Patients’ age, sex, and race were recorded from the referral form.
Treatment details. The total number of therapy sessions that the client attended were recorded from the client contact summary form in the patient folder.
Implementation of ROM. Qualitative data on the implementation of the ROM were collected by examining each of the client’s individual ORS or K10 forms and noting incorrect use of these measures, particularly responses that suggested the measure was misunderstood, and inconsistent use of the measures. These qualitative data were used to augment the field notes describing the implementation process.
Responsiveness to treatment. Data on each patients’ responsiveness to treatment were collected by recording the total ORS scores and each of the four subscale scores for each administration of the ORS. This yielded ORS data on intake and discharge as well as information about patients’ responsiveness to treatment during the course of therapy.
Ethical considerations
Permission to investigate the files of clients that attended the clinic during 2014 was granted by the clinic manager. The clinic manager provided permission on the condition that a confidentiality agreement was signed with the clinic. Ethical approval to conduct the study was also received from the Research Ethics Committee (REC) of Stellenbosch University (#SU-HSD-000364). It was, however, not possible to obtain informed consent from clients for their outcome data to be investigated in this study. At the time of the ORS and K10’s first administration, it was not known that clients’ ORS and K10 data will be used for research purposes. For that reason, clients were not requested to provide informed consent. The issue of informed consent was, however, made explicit in the application for ethical approval that was submitted to the ethics board at Stellenbosch University.
Data analysis
Field notes were analysed using thematic analysis and were summarised in the form of a narrative account of the implementation process. This narrative account describes the implementation process, the rationale for the choice of ROM measures, the procedures followed to implement the system in the clinic, and problems with implementation that were identified.
Statistical methods were used to analyse the ORS and K10 scores. All analyses were conducted using STATISTICA, version 12. Normal probability plots were inspected for deviations from normality and the presence of outliers. The following statistical analyses were performed:
Pearson product–moment correlations were performed to examine the relationship between the ORS and K10.
A McNemar test was performed to determine whether ORS and K10 scores conformed to the standard ORS and K10 outcomes reported in the literature. The percentage of clients that scored below the predetermined ORS cut-off value of 25 for the first session was established using a McNemar test. The McNemar test is useful for comparing pairs of responses at two different time points (Spencer et al., 1998). In the context of this study, the McNemar test provided an indication of the percentage of clients that scored above and below the ORS cut-off of 25 in the first and last session, respectively.
One-way analyses of variance (ANOVAs) were conducted to establish whether there was a significant difference between the first and last session means of clients’ ORS and K10 scores. This provides a measure of the responsiveness to treatment.
The predetermined ORS criteria for clinically significant change were used to construct a histogram comparing how many clients met the criteria for clinically significant change by the end of therapy and how many did not. The criteria for clinically significant change are (1) crossing the cut-off value of 25 at the termination of therapy and (b) displaying a five-point change between the intake and termination of therapy (International Centre for Clinical Excellence FIT Manuals Development Team, 2011b).
Results
Implementation process
In 2013, a decision was made to implement a ROM system in the clinic. In deciding which outcome measures to implement, the following criteria were taken into consideration by the clinic’s management team: (1) efficiency of use, (2) affordability, and (3) effectiveness. The published data on the psychometric properties of different outcome measures were considered and efforts were made to look for an instrument that had been validated for SA use. Ultimately, the ORS and the K10 were chosen. Chief among the reasons for selecting the ORS were the fact that this instrument provides an indication of clients’ subjective sense of well-being in their personal, interpersonal, social, and general life functioning and has been extensively used in the United States (International Centre for Clinical Excellence FIT Manuals Development Team, 2011a). There is a growing body of literature describing the clinical utility and validity of the ORS, as such it is a widely used instrument in the field of psychotherapy outcome monitoring (Baer & Blais, 2010). There is, however, no published data on the validity of the ORS for use in SA. A decision was thus also made to collect treatment outcome data on all patients with the use of the K10. The K10 measure was included, because unlike the ORS this scale has been used in SA. Although the K10 is not typically used as a psychotherapy ORS, it is aligned with other ORSs, such as the Outcome Questionnaire-45.2 (OQ-45.2), which monitors symptoms of psychological distress (see Table 1). The ORS and K10 were translated into Afrikaans. The ORS was also translated into isiXhosa for use in the clinic.
The ROM system was implemented in 2014. All student psychologists and supervisors received training in the use and interpretation of these ROMs and literature was provided about the clinical utility and application of these measures. Student psychologists were instructed to use these measures routinely with all patients accessing care at the clinic. Resistance was encountered from some supervisors who raised concerns that these measures would have an adverse impact on the process of psychotherapy and that the use of such tools was not consistent with a psychodynamic approach to therapy. These concerns were noted but it was still decided to proceed with the implementation, since no evidence could be found to support the concerns raised by supervisors. The decision to proceed with implementation was also premised on the belief that the potential benefits of using ROMs had been well demonstrated in the literature and that these potential benefits outweighed the problems perceived by some of the supervisors.
In June 2015, quantitative data pertaining to the ORS and K10 were retrospectively extracted from the folders of all patients who access psychological care at the clinic in 2014. The statistical analysis of this sample of data is described below. During this data extraction process, several problems associated with the implementation process were observed. Approximately 5.41% of the ROM data was missing from patient files. The authors observed inaccurate and incorrect use of the ORS instrument and evidence that the data generated were not utilised by the clinicians to inform the course of psychotherapy.
On some ORS forms, it was not clearly indicated which session the ORS form represented. For example, there was an instance in which a client had attended four sessions. However, of the four ORS forms in the client’s file, two were labelled as ‘session 4’ and one was labelled as ‘session 5’. Furthermore, there was also evidence to suggest that some clients did not fully understand how to respond to the ORS. For instance, the ORS requires clients to rate their level of well-being with an ‘x’ on each of the four lines representing a particular functioning domain. However, one client treated the lines like a questionnaire as he responded in writing on the lines. His responses were as follows: ‘I’m good, still a school boy’ (individual well-being); ‘Alone, don’t worry about family’ (interpersonal well-being); ‘vriendskappe vol op’ (social well-being); and ‘ek voel okay het nie klagtes nie’ (overall well-being). In another instance, the clinician included a note in a client’s file that read ‘discontinue ORS use – patient does not seem to understand it’. This particular client rated her well-being by marking an ‘x’ either outside of the line on the extreme end or outside of the line on the lower end, but not on the actual line.
Furthermore, there were some instances in which the total ORS scores of clients had to be calculated during the study’s data collection, as it had not been done by the clinician. With the ORS, clients’ ratings are scored using measurement in centimetres, where each centimetre represents a score of 1. For each functioning domain, a total score of 10 can be obtained. Each line should therefore be 10 cm in length. However, as the ORS values of those whose scores had not been calculated were measured, it was found that the ORS lines were longer than 10 cm.
Some of the ORS data were rendered impractical as a result of clients erroneously responding to the visual analogue items of the measure. The K10 was not consistently administered after every third session. Instead, there were several instances in which the K10 was administered the first, second, or fourth session after the session in which it had been administered before.
The relationship between ORS and K10 scores
Pearson’s correlations between clients’ scores on the overall well-being subscale of the ORS and their levels of depression (r = −.4568, p = .0035) and anxiety (r = −.4093, p = .0097) on the K10 revealed statistically significant relationships. Both correlations displayed a moderate relationship in the negative direction. The correlations between clients’ individual, interpersonal, and social sense of well-being with their levels of depression and anxiety-related distress also displayed negative relationships; however, these were not statistically significant. The total score of each ORS subscale (i.e., individual, interpersonal, social, and overall well-being) was correlated with the total score of each K10 subscale.
Responsiveness to treatment
Of the clients who entered therapy, 82% had a score below or equal to the ORS cut-off value of 25. There was a significant difference between ORS mean of clients for the first session (M = 19.86, SD = 8.33, 95% confidence interval, CI: [17.16; 22.57]) and ORS mean of clients for the last session (M = 24.36, SD = 10.17, 95% CI [20.87; 27.86]) in the positive direction, with F(1, 32) = 6.1697, p = .01842. Cohen’s effect size value (d = .49) indicated a medium effect size.
There was a significant difference between the K10 mean of clients for the first session (M = 27.33, SD = 10.28, 95% CI [24.16; 30.49]) and the K10 mean of clients for the last session (M = 22.63, SD = 7.60, 95% CI [18.35; 24.13]) in the negative direction, with F(1, 28) = 7.7642, p = .00946. Cohen’s effect size value (d = .66) indicated a medium effect size.
The ORS scores indicated that 61% of the clinic’s clients did not meet the criteria for clinically significant change at the end of their course of psychotherapy.
Discussion
The ORS measures for clients on intake were not consistent with expected scores. We found that 82% of clients entered therapy below or equal to the clinical cut-off value of 25 (International Centre for Clinical Excellence FIT Manuals Development Team, 2011b). Typically, 67%−75% of clients are expected to fall in this range at the start of psychotherapy. This suggests that there are a greater number of distressed clients seeking help at this clinic compared to populations who seek psychological treatment in other settings where the ORS has been validated. Indeed, the clinic provides free psychological services to a vulnerable population that is subjected to high levels of adversity. This finding is, however, not entirely unexpected given the results of the SASH Survey, which showed that the prevalence of common mental disorders is higher in SA than in other parts of the world (Herman et al., 2009). Still, this finding does need to be read with caution since no validation studies have as yet been conducted in SA to verify the clinical cut-off point of 25 for the ORS.
Our analysis revealed that scores on the ORS and K10 did decrease over time in response to psychotherapy, suggesting that these instruments may be sensitive enough to measure treatment outcomes among outpatients in local psychology clinics. Our findings also indicated that there was a statistically significant difference between clients’ first and last session ORS and K10 scores.
Our analysis also revealed several problems with the implementation of the ROM system in the clinic. These included inconsistent, inaccurate, and incorrect use of the ROM measures and evidence that the data generated were not utilised by the clinicians to inform the course of psychotherapy. This finding is not surprising given the literature which shows clinicians’ reluctance to employ outcome measures in routine care. It is also noteworthy that this ROM system was implemented in a training clinic, where the clinicians may have been too inexperienced to manage the demands of collecting and integrating ROMs into the normal course of psychotherapy. Nonetheless, if the routine measurement of psychotherapy outcomes is to become part of standard practice in SA, it will be important for ROM measures to be used in training clinics as these sites have an important influence on how neophyte therapists are socialised into professional practice.
Crucially, our findings about the implementation problems with the ROM system highlight the attention that needs to be given to ensuring adherence and consistent correct use of the ROM measures when such systems are implemented in novel settings. More research is needed to understand the reasons for this poor adherence and inconsistent use of the ROM measures in this clinic.
Some of the ORS data were rendered impractical as a result of clients erroneously responding to the visual analogue items of the measure. This finding may reflect the reality that many of the clients who seek help at the clinic have low levels of education. It is plausible that low literacy levels may therefore have affected clients’ ability to respond to the ORS. However, research has indicated that visual analogue measures are easy to understand and use, particularly by respondents with a lower level of education (Pigott, 2001). Furthermore, there was no evidence to suggest that clients experienced difficulty responding to the K10 items. Compared to Likert scale measures, however, visual analogue measures require significantly more time and commitment for instruction and administration (Pigott, 2001). This finding suggests that in future clinicians should devote more time to facilitating clients’ understanding of the visual analogue items of the ORS.
Although there were fewer implementation problems with the K10, we found that the K10 measure was not administered consistently every fourth session (i.e., once a month) in accordance with the clinic protocol. This may, in part, be a result of clients missing the session on which the K10 was to be administered. The literature suggests that it is difficult to capture a consistent record of clients’ therapeutic progress with measures that are administered over longer time intervals (International Centre for Clinical Excellence FIT Manuals Development Team, 2011b).
Our data suggest that the ORS and K10 measures may have potential to be implemented as outcome monitoring systems during routine clinical care in SA. However, our data also suggest that caution and care will need to be taken when implementing a ROM system to ensure adherence and correct use of the measures. We hope that this account might spur conversations about the use of ROMs within the SA context and contribute knowledge on how the successful implementation of ROMs in the practice of psychotherapy might be facilitated. These conversations are important not only in community clinics but also in the context of psychiatric hospitals and other local clinical contexts where resources are scarce and there is a need to allocate mental health resources optimally. ROM systems may also be helpful in settings where practitioners have questions about the impact of individual psychological interventions which have been modified and adapted to take account of local contextual and cultural factors.
This case study reports on the implementation of two ROM measures in the first year of implementation at a clinic which is a training site. The value of case studies is that they provide insight into specific contexts and particular situations. As such the findings are not generalisable. The sample size is also very small for any really meaningful conclusions to be drawn from the statistical analysis we conducted. Nonetheless, the findings do provide insight into some of the implementation issues that may need attention when ROM measures are introduced in other local clinical settings. It is a further limitation of this study that no qualitative data was collected from the student psychologists, their supervisors, or the clients about their experience of using these ROM measures. Such data may have provided further insight into the usefulness of these measures and the reasons for the poor implementation.
Conclusion
The use of ROM measures in local clinical settings may provide useful data to advance the clinical care of clients and build a credible body of evidence to inform local practice. However, our findings point to the need to conduct a large-scale, systematic, empirical evaluation of the validity and utility of these measures in South African clinical settings. In this context, it is important to note the work that has already been done to adapt and test Clinical Outcomes in Routine Evaluation-Outcome Measure (CORE-OM) in local settings (Campbell & Young, 2011). It may be more expedient to build on the existing work with the CORS measures rather than initiating new work with the ORS. The results of our evaluation and the data we have presented also suggest that it is not a simple matter to implement ROM systems. Should ROM measures be considered for integration into clinical practice, care will need to be taken throughout the implementation process to ensure that (1) clients and clinicians utilise these instruments correctly and (2) negative attitudes towards routine monitoring of psychotherapy outcomes are adequately considered and addressed.
Footnotes
Acknowledgements
Carla Dukas assisted with overseeing data collection and the implementation of the ROM system in the clinic.
Funding
Jason Bantjes received financial support from the South African Medical Research Council (Career Development Award). Mark Tomlinson is supported by the National Research Foundation, South Africa and is a Lead Investigator of the Centre of Excellence in Human Development, University Witwatersrand, South Africa.