Development and Initial Evaluation of the Cyber-Counseling Objective Structured Clinical Examination (COSCE)

Abstract

Objectives: This study developed and validated the Cyber-Counseling Objective Structured Clinical Examination (COSCE), a method and tool used to assess the competence level of trainees and professionals who practice cyber-counseling. Method: The COSCE’s development involved the creation of a cyber-counseling performance rating scale and two simulated client scenarios, and the recruitment and training of three raters. The COSCE was tested on six masters of social work students and six seasoned cyber-counseling practitioners. Results: We examined the COSCE’s internal consistency, interrater reliability, and interclient reliability. In addition, we assessed the construct validity through exploratory factor analysis and known-groups validation method. Conclusions: With further improvement, the COSCE can be a reliable and valid tool in assessing the competence of cyber-counseling practitioners.

Keywords

cyber-counseling competence asynchronous e-mail reliability validity

As cyber and information technology such as the Internet has become an increasingly pervasive presence in our everyday lives, the use of cyber-counseling (or e-therapy, e-counseling, online therapy, online counseling) has expanded (Barak, Klein, & Proudfoot, 2009; Chester & Glass, 2006; Emmelkamp, 2005). Aided by the advancements in technology, practitioners today can overcome the hurdles of scheduling and geographic restrictions, and reach out to clients who do not have access to or are uncomfortable with face-to-face mental health services (Centore & Milacci, 2008; Grubaugh, Cain, Elhai, Patrick, & Frueh, 2008; Mallen, Rochlen, & Day, 2005; Perle, Langsam, & Nierenberg, 2011). At their convenience, clients can contact a practitioner from any distance and receive counseling services in the forms of asynchronous e-mail or board messages, in which a lag exists between responses. Clients can also receive synchronous chat or videoconferencing, in which communication between client and practitioner happens in real-time at a scheduled time period (Barak et al., 2009; Mallen, Rochlen, et al., 2005).

Among different types of cyber-counseling, asynchronous e-mail is the most common form in use (Freeny, 2001; Perle et al., 2011; Rochlen, Zack, & Speyer, 2004). As set appointments are not required, asynchronous cyber-counseling is particularly accessible for people who are unable to commit to an appointment schedule (Mitchell & Murphy, 1998). Asynchronous counseling via e-mail also can facilitate greater expressiveness in clients who experience social discomfort when dealing with therapists face-to-face (Hanley, 2009; Mora, Nevid, & Chaplin, 2008). Moreover, the anonymity and lack of visual contact has a disinhibiting effect, which can lead to greater self-disclosure and self-reflection relative to face-to-face counseling (Freeny, 2001; Rochlen et al., 2004). Clients who engage in asynchronous cyber-counseling go through a contemplative writing process concerning their issues. This text-mediated therapy process can be therapeutic in and of itself (Murphy & Mitchell, 1998), and may enhance a sense of emotional containment as the client can set the parameters of self-disclosure (Suler, 2000). In addition, both clients and practitioners have the ability to link to multimedia resources, which may enrich the therapeutic experience (Grohol, 2000; Mallen, Vogel, & Rochlen, 2005).

Notably, although many of the skills required to perform asynchronous cyber-counseling are similar to those of face-to-face service, practitioners require specialized training to deliver effective, high-quality online counseling services (Mallen, Vogel, et al., 2005). Asynchronous cyber-counseling is unique for several reasons. First, in the absence of nonverbal communication cues such as body language and tone of voice, practitioners need to use various telepresence techniques (Barak et al., 2009; Fink, 1999) to develop and maintain an accurate portrayal of and response to the clinical situation. These techniques include emotional bracketing (i.e., the use of square brackets to express inner thoughts and feelings), descriptive immediacy (i.e., the use of descriptive language that provides the client with information about the practitioner’s context and actual or imagined nonverbal behaviors toward the client), non-lexical verbalization (i.e., textual expressions of non-lexical noises that convey meaningful communications), and time presence (i.e., the use of time terminology that takes clients away from the experience of reading an e-mail and towards the experience of interacting with a practitioner). Second, in asynchronous counseling online practitioners face the challenge of dealing with crisis situations due to the nature of distance counseling. As such, they need to be aware of emergency provisions should they identify any serious safety issue such as suicide, homicide or children at risk of harm (Mitchell & Murphy, 1998). Third, effective cyber-counseling practitioners need to have the capacity to craft words and sentences coherently, to respond to clients with clarity, and to communicate empathy in appropriate depth in online environments. These techniques are learned through repeated practice and supervision (Mallen, Vogel, et al., 2005).

Competence and Cyber-Counseling

Competence has been defined as the capacity of professionals to apply a set of knowledge, skills, and attitudes to the full range of situations in the domains associated with their profession (Bogo et al., 2006; Kane, 1992; Lysaght & Altschuld, 2000). Competence is context dependent, and involves clinical insight, good judgment, and appropriate and responsive actions (Barber, Sharpless, Klostermann, & McCarthy, 2007). As underscored in codes of ethics in professional associations such as the American Psychological Association (2010), the Canadian Association of Social Workers (2005), the National Association of Social Workers (2008), and the American Counseling Association (2005), practitioners are ethically obligated to provide competent care to their clients and are supposed to render specific intervention techniques or approaches new to them only after they undertake relevant education, training, supervision, consultation or study. Given the unique service modality of cyber-counseling, ensuring practitioners have the appropriate knowledge and skill set to deliver counseling online competently has been noted as a major challenge to cyber-counseling practice (Alleman, 2002; Graff & Hecker, 2010; Rochlen et al., 2004). Increasingly, universities and other educational programs are offering certificate programs (e.g., Murphy, MacFadden, & Mitchell, 2008; ReadyMinds, 2010) and are including cyber-counseling within the curriculum through course work and internships (Mishna, Levine, Bogo, & Van Wert, 2012; Mishna, Tufford, Cook, & Bogo, in press; Trepal, Haberstroh, Duffey, & Evans, 2007), and state licensing board and professional organizations have established guidelines for cyber-counseling (e.g., American Counseling Association, 2005; National Board for Certified Counselors and Center for Credentialing and Education, 2005). These training programs and guidelines serve as mechanisms not only to assist trainees and professionals who practice cyber-counseling, but also to increase their competence of care.

Accompanying the development of cyber-counseling education, training, and credentialing is a need for assessment methods and tools that can demonstrate the competence level of trainees and professionals. Professional competence starts in the early stages of training and is a continuing life-long process (Leigh et al., 2007). Given the developmental nature of competence, the assessment of competence not only advances the field and protects the public, but facilitates learning through identifying practitioners’ strengths and areas for further development. At a program level, assessing participants’ competence provides information to guide the development and refinement of curriculum and training programs (Kaslow, 2004). Over the years, standardized competency assessment methods and measures have been developed to address the needs of education and regulatory bodies to ensure accountable, competence-based care (see Andrews & Burruss, 2004; Badger & MacNeil, 2002; Bienenfeld, Klykylo, & Knapp, 2000; Bogo et al., 2011, 2012; Lockyer, 2003; Lu et al., 2011; Newble, 2004; Ross et al., 1988). These generic measures however, do not fully address the specific applications and techniques required in cyber-counseling. To address this need, we developed and pilot tested a new method and tool, Cyber-counseling Objective Structured Clinical Examinations (COSCE). The intended use of this assessment method is to evaluate the level of competence of trainees and practitioners who practice cyber-counseling. In this paper, we describe the development of the tool, present our preliminary findings on its internal consistency, interrater reliability, interclient reliability, and construct validity, discuss the strengths and limitations of the study, and offer some ideas for integrating the COSCE in future research.

Method

The Development of the COSCE

The COSCE was developed by an investigation team that consisted of four social work researchers, three expert cyber-counseling practitioners, one research coordinator and two doctoral students. One of the three expert cyber-counseling practitioners has designed and delivered cyber-counseling courses to graduate and postgraduate students, and the other two have a master degree in social work or psychology, have practiced cyber-counseling for over 12 years, and have delivered workshops on cyber-counseling. We began the process by comprehensively reviewing relevant practice competence methods and measures and members of our team have published a comprehensive review elsewhere (Logie, Bogo, Regehr, & Regehr, in press). Following the literature review, we decided to adapt the Objective Structured Clinical Examination (Bogo et al., 2011, 2012; Hodges, Hanson, McNaughton, & Regehr, 2002; Lu et al., 2011; Ross et al., 1988) for this project. We chose the OSCE for several reasons. First, it has well-established psychometrics, including interrater, inter-case (or inter-station) and split-half reliability, and strong content, construct, and concurrent validity (Adamo, 2003; Bogo et al., 2011; Swick, Hall, & Beresin, 2006). Second, using standardized clients, the OSCE can measure cross-sectional clinical competence while assessing complex competencies without endangering clients’ wellbeing, which increases its feasibility. Third, the OSCE has been used in the mental health professions including psychiatry (Hodges, Regehr, Hanson, & McNaughton, 1998; Walters, Osborn, & Raven, 2005), social work (Bogo et al., 2011, 2012; Lu et al., 2011), and family therapy (Le Roux, Podgorski, Rosenberg, Watson, & McDaniel, 2011), making it potentially adaptable to cyber-counseling. Finally, the OSCE has been found to not only facilitate a formative assessment process that allows both instructors and trainees to gather feedback that guides improvement in an ongoing teaching and learning context (Bogo et al., 2011; Carraccio & Englander, 2000; Park et al., 2004; Stein, Parish, & Arnsten, 2005), but also provides an excellent summative assessment of the level of competence and proficiency necessary at the end of training (Townsend, McIlvenny, Millelr, & Dunn, 2001).

The OSCE typically consists of standardized clinical scenarios with trained actors simulating a client who presents one or more psychological problems (Andrews & Burruss, 2004; Hodges et al., 2002; Tamblyn, Klass, Schnabl, & Kopelow, 1991). The person being evaluated interacts with each of a number of standardized clients in a limited time (typically 10–15 minutes for each client) while their performance is observed and rated by one or two examiners using a rating tool. A final competence score is based on the combined scores from different client scenarios (Newble, 2004).

Summarized in Table 1, the development of the COSCE involved three steps. First, we adapted the practice performance rating scale of the Social Work Objective Structured Clinical Examinations (Bogo et al., 2011), and incorporated additional items relating to key performance indicators of cyber-counseling based on the cyber-counseling literature (Collie, Mitchell, & Murphy, 2000; Murphy et al., 2008; Murphy & Mitchell, 2009), to create a cyber-counseling performance rating scale. The preliminary version of the scale was tested on four cyber-counseling e-mail sessions. Teleconferences were held among investigators to collect qualitative feedback about the scale’s applicability assisting in refining conceptual issues and item descriptions. Based on the comments generated during this testing process, we incorporated additional items related to cyber-counseling, eliminated confusing wording and further strengthened the clarity of the instructions.

Table 1.

Summary of the Development of the COSCE

Step	Process	Output
1.Creation of the cyber-counseling performance rating scale	Derived items from the practice performance rating scale of the Social Work Objective Structured Clinical Examinations (Bogo et al., 2011) Incorporated key performance indicators of cyber-counseling based on the literature (Collie et al., 2000; Murphy et al., 2008; Murphy & Mitchell, 2009) Tested the preliminary version of the scale on four email sessions Held teleconferences to discuss the rating process and collect qualitative feedback	Developed a cyber-counseling performance rating scale that has eight domains with a total of 18 items
2. Creation of standardized client scenarios	Simulated client scenarios created by two expert cyber-counseling practitioners Refined the scenarios to increase their richness and complexity	Developed two fictional scenarios that consist of two different clients with distinct, authentic issues and background
3. Recruitment and training of COSCE raters	Recruited three expert raters Raters followed the original scale instructions and assessed two cyber-counseling sessions Reviewed the ratings from the three raters and conducted teleconferences to collect their qualitative feedback, discuss their concerns and questions, and address any rating inconsistency issues	Recruited and trained three raters Refined the cyber-counseling performance rating scale based on the rater feedback Made the cyber-counseling performance rating scale available online

As shown in Table 2, the cyber-counseling performance rating scale has eight domains with a total of 18 items, reflecting general elements in counseling and specific nonverbal elements of communication in text-based cyber-counseling. Of the 18 items, 13 items were derived from a global rating scale used to evaluate the performance of social work students (Bogo et al., 2011), and five items were designed to reflect the use of telepresence, a unique feature in cyber-counseling (Barak et al., 2009; Fink, 1999). The rating scale is presented in Appendix A. All items are rated by expert raters on a 5-point Likert-type scale where 1 indicates low-level of competence and 5 indicates high-level of competence. The range of the cyber-counseling performance rating score for each session is from 18 to 90. An examinee’s final COSCE score is the average of the total performance rating scores from the client encounters being evaluated.

Table 2.

Domain and Description of the Cyber-Counseling Performance Rating Scale

Domain	Number of items	Description
Introduction	1	How effectively the examinee begins the e-mail session?
Assessment	2	How well the examinee assesses the issues presented or implied by the client?
Intervention	4	How effectively the examinee intervenes to deepen his or her understanding of the client’s situation and to promote positive change?
Relationship building	3	How well the examinee communicates understanding of the client’s thoughts and feelings, explores relationship dynamics, and maintains appropriate boundaries while building a strong relationship?
Enhancing telepresence	5	How effectively the examinee creates a sense of presence and immediacy in the text-based medium?
Cultural, cross-cultural and cross-linguistic issues	1	How well the examinee identifies, explores, and addresses cultural, cross-cultural, and linguistic issues?
Termination	1	How effectively the examinee ends the e-mail session?
Overall assessment of the competencies	1	How skillful and effective the examinee is at integrating all the necessary competencies?

The second step involved the creation of two standardized client scenarios to use in the testing process. The two scenarios are provided in Appendix B. These simulated client scenarios were devised by two expert cyber-counseling practitioners who have developed and taught cyber-counseling certificate postgraduate programs. The complexity of each scenario was based on the expectations of what the professional is likely to face in real world online clinical work. The scenarios were presented to the study team, and minor refinements were made in the research team meetings. In particular, the team felt that diversity issues needed to be enriched in the scenarios, and this consideration was incorporated.

The two simulated scenarios employed in this study were two different clients with distinct issues and background. The scenarios reflected authentic client issues that practitioners might encounter in cyber-counseling. The first client was a single mother with two children. She was troubled by relationship issues with her partner. The second client was a young man of Asian Indian descent who was experiencing intra-cultural conflicts with his girlfriend’s father and feeling not understood by his North American peers.

Finally, we recruited and trained three raters—one from the scale development team (internal) and two expert cyber-counseling practitioners who were independent to the scale development (external)—to rate the e-mail sessions using the cyber-counseling performance rating scale. The combination of internal and external expert raters allowed us to test if the ratings differed between them, and to assess the scale’s applicability in the field. To ensure quality of the rating, we first asked the three raters to follow the scale instructions and assess two cyber-counseling sessions. Then, we analyzed the ratings from the three raters and conducted teleconferences to collect their qualitative feedback, discuss their concerns and questions, and address any rating inconsistency issues. Raters raised concerns related to the wording of the instructions and descriptors of the scale, and both issues led to inconsistent ratings. After the instructions and descriptors were modified, we did not detect any more inconsistency issues. Raters suggested that the modified instructions are self-explanatory and no other orientation is needed to use the rating scale. Presented in Appendix A is the most current form of the cyber-counseling performance rating scale with modified instructions and revised descriptors.

Another concern was related to the available format of the rating scale. Initially the scale was only made available in a word processing document. Although the ratings can be sent electronically, raters found that it was cumbersome to use the checkbox and textbox functions. Moreover, it was easy to miss an item with the word processing format. We thus migrated the form to an online survey site. This change improved the flow of the rating process and increased accuracy.

Implementation and Validation Procedure of COSCE

The study protocol was approved by the Health Sciences Research Ethics Board of the University of Toronto. We tested the validity and reliability of the COSCE on a sample of 12 examinees: six masters of social work student practitioners and six expert cyber-counseling practitioners. To recruit the student practitioners, we sent an invitation letter to 12 students who had participated in a cyber-counseling training workshop and completed a yearlong comprehensive cyber-counseling internship. For the expert practitioners, we reached out to a network of cyber counselors who had at least a master’s degree in psychology, social work, or counseling, had completed cyber-counseling certificate programs, and had practiced cyber-counseling for over 5 years. Both groups of practitioners were informed of the following: (1) they were invited to participate in a cyber-counseling performance scale validation project; (2) their participation was completely voluntary; (3) their identity would be masked from the raters and the investigators with a randomly assigned number ID; and (4) they would receive a $100 gift certificate after they submitted their e-mail sessions. In addition, we emphasized to the prospective student participants that the e-mail sessions they provided would not be used as a source of evaluating their practicum and academic performance. We completed the recruitment when we reached the anticipated number of study participants.

Each examinee was given the two standardized simulated client e-mails. We asked the examinees to follow their typical practice and conduct a 1-hour session as if they were the practitioner for each of these two fictitious clients. The examinees were given 1 week to return their e-mail sessions, and were instructed not to consult with other practitioners. Our three raters independently rated all examinees’ e-mail sessions using the online cyber-counseling competence rating scale. A research staff member who was not involved in any data analysis activity masked the identities of the examinees. To avoid any halo effect, all e-mail sessions were randomly assigned an ID number, and the raters could not link the examinees based on the ID numbers in any way.

Results

We assessed the COSCE’s internal consistency, interrater reliability, interclient reliability, and construct validity. All analyses were conducted using IBM SPSS, Version 20 (IBM SPSS, Chicago). Based on the responses from 12 examinees for two client scenarios, we collected a total of 24 cyber-counseling sessions. The three raters, blind to the examinees who provided the sessions, assessed all 24 sessions using the cyber-counseling performance rating scale. The average time to administer the scale was 18.54 minutes (SD = 16.5).

Internal Consistency

We computed the Cronbach’s alpha to assess the internal consistency of the rating scale. The overall Cronbach’s alpha for the 18 item scale across the 72 assessments (3 raters × 24 sessions) was .97, suggesting that all the items were measuring the cyber-counseling performance construct.

Interrater Reliability

To estimate the interrater reliability of the COSCE, we calculated the Cronbach’s alpha for the 24 COSCE scores generated by each of the three raters. The 3-rater alpha across all examinees was .82, indicating a good interrater reliability when three independent raters are used to assess each performance. To assess the interrater reliability specifically for the student and expert examinees, we also computed the Cronbach’s alphas for the two cohorts separately. The result was weak for the student cohort (3-rater alpha = .68), and good for the expert cohort (3-rater alpha = .85).

Interclient Reliability

To assess the generalizability of the competence scores between client scenarios, we calculated the Cronbach’s alpha coefficient for the total of the two scenarios, where the score of each scenario was averaged across the three raters. The two-scenario Cronbach’s alpha was .91 for all examinees, .87 for student examinees alone, and .84 for expert examinees alone. These results suggest good to excellent interclient reliabilities, and provide some evidence for the generalizability of individual competence scores across different scenarios.

Construct Validity

We tested the construct validity of COSCE in two ways. First, an exploratory factor analysis was undertaken to examine whether the 18-item rating scale comprised coherent dimensions of items through the use of principal component analysis with varimax and oblique rotations. The Kaier-Meyer-Olkin measures of sampling adequacy statistic was .93, and the Bartlett’s test of sphericity was significant (Bartlett’s χ²(153) = 1356.97, p < .0001), indicating that the factor analysis yielded a distinct and consistent result for the data (Kaiser, 1974). The principal component analysis result suggests that all items loaded on one factor, accounting for 68.13% of the variance for the 18 items, with an Eigenvalue of 12.26. Overall, the factor analysis indicated all the items grouped together well to form a single factor concerning cyber-counseling competence.

Second, we assessed the construct validity of the COSCE based on the known-groups validation principle (Allen & Yen, 1979; DeVellis, 2012; Portney & Watkins, 2008) to test if the COSCE correctly differentiated examinees who were more experienced from those who were less experienced. We used t-test to compare the total COSCE scores between the seasoned practitioner examinees and student examinees. The experienced practitioners received a much higher average COSCE score (61.63 [SD = 17.15]) than the student practitioners (40.94 [SD = 13.19]). The difference between the two groups was statistically significant, t(66) = 5.51, p < .0001; Cohen’s d = 1.36, suggesting evidence for the construct validity.

Discussion and Applications to Social Work

A validated cyber-counseling competence assessment instrument can provide educators and researchers with an invaluable tool for examining professionals’ capacities in delivering cyber-counseling. In this paper we delineated steps taken to develop the COSCE, and presented initial evidence of the COSCE’s internal consistency, interrater reliability, inter-client, and construct validity on a sample of practitioners who practice asynchronous cyber-counseling. The study results demonstrate this tool’s potential utility for future replications, while still warranting areas for improvement.

Careful, coordinated planning and preparation set the stage for this study. Our study team comprised scholars with expertise in the OSCE method and statistical analysis and in cyber-counseling, cyber-counseling professionals, and research staff members who supervised and administered the study logistics. This composition proved to be critical in developing the COSCE as it provided the expertise and support needed for such a study and allowed the pilot process to move forward. The team collectively understood the importance of communication and reiteration in the process of developing the measurement tool. Furthermore, the three-step approach generated the three necessary parameters—a rating instrument, client scenarios, and trained raters—for the COSCE and facilitated an ongoing refinement of the tool. Such an approach ensured the COSCE’s quality and integrity.

Competence has been conceptualized as an integration of knowledge, skills, and attitudes that reflects a practitioner’s overall fitness for the profession (Bogo et al., 2006; Epstein & Hundert, 2002). This integration is seen in the enactment of complex practice behaviors. Our empirical data support this holistic view on competence, given that both the internal consistency and the exploratory factor analysis results indicate a unidimensionality of the COSCE performance scale. The data suggest that the set of 18 items collectively represent a set of interdependent processes that demonstrate levels of cyber-counseling competence, rather than distinct competence areas in which individuals could perform well or poorly.

Whether practitioners can perform consistently from client to client is an important practice issue and literature suggests that interclient reliability is perhaps the most critical aspect in clinical competence assessment (Wass, Van der Vleuten, Shatzer, & Jones, 2001). The preliminary results of our study suggest that the COSCE may have an adequate ability to consistently detect the examinee’s performance across cases. That being said, an OSCE typically involves at least four to five simulated clients (Bogo et al., 2011), which is intended to capture the complexity of client situations. In our study, we asked practitioners to respond to only two client scenarios. Although the two simulated clients illustrated distinct issues and were different in their gender orientation, age, and ethnic-racial background, such a limited number of case presentations nevertheless restricts the generalizability of the study results. Future research that attempts to replicate our study should consider incorporating more scenarios that can represent a variety of clients and reflect the complexity of situations encountered in cyber-counseling.

As with other types of OSCEs, the COSCE evaluates the practitioner’s competence through the examination of single sessions with different individual clients. As a result, the current form of the COSCE assesses the practitioner’s competence with clients in isolation and does not render information on the practitioner’s performance over the course of the counseling process. Future studies can consider using the COSCE to assess the practitioners following a single client longitudinally from the initial, middle, and end stages of the counseling process. Through the process, practitioners can demonstrate their case formulation, generate questions and provide responses that facilitate problem solving and strength building, and articulate how they would address the client’s presenting problems through a text-based environment. Such a prolonged engagement will be particularly meaningful in educational settings where the learners can learn from the COSCE results and receive feedback to build their critical thinking, judgment, text-based interpersonal interaction skills, and capacity to demonstrate all these skills online. Furthermore, this design supports holistic and client-centered practice, and can help enhance the COSCE’s validity as a “real-world” assessment as the learner follows a case through to its conclusion (Rushforth, 2007).

Although the COSCE has good overall interrater reliability and an appropriate interrater reliability for seasoned practitioners, the interrater reliability (Cronbach’s alpha = .68) for novices was questionable. This finding suggests that the COSCE scores can vary somewhat across raters when it is applied to the student population and any adaptation of the current form of the COSCE in an educational setting should be considered preliminary. In a study examining the use of an OSCE among student learners, Wilkinson and colleagues (2003) found that the degree of involvement the raters had in the construction of case scenarios was positively associated with interrater reliability and was the most important contributing factor. As two of the three raters in our study did not participate in the process of the case construction, it is possible that we could further improve the interrater reliability by having raters who were actively engaged in the development of simulated cases. Future studies could have the raters who authored the scenarios serve as raters in the evaluation and compare findings with this study.

Synchronous cyber-counseling is becoming more common than it was before (Mallen, Jenkins, Vogel, & Day, 2010). The COSCE is designed to test asynchronous cyber-counseling, and therefore it would be inappropriate to use its current form to assess other types of cyber-counseling, such as synchronous chat and video conferencing. Future research can consider modifying the COSCE for synchronous cyber-counseling and testing its applicability and psychometric quality. Specific modifications may include assessing the practitioners’ efficacy of using computer technology, and evaluating how the practitioner can accurately assess clients, provide a nurturing environment, and establish a working alliance using synchronous media.

On a broader, program level issue, the feasibility of adopting methods to assess competence in the field has been a primary consideration in the competence literature (Leigh et al., 2007). Given the holistic and multifaceted nature of practice competence, assessment of competence often involves a complex process and carrying it out requires additional investment in time and resources (Baartman, Bastiaens, Kirschner, & van der Vleuten, 2006). Although the COSCE does not involve trained actors role-playing a client as other OSCE’s do (see Adamo, 2003; Hodges et al., 2002), and is less labor-intensive and costly, the client scenarios nevertheless need to be continually refreshed and developed. Moreover, the implementation of the COSCE requires expert raters who have extensive experience in cyber-counseling. Given that cyber-counseling is still a relatively new approach, it may be difficult to identify raters with such expertise, which serves as a challenge for successful implementation. That said, we concur with the framework for competence assessment programs presented by Baartman and colleagues (2006), in that time and financial resources need to be allocated to all parts of education, including competence assessment. To ensure the high quality delivery of cyber-counseling competence assessment while retaining its practicality, it may be helpful for professions (e.g., social work, psychology, counseling) that have a common interest in cyber-counseling to collaborate and share resources (Leigh et al., 2007), and develop a pool of qualified raters who can facilitate the implementation of the COSCE.

Asynchronous cyber-counseling has unique features and challenges that are not shared by traditional modes of therapy. Practitioners working in this text-based modality must possess skills and knowledge specific to this method of delivery. As cyber-counseling is likely to continue to grow in the future, it is imperative that educators and accreditation authorities incorporate a reliable, valid and timely competence assessment tool into the education and training process. Such a tool can help guide education and training plans, provide formative information on areas for improvement for the individual being evaluated, and eventually facilitate a summative evaluation (Kaslow et al., 2007). Given that practice competence is complex and has multiple attributes, a range of assessment methods is needed to capture the practitioner performance (Wilkinson, 2007). The COSCE, with further development and replications, can be a reliable and valid tool in assessing practitioners who practice cyber-counseling. Future studies should involve coordinated efforts and careful planning to implement such an evaluation tool. Increasing the number of simulated cases, incorporating a longitudinal design, and strengthening rater involvement in case constructions may be keys to strengthen the clinical relevance and psychometric properties of the COSCE, and allow it to be a meaningful addition that ensures the quality of cyber-counseling practice.

Footnotes

Appendix A

Appendix B

Simulated Client Scenarios

Acknowledgements

The authors would like to express their sincere gratitude to the study participants and expert raters. Much appreciation is also extended to Dan Mitchell, Mona Khoury-Kassabri, Christy Hamill, Dawn Schell, Mary MacDougall, and Susan O'Quinn for their valuable consultations, and to Joanne Daciuk, Hanna Kim, and Jami-Leigh Sawyer for their coordination and assistance to this project.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

This project was funded by Bell Canada.

References

Adamo

(2003). Simulated and standardized patients in OSCEs: Achievements and challenges 1992-2003. Medical Teacher, 25, 262–270. doi:10.1080/0142159031000100300

Alleman

J. R.

(2002). Online counseling: The Internet and mental health treatment. Psychotherapy: Theory, Research, Practice, Training, 39, 199–209. doi:10.1037/0033-3204.39.2.199

Allen

M. J.

Yen

W. M.

(1979). Introduction to measurement theory. Prospect Heights, IL: Waveland Press.

American Counseling Association. (2005). ACA code of ethics: As approved by the ACA Governing Council, 2005. Alexandria, VA: American Counseling Association.

American Psychological Association. (2010). Ethical principles of psychologists and code of conduct: 2010 Amendments. Washington, DC: Author.

Andrews

L. B.

Burruss

J. W.

(2004). Core competencies for psychiatric education: Defining, teaching, and assessing resident competence. Arlington, VA: American Psychiatric Publishing.

Baartman

L. K. J.

Bastiaens

T. J.

Kirschner

P. A.

van der Vleuten

C. P. M.

(2006). The wheel of competency assessment: Presenting quality criteria for competency assessment programs. Studies in Educational Evaluation, 32, 153–170. doi:10.1016/j.stueduc.2006.04.006

Badger

L. W.

MacNeil

(2002). Standardized clients in the classroom: A novel instructional technique for social work educators. Research on Social Work Practice, 12, 364–374. doi:10.1177/1049731502012003002

Barak

Klein

Proudfoot

(2009). Defining internet-supported therapeutic interventions. Annals of Behavioral Medicine, 38, 4–17. doi:10.1007/s12160-009-9130-7

10.

Barber

J. P.

Sharpless

B. A.

Klostermann

McCarthy

K. S.

(2007). Assessing intervention competence and its relation to therapy outcome: A selected review derived from the outcome literature. Professional Psychology: Research and Practice, 38, 493–500. doi:10.1037/0735-7028.38.5.493

11.

Bienenfeld

Klykylo

Knapp

(2000). Process and product: Development of competency-based measures for psychiatry residency. Academic Psychiatry, 24, 68–76. doi:10.1176/appi.ap.24.2.68

12.

Bogo

Regehr

Katz

Logie

Tufford

Litvack

(2012). Evaluating the use of an objective structured clinical examination (OSCE) adapted for social work. Research in Social Work Practice, 22, 428–436. doi: 1049731512437557

13.

Bogo

Regehr

Logie

Katz

Mylopoulos

Regehr

(2011). Adapting objective structured clinical examinations to assess social work students’ performance and reflections. Journal of Social Work Education, 47, 5–18. doi:10.5175/JSWE.2011.200900036

14.

Bogo

Regehr

Woodford

Hughes

Power

Regehr

(2006). Beyond competencies: Field instructors’ descriptions of student performance. Journal of Social Work Education, 42, 579–594. doi:10.5175/jswe.2006.200404145

15.

Canadian Association of Social Workers. (2005). Codes of ethics. Ottawa, Canada: Author.

16.

Carraccio

Englander

(2000). The objective structured clinical examination: A step in the direction of competency-based evaluation. Archives of Pediatrics & Adolescent Medicine, 154, 736–741. doi:10.1001/archpedi.154.7.736

17.

Centore

A. J.

Milacci

(2008). A study of mental health counselors’ use of and perspectives on distance counseling. Journal of Mental Health Counseling, 30, 267–282.

18.

Chester

Glass

C. A.

(2006). Online counselling: A descriptive analysis of therapy services on the Internet. British Journal of Guidance & Counselling, 34, 145–160. doi:10.1080/03069880600583170

19.

Collie

Mitchell

D. L.

Murphy

L. J.

(2000). Skills for on-line counseling: Maximum impact at minimum bandwidth. In Bloom

J. W.

Walz

G. R.

(Eds.), Cybercounseling and cyberlearning: Strategies and resources for the millennium (pp. 219–236). Alexandria, VA: American Counseling Association.

20.

DeVellis

R. F.

(2012). Scale development: Theory and applications (3rd ed.). Thousand Oaks, CA: Sage.

21.

Emmelkamp

P. M. G.

(2005). Technological innovations in clinical assessment and psychotherapy. Psychotherapy and Psychosomatics, 74, 336–343. doi:10.1159/000087780

22.

Epstein

R. M.

Hundert

E. M.

(2002). Defining and assessing professional competence. Journal of the American Medical Association, 287, 226–235. doi:10-1001/pubs.JAMA-ISSN-0098-7484-287-2-jrv10092

23.

Fink

(1999). How to use computers and cyberspace in the clinical practice of psychotherapy. Northvale, NJ: Jason Aronson.

24.

Freeny

(2001). Better than being there. Psychotherapy Networker, 25, 31–39.

25.

Graff

C. A.

Hecker

L. L.

(2010). E-therapy: Developing an ethical online practice. In Ethics and professional issues in couple and family therapy (pp. 243–255). New York, NY: Routledge/Taylor & Francis Group.

26.

Grohol

J. M.

(2000). The insider's guide to mental health resources online. New York, NY: Guilford.

27.

Grubaugh

A. L.

Cain

G. D.

Elhai

J. D.

Patrick

S. L.

Frueh

B. C.

(2008). Attitudes toward medical and mental health care delivered via telehealth applications among rural and urban primary care patients. Journal of Nervous and Mental Disease, 196, 166–170. doi:10.1097/NMD.0b013e318162aa2d

28.

Hanley

(2009). The working alliance in online therapy with young people: Preliminary findings. British Journal of Guidance & Counselling, 37, 257–269. doi:10.1080/03069880902956991

29.

Hodges

Hanson

McNaughton

Regehr

(2002). Creating, monitoring, and improving a psychiatry OSCE. Academic Psychiatry, 26, 134–161. doi:10.1176/appi.ap.26.3.134

30.

Hodges

Regehr

Hanson

McNaughton

(1998). Validation of an objective structured clinical examination in psychiatry. Academic Medicine: Journal of the Association of American Medical Colleges, 73, 910–912. doi:10.1097/00001888-199808000-00019

31.

Kaiser

(1974). An index of factorial simplicity. Psychometrika, 39, 31–36. doi:10.1007/bf02291575

32.

Kane

M. T.

(1992). The assessment of professional competence. Evaluation & the Health Professions, 15, 163–182. doi:10.1177/016327879201500203

33.

Kaslow

N. J.

(2004). Competencies in professional psychology. American Psychologist, 59, 774–781. doi:10.1037/0003-066x.59.8.774

34.

Kaslow

N. J.

Rubin

N. J.

Bebeau

M. J.

Leigh

I. W.

Lichtenberg

J. W.

Nelson

P. D.

Smith

I. L.

(2007). Guiding principles and recommendations for the assessment of competence. Professional Psychology: Research and Practice, 38, 441–451. doi:10.1037/0735-7028.38.5.441

35.

Le Roux

Podgorski

Rosenberg

Watson

W. H.

McDaniel

(2011). Developing an outcome-based assessment for family therapy training: The Rochester Objective Structured Clinical Evaluation (ROSCE). Family Process, 50, 544–560. doi:10.1111/j.1545-5300.2011.01375.x

36.

Leigh

I. W.

Smith

I. L.

Bebeau

M. J.

Lichtenberg

J. W.

Nelson

P. D.

Portnoy

Kaslow

N. J.

(2007). Competency assessment models. Professional Psychology: Research and Practice, 38, 463–473. doi:10.1037/0735-7028.38.5.463

37.

Lockyer

(2003). Multisource feedback in the assessment of physician competencies. Journal of Continuing Education in the Health Professions, 23, 4–12. doi:10.1002/chp.1340230103

38.

Logie

Bogo

Regehr

(in press). A critical appraisal of the use of standardized client simulations in social work education. Journal of Social Work Education.

39.

Y. E.

Ain

Chamorro

Chang

C.-Y.

Feng

J. Y.

Fong

(2011). A new methodology for assessing social work practice: The adaptation of the Objective Structured Clinical Evaluation (SW-OSCE). Social Work Education, 30, 170–185. doi:10.1080/02615479.2011.540385

40.

Lysaght

R. M.

Altschuld

J. W.

(2000). Beyond initial certification: The assessment and maintenance of competency in professions. Evaluation and Program Planning, 23, 95–104. doi:10.1016/s0149-7189(99)00043-9

41.

Mallen

M. J.

Jenkins

I. M.

Vogel

D. L.

Day

S. X.

(2010). Online counselling: An initial examination of the process in a synchronous chat environment. Counselling and Psychotherapy Research, 11, 220–227. doi:10.1080/14733145.2010.486865

42.

Mallen

M. J.

Rochlen

A. B.

Day

S. X.

(2005). Online counselling: Reviewing the literature from a counseling psychology framework. Counselling Psychologist, 33, 819–871. doi:10.1177/0011000005278624

43.

Mallen

M. J.

Vogel

D. L.

Rochlen

A. B.

(2005). The practical aspects of online counselling: Ethics, training, technology, and competency. Counselling Psychologist, 33, 776–818. doi:10.1177/0011000005278625

44.

Mishna

Levine

Bogo

Van Wert

(2012). Cyber counselling: An innovative field education pilot project. Social Work Education. doi:10.1080/02615479.2012.685066

45.

Mishna

Tufford

Cook

Bogo

(in press). A pilot cyber counselling course in a graduate social work program. Journal of Social Work Education.

46.

Mitchell

D. L.

Murphy

L. J.

(1998). Confronting the challenges of therapy online: A pilot project. Paper presented at the Proceedings of the 7th National and Fifth International Conference on Information Technology and Community Health, Victoria, British Columbia, Canada.

47.

Mora

Nevid

Chaplin

(2008). Psychologist treatment recommendations for Internet-based therapeutic interventions. Computers in Human Behavior, 24, 3052–3062. doi:10.1016/j.chb.2008.05.011

48.

Murphy

L. J.

MacFadden

Mitchell

(2008). Cybercounseling online: The development of a university-based training program for e-mail counseling. Journal of Technology in Human Services, 26, 447–469. doi:10.1080/15228830802102081

49.

Murphy

L. J.

Mitchell

(2009). Overcoming the absence of tone and non-verbal elements of communication in text-based cybercounselling. In McDaniel

J. G.

(Ed.), Studies in health technology and informatics (Vol. 143, pp. 215–219). Amsterdam, The Netherlands: IOS Press BV.

50.

Murphy

L. J.

Mitchell

D. L.

(1998). When writing helps to heal: E-mail as therapy. British Journal of Guidance & Counselling, 26, 21–32. doi:10.1080/03069889808253835

51.

National Association of Social Workers. (2008). Code of ethics of the National Association of Social Workers. Washington, DC: NASW Delegate Assembly.

52.

National Board for Certified Counselors and Center for Credentialing and Education. (2005). The practice of internet counseling. Greensboro, NC: Author.

53.

Newble

(2004). Techniques for measuring clinical competence: Objective structured clinical examinations. Medical Education, 38, 199–203. doi:10.1111/j.1365-2923.2004.01755.x

54.

Park

R. S.

Chibnall

J. T.

Blaskiewicz

R. J.

Furman

G. E.

Powell

J. K.

Mohr

C. J.

(2004). Construct validity of an Objective Structured Clinical Examination (OSCE) in psychiatry: Associations with the clinical skills examination and other indicators. Academic Psychiatry, 28, 122–128. doi:10.1176/appi.ap.28.2.122

55.

Perle

J. G.

Langsam

L. C.

Nierenberg

(2011). Controversy clarified: An updated review of clinical psychology and tele-health. Clinical Psychology Review, 31, 1247–1258. doi:10.1016/j.cpr.2011.08.003

56.

Portney

L. G.

Watkins

(2008). Foundations of clinical research: Applications to practice (3rd ed.). Upper Saddle River, NJ: Prentice Hall.

57.

ReadyMinds. (2010). Distance credentialed counselor (DCC^TM) training program and credential. Retrieved March 29, 2012, from http://www.readyminds.com/training/dcc_cert.asp

58.

Rochlen

A. B.

Zack

J. S.

Speyer

(2004). Online therapy: Review of relevant definitions, debates, and current empirical support. Journal of Clinical Psychology, 60, 269–283. doi:10.1002/jclp.10263

59.

Ross

Carroll

Knight

Chamberlain

Fothergill-Bourbonnais

Linton

(1988). Using the OSCE to measure clinical skills performance in nursing. Journal of Advanced Nursing, 13, 45–56. doi:10.1111/j.1365-2648.1988.tb01390.x

60.

Rushforth

H. E.

(2007). Objective structured clinical examination (OSCE): Review of literature and implications for nursing education. Nurse Education Today, 27, 481–490. doi:10.1016/j.nedt.2006.08.009

61.

Stein

M. R.

Parish

S. J.

Arnsten

J. H.

(2005). The OSCE as a formative evaluation tool for substance abuse teaching. Medical Education, 39, 529–530. doi:10.1111/j.1365-2929.2005.02147.x

62.

Suler

J. R.

(2000). Psychotherapy in cyberspace: A 5-dimensional model of online and computer-mediated psychotherapy. CyberPsychology and Behavior, 3, 151–159. doi:10.1089/109493100315996

63.

Swick

Hall

Beresin

(2006). Assessing the ACGME competencies in psychiatry training programs. Academic Psychiatry, 30, 330–351. doi:10.1176/appi.ap.30.4.330

64.

Tamblyn

R. M.

Klass

D. J.

Schnabl

G. K.

Kopelow

M. L.

(1991). The accuracy of standardized patient presentation. Medical Education, 25, 100–109. doi:10.1111/j.1365-2923.1991.tb00035.x

65.

Townsend

A. H.

McIlvenny

Millelr

C. J.

Dunn

E. V.

(2001). The use of an objective structured clinical examination (OSCE) for formative and summative assessment in a general practice clinical attachment and its relationship to final medical school examination performance. Medical Education, 35, 841–846. doi:10.1046/j.1365-2923.2001.00957.x

66.

Trepal

Haberstroh

Duffey

Evans

(2007). Considerations and strategies for teaching online counseling skills: Establishing relationships in cyberspace. Counselor Education & Supervision, 46, 266–279. doi:10.1002/j.1556-6978.2007.tb00031.x

67.

Walters

Osborn

Raven

(2005). The development, validity and reliability of a multimodality objective structured clinical examination in psychiatry. Medical Education, 39, 292–298. doi:10.1111/j.1365-2929.2005.02091.x

68.

Wass

Van der Vleuten

Shatzer

Jones

(2001). Assessment of clinical competence. Lancet, 357, 945–949. doi:10.1016/s0140-6736(00)04221-5

69.

Wilkinson

T. J.

(2007). Assessment of clinical performance: Gathering evidence. Internal Medicine Journal, 37, 631–636. doi:10.1111/j.1445-5994.2007.01483.x

70.

Wilkinson

T. J.

Frampton

C. M.

Thompson-Fawcett

Egan

(2003). Objectivity in objective structured clinical examinations: Checklists are no substitute for examiner commitment. Academic Medicine, 78, 219–223.