Assessing Clinical Mental Health Counseling Skills and Practice Standards in Distance Education

Learning Outcomes Assessment in Distance Education

The emergence of online formats within higher education has prompted a body of emerging research to explore this learning modality. Means, Toyama, Murphy, Bakia, and Jones (2010) in a meta-analysis of 45 studies examined the efficacy of online instruction for SLOs compared to traditional face-to-face instruction and hybrid instruction. They reported that 11 studies showed stronger benefits for online or blended learning and three studies showed stronger benefits for traditional face-to-face learning. They further discovered that online learners performed “modestly better” than learners receiving traditional face-to-face instruction. The results further revealed than blended or hybrid instruction which combined face-to-face teaching with online instruction produced a larger effect size (+.35) than purely online or face-to-face teaching, which had an average effect size of +.05. Additional results demonstrated that effect sizes were larger for face-to-face, online, and hybrid programs that used collaborative, peer-led (+.25), or instructor-led (+.39) models of teaching and learning, than those programs that were purely self-directed (+.05).

Although these findings overall indicate benefits of hybrid programs, it is important to note that variation exists for classroom environments and instructor characteristics and behaviors. Quality indicators, thus, may be less dependent on whether a program is face-to-face, distance, or hybrid based, and more dependent on variables such as the classroom structure and learning platform, level of commitment to student learning that is made by the counselor educator, and program emphasis on achievement measurement of SLOs.

Online education programs are developing specific measures to assess learning outcomes for their unique environments. Menchaca and Hoffman (2009) constructed a process for conducting program evaluations in online classrooms through cross-university collaboration with faculty, administrators, and college alumni. The process was intended to be used at the early stages of program design, implementation, and evaluation. They identified that programs often plan distance components with little thought to unique considerations for teaching and assessment, assuming instead that the same practices that are applied for face-to-face classrooms will work similarly at a distance. They emphasized that the appropriateness of distance education for meeting the program and institutional objectives as well as for identifying assessment methods is an important consideration for the evaluation process. Applied to counselor education, consideration needs to be given at a program’s inception to determine both how and for what purpose distance education is being offered as well as how the program will be evaluated. For programs currently accredited or seeking accreditation under the CACREP (2009) Standards, consideration must be given to the assessment of knowledge, skills, and practice in the distance environment.

Accreditation standards that are typically written for traditional institutions present unique challenges for distance education programs. Beebe, Vonderwell, and Boboc (2010) explored in a phenomenological study the process of transitioning assessment practices from a traditional to distance institution. These authors identified five themes from seven higher education administrators as areas of focus in supporting the development of online learning: (a) faculty and student ability to manage time differently; (b) student efficacy in asking questions and seeking solutions; (c) structure of the online platform in facilitating dialogue and knowledge interchange; (d) assessment of course content that is more advanced, philosophical or abstract; and (e) the need to informally assess students within the structure of an online forum. The authors concluded that the greatest challenges in distance education assessment were related to identifying qualitative elements of the student’s progress that were not as easily identified within the structure of online classrooms. Applied to counselor education, assessing skills- and practice-based components of counselor training may be sometimes more difficult to do, particularly in asynchronous learning environments in which the faculty and students attend an online classroom at different times.

Clinical Skills Assessment in Distance Education

A particularly important qualitative assessment is in the ability to examine and respond to student skill development. Regan and Youn (2008) identified three themes in a literature review of social work distance education: (a) the historical challenge of skill development when learning is occurring in a distance environment, away from peer and instructor feedback; (b)resistance to the use of web-based teaching models among social workers; and (c) a general lack of evidence that supports the best models for facilitating the development of clinical skills in distance environments. The authors concluded that, whereas the web offers learning opportunities for students who would otherwise be geographically or otherwise constrained from access to education in a face-to-face education environment, learning should not be entirely asynchronous. With web-based clinical skill development, there is a need for a blended, or hybrid model that encompasses both asynchronous and synchronous web-based training, as well as face-to-face teaching, to assure that students’ skill development has taken place.

Although research and practice-based literature on counseling skills assessment has primarily focused on observations in traditional classroom settings, there is some literature available on assessing skill development in distance education. Shibusawa, VanEsselstyn, and Oppenheim (2006) presented findings about a program called Third Space through which they provided skill development for second-year Master of Social Work students in a couples therapy class. Triadic role-plays (one student playing the therapist, the other two playing a couple) were developed in-person and recorded, then discussed in the classroom via an interactive bulletin board that allowed students to paste excerpts from the video and create dialogue around the video portion that was being observed. Shibusawa and colleagues found that maintaining the video in the Third Space allowed students to review their own and their peers’ videos as often as needed, and that the additional benefit of having commentary included with specific video segments was particularly helpful. The authors identified challenges in managing the feedback process, however. In traditional face-to-face clinical skill development, the instructor can intervene if peer feedback is becoming too negatively focused, but the use of an electronic bulletin board in an asynchronous learning environment compromised the instructor’s control over timely intervention. The authors concluded that more research is needed to develop pedagogical and supervisory knowledge in distinctions between web-based and in-person skill development. Their concern was that the distance nature of the feedback process, in particular, might allow students to be more critical and less empathic of each other than they would be in a face-to-face classroom.

Second, Rockinson-Szapkiw and Walker (2009) examined Web 2.0 technology used in a counseling skills course and demonstrated the use of asynchronous web-based modalities (e.g., instructional videos of lectures, instructor skill demonstrations, and student blogs that allow commentary about learning material), and those that occur in real time (e.g., virtual counseling involving avatars that simulate an actual client and counselor with accompanying voice technology). Rockinson-Szapkiw and Walker used this modality to complete a number of counseling scenarios. Whereas the authors were optimistic about the possibilities for developing student counseling skills, they cautioned that students might experience difficulty in mastering the technology beforehand, thus creating a potential barrier to successful skills demonstrations.

Counselor educators, then, are charged with developing skills-based training across the primary domains of a distance education program to ensure that students have the necessary skills and competencies to carry out professional responsibilities. SLOs that are linked to skills-based competencies in the CACREP (2009) Standards must be measurable and rely on a process for student remediation when skills do not meet program expectations. This places responsibility on counselor training programs to assure the accuracy and fairness in stated policies surrounding skills expectations and the steps that are taken to address these deficits.

Remediation in SLO Assessment

McAdams and Foster (2007) noted the importance of ensuring that students’ due process rights were preserved in any remediation and/or program dismissal actions. Programs are challenged to ensure that assessment of student professional behavior is fair and that dismissal from a training program would be justified, particularly as there is little guidance in the professional literature about how to apply professional standards. As these authors indicated, “For evidence of fairness in student remediation, the courts have looked beyond an educational institution’s published remedial policy and have examined the accessibility, adaptability, and consistency of that policy as it has been applied in specific students’ situations” (p. 10).

Foster and McAdams (2009) noted the importance of publishing evaluation procedures to students in the form of new student orientations to meet the burden of informed consent and help develop a climate of accountability to both program faculty and students in professional development. Transparency in the student assessment process is manifest when “students’ perceptions of their academic programs’ values, intentions and expectations are congruent with those of the program faculty and administration members” (p. 275). Failure on the part of program faculty and administration to intentionally perform their roles as gatekeepers to the profession may result in diminishing the importance of personal, ethical, emotional, and psychological factors that influence professional performance and protection of clients. Developing a culture of transparency related to gatekeeping involves striking a balance between “top-down” (faculty to student) and “bottom-up” (student to faculty) communication. This can best be facilitated through intentional discussion of assessment procedures in a new student orientation, classroom objectives outlined in course syllabi, regular student advisement, and faculty supervision in practicum and internship. These opportunities can allow for dialogue between faculty and students, develop trust, and promote mutual understanding regarding the assessment of skill development.

The refinement of SLO assessment under the CACREP (2009) Standards follows a general trend toward greater accountability in higher education programmatic assessment. This development has occurred alongside the refinement of distance education in general, and the means by which to assess clinical skills outside of face-to-face classrooms (Day & Bobeva, 2005; Means, Toyama, Murphy, Bakia, & Jones, 2010; Menchaca & Hoffman, 2009; Rockinson-Szapkiw and Walker, 2009). CACREP-accredited distance education programs must identify methods for SLO skill assessments that assure accountability to the gatekeeping function, but that are workable within the program’s unique structure. This may include carrying out some or all of the skills development, assessment, and, if needed, remediation at a distance. In the next section, we will demonstrate the SLO assessment and remediation process used by one university’s CACREP accredited hybrid-based clinical mental health counseling program.

Program Case Presentation

To illustrate our discussion of addressing skills and practice for SLOs, we will use the case of a hybrid-based M.S. in Mental Health Counseling program that was accredited under the CACREP (2001) Standards. This hybrid model consists of online coursework, face-to-face residencies for the skills development and assessment, and a fieldwork sequence including a 100-hr practicum and 600-hr internship. In response to the specific skills and practice requirements under the CACREP (2009) Standards for clinical mental health counseling, the program has developed processes for program assessment and has improved instrumentation to more precisely address skills deficits. Importantly, we will focus this section on strategies used for skill-building in accordance with the SLOs for Clinical mental health counseling, as well as remediation efforts used with students who had difficulties in demonstrating the required skills. The student cases of “John” and “Mary” (aliases) will demonstrate skills remediation processes that were used to address each student’s learning gap.

SLOs were developed directly from the language of CACREP (2009) Standards. For each standard, there is a corresponding SLO that is measured in both knowledge, and skills and practice. For example, as a skills and practice SLO that addresses the CACREP 2009 assessment standard for clinical mental health counseling, we use “Students will be able to employ various assessment techniques to appropriately intervene in meeting the needs of diverse clientele.” The program relies on four sets of tools in assessing SLOs. These include the following:

As an annual process, we collect data for the LORbooks from artifacts identified by the program as indicators for the knowledge dimension on each SLO. As these data come from online courses that are taught in a Blackboard course platform, the program is able to pull a year’s worth of grades for an assignment (each section of a course is identical to assure consistency in quality and adherence to SLOs). The student average for a particular assignment must be 80% or higher for a benchmark to be met. When we are below 80%, we review the assignment and address the matter with program faculty to determine the nature of the low grades.

The surveys and field experience site supervisor evaluations, distributed electronically using Qualtrics, use 5-point Likert-type scales to determine performance on each SLO. With the 80% benchmark, average scores of less than 4 are reviewed, as with the LORbook, to determine the gap. The student, graduate, and employer surveys are administered annually. The site supervisor evaluations are specific to each student and completed quarterly, allowing us to address students’ specific knowledge and skill deficits. Because these evaluations occur quarterly, we are able to track individual student performance over the life of the field experience. Students who average below 80% overall in their evaluations by the end of their first internship are placed on skills remediation plans that are coordinated in cooperation between their field experience instructor and site supervisor.

Of the four sets of SLO assessment tools used by the program, the Residency Skills Assessments and Techniques in counseling rubric is the most uniquely structured to fit the university’s hybrid model. Micro skills are taught and assessed sequentially beginning with Residency 1 or Techniques (students attend Residency 1 in their first three quarters at the time of their choosing, and techniques is taken in the second quarter). The final counseling skills assessment occurs at Residency 2, taken prior to field experience. Skills that are taught and assessed in Residency 1 and techniques correspond with coursework that the student has completed at that point, and are expected to be later demonstrated at Residency 2, but with greater sophistication.

As an example, for assessment of the skills component of the Multicultural Counseling SLO on the Residency Skills Assessment, “Student acknowledged, and addressed, in a professional manner, issues related to diversity (i.e., race, sexual orientation, gender, religion, ethnicity, nationality, or other affiliations) relevant to her or his work with the client(s).” For measurement of the Professional Ethics in Counseling SLO skill component of the Residency Skills Assessment, we assess micro skills that include, “Student presents in a professional manner during interactions; demonstrates self-awareness and awareness of impact on others; demonstrates appropriate self-disclosure.” Reviewing student skills in these areas at three points in the program prior to field experience allows a clear sense of how individual students are progressing, and provides the program with specific information on how well SLOs are being achieved.

Given that both the professional ethics in counseling and multicultural counseling are courses taught early in the program, we have had relative ease in identifying three benchmarks at which we are able to assess these skills. The program’s effort is still growing and has yet to define how to assess skills, prior to the practicum and internship experience, in the areas of diagnosis, assessment, and research. The distance-based courses that correspond with these areas occur later in the program, and do not currently include counseling demonstrations that allow assessment of the skills aspects of SLOs. Whereas students do submit written assignments that include differential diagnosis and global assessment of functioning scoring, reviewing and describing use of instruments such as the Minnesota Multiphasic Personality Inventory-2 and a mental status examination, and literature reviews that address clinical work with the student’s population of interest, there are currently no skills demonstrations that indicate student learning has taken place in a manner that can be applied to a direct clinical experience. Outside of the techniques in counseling course, which uses video feeds of student mock taping sessions with consenting individuals, we do not currently assess counseling skills in the online courses prior to residency.

As a program, we are continuing to refine our courses to include more skills components. These additions rely on new technology that, at this point, still creates compatibility challenges for some student end users. The technology for uploading video feeds directly into the classroom is still relatively new, and we have only had this technical capability since 2011. Because adding additional residencies would be financially unfeasible for many students and logistically challenging to manage for faculty, we are currently working to build our online platform for assessing a greater breadth of counseling skills that directly address the SLOs.

Strengths and Limitations of Our Model for Assessing SLOs

Our processes for measuring SLOs and using the data for program decisions are evolving. These efforts are supported by a university-wide emphasis on assessment that has seen the use of systems enhancements for data collection. In addition, we have designated a counseling faculty member to manage program assessment. As we move through the process and adapt the CACREP (2009) Standards to our hybrid environment, we are identifying both strengths and limitations to our current model. The LORbook process for capturing knowledge components of SLOs throughout the curriculum is well designed for the distance environment. Course assignment grades are easily pulled from the Blackboard course shell, allowing ease and efficiency in aggregating results. Site supervisor evaluations also provide a useful data source, and we have found that conducting site “visits” with supervisors via teleconference at the beginning and middle of the quarter are useful for clarifying their role in the SLO assessment process. Survey data of students, graduates, and employers have some value, although low return rates sometimes hinder practical use of the data in SLO assessment.

It is not uncommon for counseling programs to reserve the assessment of clinical skills development learning outcomes for practicum and internship experiences; however, programs are cautioned to be careful that they do not save all evaluation of clinical skill development for fieldwork (Urofsky, 2009). Skills that are based in knowledge gained from earlier courses (e.g., professional ethics in counseling, multicultural counseling) are emphasized and developed over the life of the program through the residency model; however, as stated in the previous section, face-to-face counseling skills that apply to later coursework, such as diagnostic impressions about a “live” client, are not assessed until the field experience.

The residency model provides a level of intense skill development that occurs over two distinct periods. Whereas several hours of skill development with each residency student generally provide a clear demonstration of that student’s skill attainment, we sometimes find that students with skills deficits have difficulty improving over the course of 1 week. In addition, some students become especially anxious in the experience of intensive skills taping, and do not perform to their full potentials. The aggregate data from the Residency Skills Assessment that is used for evaluating SLOs have been impacted at times when we have had sizable numbers of students who were not able to demonstrate skills under the residency conditions. Generally, this has been the case when faculty were supporting larger groups of students (up to 15). We have found that students tend to perform better in smaller groups, and identified that assigning one faculty to 10 or fewer residency students presents an ideal condition for skill development. Under these circumstances, students generally become less anxious during their fishbowl exercises, and the opportunity for more intensive feedback and development allows greater opportunity for final skills results that reflect each student’s actual ability.

Our model is limited in that, because we use the same syllabus and structure to teach all sections of an online course, it is both expensive and time-consuming to make course modifications. This is a challenge in that even as we receive SLO data that inform us that we have a deficit in student learning or in our measurement for a particular area, we do not always have the fiscal ability to change in as timely a manner as we would wish. Courses are updated on a multiyear cycle. Under the CACREP (2009) Standards, this presents an additional challenge in that courses that do not currently contain a skills component, such as our diagnosis class, will not be updated until the budget is allocated for a change.

With the technology to provide educational access to a greater student population, counseling programs are opting to include online and hybrid program offerings. For counselor educators who are establishing hybrid-based programs and considering how to assess skills across the distance and face-to-face components of the program, we offer these recommendations:

Skill components can present the most uniquely challenging component of a comprehensive assessment plan in a hybrid program. Whereas most of our students successfully demonstrate the required micro skills that align with SLOs at each benchmark (Techniques in Counseling, Residency 1 and 2), some require additional remediation in order for the program to assure that student learning has been achieved. Given that residencies are only 1 week and held at conference hotels, there is a logistical time constraint in being able to demonstrate necessary skills. The following cases of John and Mary demonstrate how the program addressed skills remediation at a distance for students who did not pass the residency skills requirements.

Case of John

John presented to Residency I during his third quarter in the Mental Health Counseling program. On the first day in the residency small skills group John’s instructor, “Dr. Jones,” noticed that he did not engage with other students at all, and minimally participated in activities with great difficulty and only in response to direct faculty requests. Dr. Jones provided information to the skills coordinator (a faculty member designated for tracking individual student skill performance across the program’s residency benchmarks) about her concerns with John’s presentation during Day 1. It was agreed that Dr. Jones should approach John with the concerns and monitor his behavior for one additional day with an agreement to provide an update to the skills coordinator after the Day 2 skills group (in which students practice their skills as a fishbowl exercise). After Day 2 instruction, Dr. Jones continued to identify concern about John’s inability to express empathy or reflect feelings, micro skills used to measure the Helping Relationship SLO on the Residency Skills Assessment. In an effort to intervene early, a plan was devised for Dr. Jones, the Skills Coordinator, and another program faculty member to meet with John to address the concerns.

In the meeting, John clearly stated that the residency was very different than he expected it to be and that he was very uncomfortable with the idea of having such a strong focus on client’s personal feelings. During the discussion, John professed that he was unsure that counseling was a good fit for him. He had also decided to leave the residency in order to ponder whether or not he wanted to continue in the program.

John returned to repeat the first residency 6 months later. He worked with a different Residency I faculty member. Again, John evidenced great struggle in delivering the required basic counseling skills for the Helping Relationships SLO, despite an obvious effort to perform. Despite two attempts at delivering the required videotaped final product skills demonstration, John performed poorly and was referred to the skills coordinator for remediation. He was permitted to make a third attempt at producing a skills video. While John’s scores on the videos were only slightly below the minimum requirement for passing (with a passing score of 80% corresponding with the B required for a student to successfully demonstrate the SLO), John did not demonstrate sufficient skills to pass the third video. At this point, a postresidency skills remediation plan was implemented for John.

Case of Mary

Mary presented for Residency II 11 months after completing her first residency. On the third day of the residency, “Dr. Smith” noticed that Mary has avoided opportunities to volunteer in the group and contributed little to course discussions. Dr. Smith decided to call on Mary to share aloud the case notes all Residency II students are to compose while observing the role-plays and participating in reflecting teams. Mary’s case notes reflected her apparent difficulty with case conceptualization and professional note taking, which were addressed in the counseling skills, prevention, and intervention SLO skills component of the Residency Skills Assessment.

Dr. Smith, in an attempt to help Mary, suggested some readings that provide information about record keeping. Dr. Smith also proposed that Mary revisit the information on notetaking from the techniques of counseling course. Dr. Smith expressed concerns to the skills coordinator about Mary’s difficulty with case notes and identified that she had a small skill deficit, but did not yet have sufficient support for remediation. Dr. Smith also noted that Mary was scheduled to do a live skills demonstration the following day and would report more about Mary’s progress the next day.

After two attempts at demonstrating skills during her fishbowl exercise, the first of which illustrated Mary’s difficulty with the advanced skills of appropriate self-disclosure and silence, Mary “froze” and stated that she did not know what to do with someone who had this kind of problem and that she had no intention of working with “that” population (the clinical scenario had the student in the client role-play a lesbian woman who is coming out to her family). The student did not pass the Ethics or Multicultural SLO skills components, as a result of her refusal to demonstrate the required skills.

Dr. Smith referred Mary to the skills coordinator, who assigned Mary to work with another faculty member, individually. This one-on-one environment provided Mary with opportunities outside of her small cohort group to practice identified skills. Mary evidenced skills deficits similar to those reported by Dr. Smith and did not show any improvement in her skills delivery on the SLOs for Ethics or Multiculturalism. After discussion between the two faculty who had worked with Mary and the skills coordinator, a meeting was scheduled at the end of residency.

Ethical, Legal, and Multicultural Considerations

Hybrid-based programs that use course technology for students to upload skills demonstrations must be clear on the degree of risk involved with this activity. Whereas, in face-to-face training, students demonstrate counseling skills with each other while supervised by a faculty member, in the distance environment, students record sessions with a person who signs a consent form to act as a “client” for the training scenario. Even with a well-defined consent process and the establishment that these are mock exercises for the purpose of student practice, this potentially establishes a situation in which a student is in a position of knowledge/authority, and the friend who agreed to participate in the video as the “client” may come to expect the student to engage as a counselor more routinely. Whereas our syllabus stresses guidelines for students in managing these training experiences, we recognize that there is a level of risk involved in students completing this work with the help of friends from outside the program. To help assure the privacy of all video material received by students, we recommend using online classrooms for submittals of all video transmissions. Course platforms, such as Blackboard and eCollege, are designed to support the safe sharing of information and data between faculty and students, and are generally more secure than e-mail.

Similar to face-to-face programs, we have addressed individual students who, as in the case of Mary, cite personal reasons for refusal to work with lesbian, gay, bisexual, and transgender (LGBT) clients. As the students are distance-based and not necessarily living in communities that are safe for visible expressions of gender and sexual diversity, the residency experience often brings them into encounter moments very rapidly with LGBT faculty, students, and staff. Cultural competence-based skills scenarios that require the student to demonstrate abilities in working with LGBT clients present unique challenges for a small number of our students. Some students, like Mary, actively refuse to serve the LGBT community, and may cite religious values in their decision making. Whereas we have to this point been successful in helping students identify better academic fits than our counseling programs, the issue is one of concern. As a distance-based program, our students are located throughout the United States and internationally. Different states appear to be making very different determinations of grounds for dismissal from counselor training programs in regard to multicultural issues. Additionally, some of our students come from countries that criminalize gender and sexual diversity. Our task has been to hold students accountable to the SLOs and maintain consistent expectation on the delivery of these by each student while maintaining awareness of the cultural application of these SLOs for the communities these students will serve.