Evaluating Students’ Use of Therapeutic Communication in Entry-Level Education: The Observer Version of the Clinical Assessment of Modes (CAM

Abstract

Importance: The Intentional Relationship Model (IRM) guides learning about therapeutic use of self. The observer version of the Clinical Assessment of Modes (CAM–Observer) may be used to evaluate students’ therapeutic communication as the process is defined in the IRM.

Objective: To assess the structural validity of the CAM–Observer.

Design: Cross-sectional, psychometric study.

Setting: Master’s in occupational therapy program.

Participants: One hundred thirty-four entry-level students.

Outcomes and Measures: The overall CAM–Observer and the individual subscales (Advocating, Collaborating, Empathizing, Encouraging, Instructing, Problem-Solving) were used to assess students’ communication from the instructor’s perspective.

Results: The overall CAM–Observer and six subscales demonstrated appropriate rating scale functioning and dimensionality. The Advocating subscale demonstrated poor item fit, floor effects, and low person separation. One Collaborating item demonstrated poor fit to the overall CAM–Observer and the Collaborating subscale, requiring revision. Instructing and Encouraging items were most likely to be endorsed by the instructor–observers, resulting in a ceiling effect for the Instructing and Encouraging subscales. The Advocating and Problem-Solving items were least likely to be endorsed by the instructor–observers. Except for the Problem-Solving subscale, the overall CAM–Observer and the individual subscales could reliably separate items according to difficulty. Except for the Advocating subscale, the overall CAM–Observer and the individual subscales could reliably separate people into high- and low-performance groups.

Conclusions and Relevance: The CAM–Observer offers a structurally valid and theoretically grounded assessment of students’ therapeutic communication as the process is defined in the IRM. The CAM–Observer can be integrated into occupational therapy education to guide students’ critical reflection on their interpersonal communication.

What This Article Adds: The CAM–Observer offers a means of evaluating students’ therapeutic use of self.

Therapeutic use of self is a critical component of occupational therapy practice (American Occupational Therapy Association, 2014), and its importance is made explicit in the Accreditation Council for Occupational Therapy Education (ACOTE^®) Standards (ACOTE, 2012, 2018). Although therapeutic use of self is a fundamental feature of entry-level occupational therapy education, research on evidence-informed strategies for teaching therapeutic use of self remains limited. Effective instructional methods for promoting students’ competency regarding therapeutic use of self may be viewed as “elusive” and “difficult to pin down” (Davidson, 2011, p. 94). In a 2011 review of entry-level occupational therapy programs, Davidson found that instructional time related to therapeutic use of self often consisted of discussions, lectures, and modeling by the instructor. The majority of the programs did not use a textbook to guide student learning, and self- and peer-assessment were rarely incorporated into teaching (Davidson, 2011).

Therapeutic use of self encompasses a dynamic and reciprocal interaction between the occupational therapy practitioner and the client, which requires a high level of interpersonal self-awareness, self-reflection, and flexibility on the part of the practitioner. The Intentional Relationship Model (IRM; Taylor, 2020) was developed to address the complexity and nuances of therapeutic use of self in occupational therapy. The IRM offers a theoretical framework for examining therapeutic relationships in occupational therapy, and it has the potential to bridge the gap between professional value of therapeutic use of self and its implementation in practice (Solman & Clouston, 2016). Although research on the IRM is in its early stages, integration of the IRM into education has been shown to be effective in enhancing students’ self-efficacy in the therapeutic use of self (Schwank et al., 2018).

The IRM offers a systematic approach to teaching and evaluating the interpersonal skill base necessary for entry-level occupational therapy practice. According to the IRM, one of the core components of therapeutic use of self is intentional and flexible use of the six therapeutic communication modes: Advocating, Collaborating, Empathizing, Encouraging, Instructing, and Problem-Solving (Taylor, 2020). A practitioner who can demonstrate flexibility and ability to switch between the six modes comfortably manifests a multimodal interpersonal style, which, in turn, supports his or her ability to communicate more effectively with a broader range of clients (Taylor, 2020).

To support practitioners’ ability to remain reflective and critically self-aware in regard to their communication, we developed the Clinical Assessment of Modes (CAM; Taylor & Popova, 2019) for evaluating their use of the six IRM modes (Fan & Taylor, 2016). The CAM demonstrated appropriate construct validity for assessing practitioners’ communication with clients in inpatient and outpatient rehabilitation (Fan & Taylor, 2016). However, to our knowledge, the validity of this assessment in occupational therapy education has not been examined to date. To expand the existing methods for assessing students’ therapeutic use of self in an academic setting, we evaluated the structural validity of the observer version of the CAM (CAM–Observer) from the perspective of the instructor–observers.

Method

This psychometric study was conducted with a cross-sectional research design and was approved by the institutional review board at the University of Illinois at Chicago. Three cohorts of students were invited and consented to participate in the study during their first semester of a 2-yr master’s-level program in occupational therapy. Each participant completed a written informed consent after enrollment. Data collection took place as part of a standard educational curriculum, and demographic data were not collected. The data collection took place during a 16-wk course titled “Developing a Therapeutic Self.” The IRM textbook, The Intentional Relationship: Occupational Therapy and Use of Self (Taylor, 2020), guided the course curriculum.

The course consisted of weekly lectures (46 students per cohort) and lab (15–16 students in each of three sections). Each lab group met separately and was facilitated by three different instructors (licensed occupational therapy practitioners) who agreed to serve as data collectors in this study. Each student was required to design and lead a therapy group as a final assignment. The students were free to choose from among their peers to serve as client–actors, with at least 3 and no more than 15 per group. Lab instructors assigned 1 client–actor per group to role-play challenging interpersonal characteristics. The lab instructor evaluated leaders’ mode use using the CAM–Observer (Taylor & Popova, 2019).

Measures

The CAM–Observer (Taylor & Popova, 2019) was used in this study. The CAM–Observer consists of 30 items that are distributed across six subscales: Advocating, Collaborating, Empathizing, Encouraging, Instructing, and Problem-Solving. Each item is rated on a 4-point rating scale (0 = never, 1 = rarely, 2 = occasionally, 3 = frequently). Overall and subscale scores may be obtained from the measure. An average score across the 30 items may be used as a measure of overall mode use. An average score across the five items in each of the six subscales can be used as a measure of individual mode use.

Analysis

We evaluated structural validity of the CAM–Observer using the Rasch–Andrich Rating Scale Model analytic approach. We analyzed the data using the Winsteps® Rasch measurement computer program (Version 3.93.0; Winsteps.com, Beaverton, OR). The assessment was evaluated by examining rating scale functioning, dimensionality, item and person fit, item targeting, and item and person separation.

As evidence of appropriate rating scale functioning, it was expected that each individual rating scale response category would have ≥10 responses, outfit mean square (MnSq) would be <2.00, and the observed averages would advance monotonically from least to most frequent (Linacre, 2002). It was expected that the items would work together to measure a single construct, accounting for ≥50% of the raw variance (Smith & Miao, 1994). Item- and person-fit statistics were expected to range between 0.50 < Infit MnSq < 1.50 and −2.00 < Infit Zstandard (Zstd) < 2.00 (Linacre, 2019). It was expected that <5% of items and <10% of people would demonstrate inappropriate fit to the Rasch model.

Item targeting was evaluated in two subsequent steps: (1) examination of item difficulty and (2) evaluation of ceiling and floor effects. Relative item difficulty was evaluated by comparing item measure scores and standard errors for each item. Item difficulty was further examined against person ability through visual examination of the person–item hierarchy map. Ceiling and floor effects were evaluated with an expectation that <15% of participants would receive an absolute maximum or minimum score (McHorney & Tarlov, 1995). Item and person separation coefficients and reliabilities were examined for assessment sensitivity to different levels of item difficulty and person ability. A separation coefficient ≥2.0 and reliability ≥0.80 were used as indicators that the measure could separate items and people into at least three distinct groups (Arnadóttir & Fisher, 2008). A separation coefficient ≥1.5 and reliability ≥0.70 were used as indicators that the measure could separate items and people into at least two distinct groups (Souza et al., 2017).

Results

A convenience sample of 134 entry-level occupational therapy students were enrolled in the study. Twenty-one (<1%) responses were missing for the overall CAM–Observer. Five responses were missing for the Advocating, Empathizing, and Encouraging subscales. Six responses were missing for the Collaborating subscale. Provided that the Rasch approach accounts for missing data (Bond & Fox, 2015), all participant questionnaires were included in the data analysis regardless of the missing data.

The 4-point rating scale demonstrated appropriate rating scale functioning for the overall CAM–Observer and the six subscales. Each rating category demonstrated ≥10 responses, outfit MnSq < 2.00, and monotonic advancement of the observed averages. The overall CAM–Observer and each of the six subscales met the dimensionality criteria, but the item fit for the overall CAM–Observer and the Collaborating subscale did not meet the cutoff criteria (Table 1). Three Advocating items and one Collaborating item demonstrated inappropriate fit to the Rasch model for the overall CAM–Observer, but the same Collaborating item also demonstrated inappropriate fit to the Rasch model for the Collaborating subscale (Table 2). Person fit fell below the criteria for the overall CAM–Observer but met the criteria for each of the six subscales (see Table 1).

Table 1.

Summary of Rasch Analysis Findings

Variable	Overall CAM–Observer	Advocating	Collaborating	Empathizing	Encouraging	Instructing	Problem-Solving
Dimensionality, %	57	61	64	53	62	59	52
Item fit, %	87 a	100	80 a	100	100	100	100
Person fit, %	85 a	94	99	96	93	96	96
Item targeting
Ceiling effect, %	0	0	4	9	19 a	24 a	5
Floor effect, %	0	42 a	2	0	0	0	6
Item separation
Coefficient	8.41	4.53	6.58	6.91	4.77	6.73	1.50 a
Reliability	0.99	0.95	0.98	0.98	0.96	0.98	0.69 a
Person separation
Coefficient	2.92	0.00 a	2.03	1.56	2.00	1.61	1.88
Reliability	0.89	0.00 a	0.81	0.71	0.80	0.72	0.78

Note. CAM = Clinical Assessment of Modes.

Did not meet the cutoff criteria.

Table 2.

Item-Fit Statistics for the Overall CAM-Observer

Mode/Abbreviated Item Description	Overall CAM-Observer		Individual Mode Subscales
Mode/Abbreviated Item Description	MnSq	Zstd	MnSq	Zstd
Advocating
1. Helped the client get access to resources or people in the community	2.4 a	5.1 a	1.3	1.2
2. Talked about legal/educational/social rights for people with disabilities	0.9	0.0	1.0	0.3
3. Reminded the client that other people like them shared similar experiences	1.4	2.8	0.6	−2.8
4. Made the client aware of individuals, social groups, and/or places in the community	2.1 a	4.1 a	1.3	1.0
5. Helped the client make contact with people who had a similar experience	1.9 a	5.0 a	1.1	0.6
Collaborating
6. Allowed the client to choose what would happen next	1.2	1.5	0.8	−1.9
7. Made sure that the client worked on what mattered most to him or her	1.0	−0.4	0.7	−2.6
8. Improved or changed something when the client pointed out that it was not helpful	1.8 a	5.8 a	1.8 a	5.2 a
9. Said or did things that made the client feel like he or she was an active participant	0.7	−2.7	0.8	−1.9
10. Gave the client control over what he or she accomplished	0.9	−1.1	0.8	−1.4
Empathizing
11. Listened to the client with true interest	0.7	−2.5	0.6	−3.2
12. Asked the client questions with a high level of sensitivity	1.0	−0.3	0.9	−0.5
13. Tried to understand the client’s thoughts and feelings	0.9	−1.4	0.8	−1.5
14. Shared something about his or her personal experience	1.2	2.1	1.5	3.8
15. Tried hard to understand the client’s needs by listening, observing, or asking questions	0.8	−1.5	1.0	−0.2
Encouraging
16. Pointed out what the client was good at doing	0.7	−3.0	0.8	−1.4
17. Made the client feel confident about what he or she was doing	0.6	−4.5	0.5	−4.0
18. Positive attitude showed the client that he or she believed the client was ready	1.0	0.0	1.2	1.6
19. Said things that made the client feel hopeful	1.0	−0.1	1.5	3.3
20. Gave the client a compliment or other kind of reward for something he or she did	0.8	−2.1	0.8	−1.4
Instructing
21. Explained what was happening or told the client what would happen next	0.7	−1.7	0.7	−1.6
22. Told or demonstrated to the client how to improve his or her performance or behavior	1.0	0.0	1.2	1.4
23. Gave the client clear directions	0.7	−2.4	0.9	−0.8
24. Showed a sense of conviction when making a recommendation	0.8	−1.6	1.1	0.8
25. Taught the client something	0.8	−1.8	0.9	−0.4
Problem-solving
26. Helped the client to think about a problem or activity in a different way	0.7	−3.1	0.7	−3.3
27. Explained different alternatives when guiding the client to make a decision	1.1	1.0	0.9	−0.7
28. Helped the client think about a problem in a clear-headed, nonemotional way	1.1	0.5	1.4	3.0
29. Helped the client consider many different ways of doing things	0.9	−0.6	0.7	−2.7
30. Helped the client look at a problem by breaking it down into smaller parts	1.2	1.6	1.3	2.4

Note. CAM = Clinical Assessment of Modes; MnSq = mean square; Zstd = Zstandard.

Item demonstrated inappropriate fit to the expectations of the Rasch model.

Ceiling effects were found for the Encouraging and the Instructing subscales, and a floor effect was found for the Advocating subscale (see Table 1). Visual inspection of the person–item hierarchy map suggested that the item difficulty sufficiently targeted the ability of the people in the sample (Figure 1). The Advocating and Problem-Solving items were least likely to be endorsed by the instructor–observers. Instructing and encouraging items were the most likely to be endorsed by the instructor–observers.

Figure 1.

Person–item hierarchy map for the CAM–Observer.

With the exception of the Problem-Solving subscale, the overall CAM–Observer and the individual subscales demonstrated item separation coefficients >4.00 with reliability ≥0.95 (see Table 1). The Advocating subscale demonstrated person separation coefficient and reliability of 0.00 (see Table 1). The overall CAM–Observer and the Collaborating and Encouraging subscales demonstrated person separation coefficients ≥2.00 with reliability ≥0.80 (see Table 1). The Empathizing, Instructing, and Problem-Solving subscales demonstrated person separation coefficients >1.50 with reliability >0.70 (see Table 1).

Discussion

Study findings support the structural validity of the CAM–Observer for evaluating students’ use of therapeutic communication modes from the instructor’s perspective. The instructors used the 4-point rating scale as intended, with an appropriate spread of responses across the four rating categories. Dimensionality analysis suggested that the items on the overall CAM–Observer and the six subscales measured a single construct of communication. This finding supports the notion that the assessment can be used to measure global and individual modes of communication. With a few exceptions (discussed later in this article), the overall CAM–Observer and the six individual subscales demonstrated appropriate item and person fit and item targeting in this sample. Except for the Problem-Solving subscale, the item separation statistics confirmed that the sample size was sufficient to reliably separate items according to difficulty. Except for the Advocating subscale, the item separation statistics confirmed that the sample was adequate for separating people into high- and low-performance groups.

Three Advocating items on the overall CAM–Observer demonstrated inappropriate fit, which was likely because of the low frequency of observed communication associated with the Advocating mode in this sample. Low frequency of item responses may have also contributed to greater than expected floor effects and low person separation for the Advocating subscale, suggesting that the Advocating subscale may not be effective in distinguishing between different levels of student performance. The Advocating mode encompasses communication that raises clients’ awareness of their legal rights, empowers them to access supports and resources in the community, and normalizes their experience by introducing them to others with a similar experience. Additional research is necessary to investigate the prevalence of Advocating mode use.

One Collaborating item demonstrated inappropriate fit on the overall CAM–Observer and the Collaborating subscale. This finding may be the result of individual differences in interpretation of the item, and the instructors may not have been consistent in identifying the clients’ verbal versus nonverbal approaches to providing feedback. To support clarity and consistency in item interpretation, we recommend the following revision to this item: “The therapist changed something about their approach in response to the client’s body language or feedback.” Structural validity of the CAM–Observer with the revised item will need to be evaluated in future research.

Person fit fell slightly below the criteria for the overall CAM–Observer but met the criteria for each of the six subscales. Because the person-fit criteria were exceeded by only 5%, it was not deemed a significant threat to validity to justify item removal or revision. The Encouraging and the Instructing subscales demonstrated greater than expected ceiling effects and were most frequently endorsed by the instructors in this sample. These modes may have been more natural for students to acquire and use at the start of their occupational therapy education. It is possible that students may be overrelying on these modes or experiencing difficulty in being able to switch out of these modes while interacting with clients.

Item separation was lower than expected for the Problem-Solving subscale, and there was not sufficient evidence to confirm item difficulty. Although there was no evidence of a significant floor effect, the visual inspection of the person–item hierarchy maps suggested that the instructors infrequently endorsed the items on this subscale. The Problem-Solving mode requires students to shift their communication toward statements that facilitate the client’s ability to think critically about different options and solutions to a problem. Students may require increased experience and practice interacting with clients in a clinical capacity before being able to effectively switch to the Problem-Solving mode of communication.

To our knowledge, this is the first study to evaluate an assessment of student performance of the therapeutic use of self in occupational therapy education. In the process of self-reflection, students often seek out input from others (Iliff et al., 2019). Equipping instructors with a structured approach for evaluating students’ communication can support their efforts to provide constructive and theoretically grounded feedback. Future research may consider combining qualitative and quantitative approaches to self-reflection, which is recommended for teaching professional behaviors in academic settings (Howard & Barton, 2019). The CAM–Observer may enhance student learning by providing a quantitative measure, which serves as a starting point for additional written reflection assignments widely used in entry-level occupational therapy curricula.

Limitations

Study findings are limited by the small sample size and convenience sampling, and they may not apply to other settings. The data collection was limited to a single academic institution that places a strong emphasis on promoting students’ therapeutic use of self according to the IRM. The instructors of the course on the therapeutic use of self served as data collectors in the study. In addition to their experience with teaching the course, the instructors received ongoing mentorship on the IRM and the CAM–Observer from the second author.

Implications for Occupational Therapy Education

The results of this study have the following implications for occupational therapy education:

The CAM–Observer offers a structured means of assessing students’ therapeutic use of self according to the IRM.

Except for the Advocating subscale, the CAM–Observer demonstrated appropriate structural validity for evaluating students’ overall and individual mode use from the instructor’s perspective.

The Advocating subscale was prone to floor effects and was not effective in distinguishing between students who demonstrated high- and low-frequency use of the Advocating mode.

Conclusion

The CAM–Observer offers occupational therapy educators a theoretically grounded approach to evaluating students’ therapeutic communication in an academic setting. The CAM–Observer can be used to guide constructive feedback on students’ interpersonal competencies and, as such, can support student readiness for entry-level practice.

Footnotes

Acknowledgments

We thank Jenica Lee and Celeste Januszewski for contributing to data collection and participant recruitment in this study. We also thank all of the student participants.

References

Accreditation Council for Occupational Therapy Education. (2012). 2011 Accreditation Council for Occupational Therapy Education (ACOTE®) Standards. American Journal of Occupational Therapy, 66(Suppl. 6), S6–S74. https://doi.org/10.5014/ajot.2012.66S6

Accreditation Council for Occupational Therapy Education. (2018). 2018 Accreditation Council for Occupational Therapy Education (ACOTE^®) Standards and Interpretive Guide (effective July 31, 2020). American Journal of Occupational Therapy, 72(Suppl. 2), 7212410005. https://doi.org/10.5014/ajot.2018.72S217

American Occupational Therapy Association. (2014). Occupational therapy practice framework: Domain and process (3rd ed.). American Journal of Occupational Therapy, 68(Suppl. 1), S1–S48. https://doi.org/10.5014/ajot.2014.682006

Arnadóttir

, & Fisher

A. G.

(2008). Rasch analysis of the ADL scale of the A-ONE. American Journal of Occupational Therapy, 62, 51–60. https://doi.org/10.5014/ajot.62.1.51

Bond

, & Fox

C. M.

(2015). Applying the Rasch model: Fundamental measurement in the human sciences (3rd ed.). New York: Routledge. https://doi.org/10.4324/9781315814698

Davidson

D. A.

(2011). Therapeutic use of self in academic education: A mixed-methods study. Occupational Therapy in Mental Health, 27, 87–102. https://doi.org/10.1080/0164212X.2011.543966

Fan

C.-W.

, & Taylor

R. R.

(2016). Assessing therapeutic communication during rehabilitation: The Clinical Assessment of Modes. American Journal of Occupational Therapy, 70, 7004280010. https://doi.org/10.5014/ajot.2016.018846

Howard

, & Barton

(2019). Self-reflection and measurement of professional behavior growth in entry-level occupational therapy students. Journal of Occupational Therapy Education, 3, 3. https://doi.org/10.26681/jote.2019.030103

Iliff

Tool

Bowyer

Parham

Fletcher

, & Freysteinson

(2019). Occupational therapy student conceptions of self-reflection in Level II fieldwork. Journal of Occupational Therapy Education, 3, 5. https://doi.org/10.26681/jote.2019.030105

10.

Linacre

J. M.

(2002). Optimizing rating scale category effectiveness. Journal of Applied Measurement, 3, 85–106.

11.

Linacre

J. M.

(2019). Winsteps Rasch measurement computer program user’s guide. Retrieved from https://www.winsteps.com/a/Winsteps-Manual.pdf

12.

McHorney

C. A.

, & Tarlov

A. R.

(1995). Individual-patient monitoring in clinical practice: Are available health status surveys adequate? Quality of Life Research, 4, 293–307. https://doi.org/10.1007/BF01593882

13.

Schwank

Carstensen

Yazdani

, & Bonsaksen

(2018). The course of self-efficacy for therapeutic use of self in Norwegian occupational therapy students: A 10-month follow-up study. Occupational Therapy International, 2018, 2962747. https://doi.org/10.1155/2018/2962747

14.

Smith

R. M.

, & Miao

C. Y.

(1994). Assessing unidimensionality for Rasch measurement. In Greenwich

W. M.

(Ed.), Objective measurement: Theory into practice (Vol. 2, pp. 316–328). Norwood, NJ: Ablex.

15.

Solman

, & Clouston

(2016). Occupational therapy and the therapeutic use of self. British Journal of Occupational Therapy, 79, 514–516. https://doi.org/10.1177/0308022616638675

16.

Souza

M. A. P.

Coster

W. J.

Mancini

M. C.

Dutra

F. C. M. S.

Kramer

, & Sampaio

R. F.

(2017). Rasch analysis of the Participation Scale (P-Scale): Usefulness of the P-Scale to a rehabilitation services network. BMC Public Health, 17, 934. https://doi.org/10.1186/s12889-017-4945-9

17.

Taylor

R. R.

(2020). The intentional relationship: Occupational therapy and use of self (2nd ed.). Philadelphia: F. A. Davis

18.

Taylor

R. R.

, & Popova

E. S.

(2019). Clinical Assessment of Modes–Observer, Version 3.0. Chicago: University of Illinois at Chicago, College of Applied Health Sciences, Department of Occupational Therapy. Retrieved from http://irm.ahslabs.uic.edu/assessments/

Evaluating Students’ Use of Therapeutic Communication in Entry-Level Education: The Observer Version of the Clinical Assessment of Modes (CAM–Observer)

Abstract

Method

Measures

Analysis

Results

Discussion

Limitations

Implications for Occupational Therapy Education

Conclusion

Footnotes

Acknowledgments

References