Development and Validation of a Collaborative Behaviors Objective Assessment Tool for End-of-Life Communication

Abstract

Context:

Helping families make end-of-life decisions requires close collaboration between physicians and nurses. However, medical and nursing students have little formal training in how to collaborate in this task, and few instruments are available to measure collaborative behaviors.

Objectives:

The objective of this project was to develop and validate observational assessment tools to measure specific interprofessional competencies in medical and nursing students related to end-of-life discussions.

Design:

A literature search for evidence-based guidelines and competencies and focus groups with an expert panel of nurses and physicians were used to outline best collaborative practice behaviors for nurses and physicians in an end-of-life decision making simulation. The panel used these practice-behavior checklists to rate videotaped student scenarios and then refined the checklists for validity and clarity until the tools had acceptable inter-rater reliability.

Setting:

The setting was a workshop teaching end-of-life communication to third-year nursing and medical students.

Measurements:

Inter-rater reliability was measured using percent agreement and kappa; internal consistency was measured using Cronbach's alpha.

Results:

Collaborative behaviors objective assessment tools (CBOATs) for nursing and medical students were developed. For the medical CBOAT we found 85% agreement between raters, with an overall kappa of 0.744 and Cronbach's alpha of 0.806. For the nursing CBOAT there was 81% agreement, with a kappa of 0.686 and Cronbach's alpha of 0.845.

Conclusions:

Development of an end-of-life CBOAT clarified the important collaborative behaviors needed by physician and nurse. The resulting instruments provide a helpful guide for teaching interprofessional sessions related to the end of life and measuring student outcomes using an objective strategy.

Introduction

Patients with life-limiting illnesses face difficult choices about the use of CPR, mechanical ventilation, and other interventions. Caring for these patients requires close collaboration^1–4 between disciplines including nursing, medicine, social work, chaplaincy, and many others. Unfortunately, these professionals may not communicate well about end-of-life care. In particular, many studies document inadequate physician-nurse collaboration, including poor communication^5,6 and conflicts about end-of-life care.⁷ Poor collaboration between these professionals has been identified as a major factor in limiting quality of end-of-life care² (EOLC), contributing to moral distress in physicians and nurses, and increasing turnover among ICU nurses.⁷ For these reasons interprofessional collaboration has been identified as a core competency for clinicians learning to care for dying patients and their families.^8–11

Despite the value assigned to interprofessional collaboration in this setting, medical and nursing students often have little formal training in how to work together in helping patients and families make difficult decisions about EOLC.^12,13 Interprofessional education (IPE), in which nursing and medical students gain experience working together, is uncommon, since undergraduate education in nursing and medicine tends to operate in “silos.”¹⁴ Furthermore, the concept of collaboration in this setting is not clearly defined. A review of the literature reveals that there are very few observational tools available for measuring interprofessional competencies in any setting.^15,16 Existing tools that measure collaboration are almost exclusively participant self-assessment surveys of attitudes and knowledge, rather than observational tools that objectively measure interprofessional competencies.^{12,13,17–19} Observational tools that measure communication skills during family meetings tend to evaluate physician performance and do not account for the role of the nurse or other professionals during such interactions.^20,21 No objective tools are available to measure nurse behaviors during family meetings.

The purpose of this project was to develop and validate observational assessment tools to measure interprofessional competencies for EOLC in nursing and medical students. While EOLC collaboration will include many disciplines, these two professions were chosen due to their central role in EOLC. These collaborative behaviors objective assessment tools (CBOATs) were designed to be used to evaluate the skills of students before and after participating in an interprofessional EOLC workshop using a simulated family meeting scenario.

Methods

Step 1: Literature search for evidence-based guidelines and competencies. In order to establish content validity, we began with a literature search on interprofessional collaboration,^{1–12,21–28} EOLC communication,^29–38 and existing instruments that measure collaboration and/or communication skills.^{12,13,15–19,21,33,38–46}

Step 2: Case development. The Anita Bowes case (see Appendix 1) was initially developed by the authors based on a literature review and clinical experience. A draft of the case was reviewed by our expert panel (membership described below) and refined to ensure that the case was clinically relevant and appropriately illustrated physician-nurse collaboration in an EOLC discussion. (See Fig. 1.)

Step 3: Expert panel outlines best practice behaviors and learning objectives. Members of this panel included the medical and nursing directors of the medical ICU; members of the nursing and medical faculty; and staff nurses in oncology, critical care, and palliative care. This panel included equal numbers of physicians and nurses and was jointly led by a physician and a nurse investigator (LJB, JME) using a semistructured guide. Experts were asked to use the Anita Bowes EOLC family meeting case to define behaviors that reflect best practices for collaborative care in this setting. These discussions were audiotaped and transcribed. Based on these discussions, checklists of objective behaviors (CBOATs) were developed, indicating competency skills for medical students and for nursing students.

Step 4: Content validity and inter-rater reliability.

• Focus group raters. Once draft CBOATs were completed, focus groups of expert nurses and physicians viewed videotaped pilots of the Anita Bowes case and used them to rate the skills of nursing and medical students. Areas of disagreement were identified, and participants reviewed the tapes and discussed differences. Items on the CBOATs were revised to better define the behaviors being measured. Descriptions of the behaviors (operationalization) and a three-point rating scale (satisfactory, less than satisfactory, did not attempt) were added to rate each behavior.

• Project team. Once general agreement was reached in the focus groups regarding CBOAT items, inter-rater reliability was assessed by the core project team. Five project team members viewed 10 randomly chosen tapes of five medical and five nursing students who had completed simulations with standardized patients and providers, enacting the Anita Bowes case, and rated the students' performances using the CBOATs. Ratings were compared and differences were discussed. This data was used to further refine the rating scales to improve clarity and to address specific problems unique to the IPE assessment. The project team repeated this process until the results showed 90% or more agreement on the medical and nursing CBOATs.

• Standardized patients (SPs). SPs who would be evaluating the students on their performance in the simulated case were given instructions on how to rate medical and nursing student performance using the CBOAT scale. Each SP was then instructed to evaluate 10 randomly selected videos using the CBOAT. Inter-rater reliability was assessed using the kappa statistic and percentage agreement among raters, and Chronbach's alpha was applied to test for internal consistency.

Results

The process described above resulted in the development of medical and nursing CBOATs to use in evaluating students who have participated in a workshop with a simulated family meeting scenario to discuss goals of care. The behaviors listed in the CBOATs reflect the University of Virginia IPE competencies communication, professionalism, shared problem solving, shared decision making, and conflict resolution.⁹ Each CBOAT divides the family meeting scenario into two sections. The first section is the interaction between nurse and physician prior to the family meeting (outlined in Table 1), and the second section addresses their behaviors in the family meeting (outlined in Table 2).

Table 1.

Pre-Meeting Collaborative Behaviors Objective Assessment Items, Specific Roles, and IPE Competencies

Item	Behavior measured	Profession-specific roles in Anita Bowes case	IPE competencies/collaborative aspects of behavior
1	Introductions (to co-worker)	MD, RN introduce themselves to each other	Professionalism Collaboration demonstrated by respectful introductions
2	Exchange information with co-worker: clinical information	MD: presents clinical information from signout and bedside assessment RN: presents clinical information from nursing assessment and longitudinal information from prior interactions	Communication Collaboration demonstrated by presenting clear, complete information and active listening to information from co-worker
3	Exchange information with co-worker: psychosocial information	RN: has more information than MD about family; should take lead in presenting this information MD: requests information regarding family if RN does not present this	Communication Collaboration demonstrated by presenting clear, complete information and active listening to information from co-worker
4	Exchange information with co-worker: information regarding prior goals of care discussions	RN: Presents information regarding prior discussion with patient about end-of-life preferences MD: responds to information	Communication Collaboration demonstrated by presenting clear complete information and active listening to information from co-worker
5	Plan family meeting	MD and RN: must agree on purpose of meeting and roles within meeting	Shared problem solving and decision making Collaboration demonstrated by both clinicians taking responsibility for ensuring this task is completed

IPE, Interprofessional education.

Table 2.

Within-Meeting Collaborative Behaviors Objective Assessment Items, Profession-specific roles, and IPE Competencies

Item	Behavior measured	Profession-specific roles in Anita Bowes case	IPE competencies / collaborative aspects of behavior
6	Introductions to family member	RN knows sister, will introduce MD	Professionalism Collaboration demonstrated by both clinicians taking responsibility for ensuring this task is completed
7	Explain purpose of family meeting	MD or RN: briefly state purpose of meeting	Communication Collaboration demonstrated by both clinicians taking responsibility for ensuring this task is completed
8	Ask family member about understanding of clinical issues	MD or RN: Ask what sister knows about patient's condition	Communication Collaboration demonstrated by both clinicians taking responsibility for ensuring this task is completed
9	Provide information about clinical status	MD: takes lead in explaining clinical situation RN: adds to discussion regarding patient's clinical condition	Shared problem solving and decision making MD demonstrates collaboration by using “we language,” inviting nursing perspective. RN demonstrates collaboration by adding nursing perspective, clarifying or asking questions
10	Respond to emotions	MD and RN: both should respond by allowing time to react, identifying and supporting emotions	Professionalism Collaboration demonstrated by both clinicians taking responsibility for ensuring this task is completed
11	Provide information regarding EOLC choices	MD: takes lead in describing EOLC choices RN: adds to discussion regarding EOLC choices	Shared problem solving and decision making MD demonstrates collaboration by using “we language,” inviting nursing perspective. RN demonstrates collaboration by adding nursing perspective, clarifying information, or asking questions
12	Ask family member about patient's EOLC wishes	MD or RN: ask sister about her understanding of patient's EOLC wishes	Shared problem solving and decision making Collaboration demonstrated by both clinicians taking responsibility for ensuring this task is completed
13	Provide information regarding patient's statements about EOLC wishes	RN: takes lead in describing prior discussions with patient MD adds to discussion	Shared problem solving and decision making; conflict resolution RN demonstrates collaboration by presenting clear and complete information MD demonstrates collaboration by adding physician perspective, clarifying, or asking questions
14	Summary and follow-up plan	MD and RN: both should contribute to a follow-up plan	Shared problem solving and decision making Collaboration demonstrated by both clinicians taking responsibility for ensuring this task is completed, using “we” language and follow-up plan that indicates ongoing teamwork

IPE, Interprofessional education.

Medical CBOAT

Pre-meeting

This section (see Table 1) has five items measuring the following behaviors: introductions; exchange of clinical information, psychosocial information, and information about goals of care; and planning for the meeting.

Meeting

The meeting (see Table 2) section consists of nine items (items 6–14). These items measure the following behaviors: introductions, asking about the family member's understanding of the situation, providing information to the family member, discussion of choices regarding EOLC, and responding to emotions. Item 13 was designed to assess collaboration in discussing Anita's wishes regarding life support. We felt that the nurse had responsibility for conveying information to the family member about Anita's statements; thus the medical CBOAT item measures the medical student's collaboration with the nurse in that discussion.

General items

The final two items (not noted in Table 2) ask the rater to reflect on the overall skills of the medical student in collaborating with the standardized nurse (SRN) and overall skills in communicating with the standardized family member (SFM).

Nursing CBOAT

Pre-meeting

The pre-meeting (see Table 1) section's five items, which are very similar to the medical CBOAT, measure introductions, exchange of information, and planning for the meeting.

Meeting

The meeting section (see Table 2) consists of nine items, measuring the same behaviors as the medical CBOAT. However, several differences from the medical CBOAT are notable. Items 9 and 14 measure behaviors where we expected the physician to take the lead: provision of clinical information (item 9) and presentation of EOLC choices (item 14). In the nursing CBOAT these items measure whether or not the nursing student contributed to the discussion of this topic. In item 13 the nurse would be expected to take the lead in providing information to the family member regarding his or her discussion with Anita about her EOLC wishes.

General items

Two final items ask the rater to reflect on the overall skills of the nursing student in collaborating with the standardized physician (SMD) and overall skills in communicating with the SFM.

Inter-rater reliability

For the medical CBOAT, we found 85% agreement between raters, with an overall kappa of 0.744. For the nursing CBOAT, agreement was 81% between raters and an overall kappa of 0.686. Cronbach's alpha was calculated to measure internal consistency of the instruments. For the medical CBOAT, Cronbach's alpha=0.806, and for the nursing CBOAT, Cronbach's alpha=0.845.

Challenges

This project resulted in the creation and validation of objective assessment tools to measure interprofessional collaborative compentencies for medical and nursing students in end-of-life situations. The present versions of the tools demonstrate good inter-rater reliability and internal consistency for a newly developed tool. Difficulties in obtaining higher inter-rater reliability for items on the CBOATs occurred mainly in two circumstances. In the first instance, there is difficulty in rating an individual student in an activity that is collaborative in nature, since the performance of one participant partly depends on the performance of his or her colleague. In the videotaped encounters used to pilot the CBOATs, each student interacted with two standardized performers—one playing the part of the family member and one playing the part of the other team member. Despite extensive training, the standardized clinicians would often engage in some of the behaviors that were expected of the student; for example the SRN might present prognostic data or the SMD might talk about the patient's wishes. This made it difficult to rate the student's performance.

The second issue relates to rating the student on a three-point scale (satisfactory, less than satisfactory, behavior not attempted) versus a two-point scale (behavior performed or not performed). For some of the items a two-point scale was used because the behavior was simple (for example, introductions). A two-point scale was also used when the behavior related to responding to or contributing to a behavior was mainly the responsibility of the colleague.

However, for many of the items, we felt it was worth the effort to attempt to use a three-point scale to differentiate between satisfactory and less than satisfactory performance of a critical behavior, such as providing clinical information, in order to tease out some of the more complex skills necessary for effective collaboration. Defining what it means to perform these complex behaviors well occupied most of the time spent in developing this instrument. Despite attempts to clearly define the rating scale, the choice to use a three-point versus two-point scale was a major cause of decreased rater agreement. This was particularly true with the nursing CBOAT, which would have 96% agreement if the scale was collapsed to a two-point (indicating presence or absence of the behavior) rather than three-point scale. Inter-rater agreement for the medical CBOAT would have increased to 89% in a binary rating scale.

Discussion

Barriers to interprofessional undergraduate education for medical and nursing students include structural problems, such as difficulties in aligning academic calendars. On a more fundamental level, however, it is difficult to teach collaboration when it is not clear that we agree about what that means. When physicians and nurses who work closely together are asked about the quality of their collaboration, physicians commonly rate the level of collaboration higher than their nursing colleagues, suggesting that these professionals have differing ideas about what constitutes collaborative behavior.^6,7,17 Even studies that stress the importance of having the nurse at family meetings often fail to outline how physicians and nurses should collaborate in presenting information and offering choices to families.³ For this reason, general exhortations to collaborate may not improve interprofessional teamwork around EOLC without more specific delineation of what constitutes collaborative behavior and tools to measure whether such behavior is occurring.

Development of the EOLC CBOAT scales was a complex process, requiring consideration of the appropriate roles of the physician and nurse in EOLC discussions. We came to a consensus that while the physician should generally take the lead in presenting information about the patient's clinical situation including prognosis and EOLC choices, the nurse should be expected to make significant contributions to discussion of these topics. We also agreed that when the nurse has unique knowledge concerning information relevant to the discussion, he or she should take the lead in presenting that information. The Anita Bowes case was constructed as a scenario in which the nurse had access to information concerning the patient's wishes, which the family member and the physician did not have, providing an example where the nurse would be expected to take the lead for a portion of this discussion.

In addition to considering specific roles of the physician and nurse as individual team members, we also had to clarify what we meant by the word collaboration and how we might measure it. One objection to the CBOATs is that they confound measuring skills related to collaboration with measuring skill at end-of-life communication. However, we would argue that collaboration does not exist separately from collaborating about something; it is always within the context of an activity. The form collaboration takes will change depending on the nature of that activity—collaboration in the operating room is likely to be very different from collaboration in a family meeting. Furthermore, teaching and measuring a skill such as end-of-life communication to medical students without integrating the role of other disciplines reinforces the idea that this activity is the job of the physician and that collaboration is not essential to the task. Our approach assumes that collaboration does not mean two individuals doing their complementary roles separately,⁹ and that teaching collaboration means teaching a skill (such as how to appropriately participate in a family meeting) so that those performing the skill are learning to do it in a collaborative manner.

Our tool reflects this definition of collaboration. The pre-meeting section of the CBOAT measures that aspect of collaboration that consists of communication of information. Students are rated not only on how well they convey information but also on how well they obtain and respond to information from their colleague. Collaboration during the family meeting is measured in several ways. When students are rated on their role-specific responsibilities, they are given the highest rating only when they do it in a collaborative manner. Specifically this means that in order to obtain a satisfactory rating for provision of information, students must convey this information clearly and completely (the role-specific skill) and show collaboration by using “we” language or by asking for input from their colleague. During portions of the family meeting where the other professional is taking the lead, students are expected to show collaboration by also contributing to the discussion. Some behaviors are not specific to either role; collaboration in those behaviors means that both parties take responsibility for ensuring that this is done. For example, asking what the family member knows about the patient's illness is crucial, but either professional could ask this question. In a collaborative interaction, if the physician does not ask this question, the nurse should do so. Finally, the last two items measure the rater's overall sense of how well the student collaborated with his or her colleague, in recognition of the fact that aspects of collaboration such as body language and eye contact were difficult to measure with the very specific items we developed.

One significant weakness of these tools is that they only focus on the roles of two professionals (nursing and medicine), when in reality excellent EOLC depends on the work of many other disciplines—in particular social work and chaplaincy.¹² We chose these two professionals as a starting point due to the key roles they play in such decisions¹⁴ and because of prior studies that document disagreement about nurse-physician collaboration in EOLC.^6,22,25,27 Expanding the scope of IPE to include the development of teaching and assessment methods that include a wider range of disciplines is an important goal for future work.

Our results show excellent agreement on whether the medical students and nursing students performed the identified collaborative behaviors. Agreement was moderate as to the skills involved in this scenario, despite strenuous efforts to clarify the rating scale in order to make these distinctions explicit. Interestingly, the challenge to outline and rate collaborative behaviors for the nurse was greater than for the physician, and this finding may indicate a greater need to define the role of the nurse in such collaborative EOLC situations. Despite any limitations of these tools, we found the process of identifying and rating these behaviors to be extremely valuable, and hope they will serve as a contribution to the conversation regarding interprofessional interactions in EOLC planning.

Footnotes

Acknowledgments

This work was supported by a grant from the Josiah Macy Jr. Foundation; pilot work was also supported by the Academy of Distinguished Educators and the Oncology Nursing Society.

Author Disclosure Statement

No competing financial interests exist.

Appendix 1. Bowes Case

References

Christ

, Blacker

: Improving interdisciplinary communication skills with families. J Palliat Med, 2005; 8:855–856.

Mularski

, Bascom

, Osborne

: Educational agendas for interdisciplinary end-of-life care curricula. Crit Care Med, 2001; 29(suppl):N16–N23.

Lilly

, De Meo

, Sonna

, Haley

, Massaro

, Wallace

, Cody

: An intensive communication intervention for the critically ill. Am J Med, 2000:109:469–475.

von Gunten

, Ferris

, Emanuel

: Ensuring competency in end-of-life care: Communication and relational skills. JAMA, 2000; 284:3051–3057.

Azoulay

, Timsit

, Sprung

, et al., for the Conflictus Study Investigators and the Ethics Section of the European Society of Intensive Care Medicine: Prevalence and factors of intensive care unit conflicts. Am J Respir Crit Care Med, 2009; 180:853–858.

Hamric

, Blackhall

: Nurse-physician perspectives on the care of dying patients in intensive care units: Collaboration, moral distress, and ethical climate. Crit Care Med, 2007; 35:422–429.

Ferrand

, Lemarie

, Regnair

, et al.: Discrepancies between perceptions by physicians and nursing staff of intensive care unit end-of-life decisions. Am J Respir Crit Care, 2003; 1310–1315.

Ferrell

, Dahlin

, Campbell

, et al: End-of-Life Nursing Education Consortium (ELNEC) Training Program: Improving palliative care in critical care. Crit Care Nurse Q, 2007; 30(3):206–212.

AAMC Interprosessional Education Collaborative Expert Panel. 2011. Core competencies for interprofessional collaborative practice: Report of an expert panel. Washington, DC.

10.

ACGME: ACGME Core Competencies for Hospice and Palliative Medicine. www.acgme.org/acgmeweb/Portals/0/PFAssets/ProgramRequirements/540_hospice_and_palliative_medicine_companion_09_2009.pdf. ACGME.

11.

AMA CME-CMS Joint Report 1-12. The Structure and Function of Interprofessional Health Care Teams. 2012. www.ama-assn.org/resources/doc/cms/i12-cmecms-jointreport.pdf

12.

Fineberg

: Preparing professionals for family conferences in palliative care: Evaluation results of an interdisciplinary approach. J Palliat Med, 2005; 8:857–866.

13.

Quinn

, Hudson

, Ashby

, Thomas

: “Palliative care: the essentials:” Evaluation of a multidisciplinary education program. J Palliat Med, 2008; 11:1122–1129.

14.

Curtis

, Shannon

: Transcending the silos: Toward an interdisciplinary approach to end-of-life care in the ICU. Intensive Care Med, 2006; 32:15–17.

15.

Freeth

, Hammick

, Reeves

, Koppel

, Barr

: Effective Interprofessional Education: Development, Delivery, and Evaluation. Oxford, UK: Blackwell Publishing, 2005.

16.

Heinemann

, Zeiss

: Team Performance in Health Care Assessment and Development. New York: Kluwer Academic/Plenum Publishers, 2002.

17.

Baggs

, Schmitt

, Mushlin

, et al.: Association between nurse-physician collaboration and patient outcomes in three intensive care units. Crit Care Med, 1999; 27:1991–1998.

18.

Hojat

, Fields

, Veloski

, et al.: Psychometric properties of an attitude scale measuring physician-nurse collaboration. Eval Health Prof, 1999; 22:208–220.

19.

Shortell

, Rosseau

, Gillies

, et al.: Organizational assessment in ICUs: Construct development, reliability and validity of the ICU nurse-physician questionnaire. Med Care, 1991; 29:709–723.

20.

Lorin

, Rho

, Wisnivesky

, Nierman

: Improving medical student intensive care unit communication skills: A novel educational initiative using standardized family members. Crit Care Med, 2006; 34:2386–2391.

21.

Von Guten

, Ferris

, Emanuel

: Ensuring competency in end-of-life care: Communication and relational skills. JAMA, 2000; 284:3051–3057.

22.

Frick

, Uehlinger

, Zenklusen

: Medical futility: Predicting outcome of intensive care unit patients by nurses and doctors: A prospective comparative study. Crit Care Med, 2003; 31:456–460.

23.

Oberle

, Hughes

: Doctors' and nurses' perceptions of ethical problems in end-of-life decisions. J Adv Nurs, 2001; 33:707–715.

24.

Breen

, Abernathy

, Abbott

, et al.: Conflict associated with decisions to limit life-sustaining treatment in intensive care units. J Gen Intern Med, 2001; 16:283–289.

25.

Reckling

: Who plays what role in decisions about withholding and withdrawing life sustaining treatment?. J Clin Ethics, 1997; 8:39–45.

26.

Thomas

, Sexton

, Helmreich

: Discrepant attitudes about teamwork among critical care nurses and physicians. Crit Care Med, 2003; 31:956–959.

27.

Truog

, Cist

, Brackett

, et al.: Recommendation for end-of-life care in the intensive care unit: The Ethics Committee of the Society of Critical Care Medicine. Crit Care Med, 2001; 29:2332–2348.

28.

Buckman

: How to Break Bad News. Baltimore, MD: JHU Press, 1992.

29.

Walling

, Lorenz

, Dy

, et al.: Evidence-based recommendations for information and care planning in cancer care. J Clin Oncol, 2008; 26:3896–3902.

30.

Back

, Anderson

, Bunch

, et al.: Communication about cancer near the end of life. Cancer, 2008; 113(7 suppl):1897–1910.

31.

Fried

, Bradley

, O'Leary

: Prognosis communication in serious illness: Perceptions of older patients, caregivers, and clinicians. J Am Geriatrics Soc, 2003; 51:1398–1403.

32.

Butow

, Dowsett

, Hagerty

, Tattersall

MH.

Communicating prognosis to patients with metastatic disease: What do they really want to know?. Support Care Cancer, 2002; 10:161–168.

33.

Zier

, Sottile

, Hong

, Weissfield

, White

: Surrogate decision makers' interpretation of prognostic information: A mixed-methods study. Ann Intern Med, 2012; 156:360–366.

34.

Henman

, Butow

, Brown

, Boyle

, Tattersally

MHN

: Constructions of decision-making in cancer. Psycho-oncology, 2002; 11:295–306.

35.

Hofmann

, Wenger

, Davis

, et al.: Patient preferences for communication with physicians about end-of-life decisions. SUPPORT investigators. Study to Understand Prognoses and Preference for Outcomes and Risks of Treatment. Ann Intern Med, 1997; 127:1–12.

36.

Chapman

, Abraham

, Jenkins

, Fallowfield

: Lay understanding of terms used in cancer consultations. Psycho-oncology, 2003; 12:557–566.

37.

The

, Hak

, Koeter

, van der Wal

: Collusion in doctor–patient communication about imminent death: An ethnographic study. Western J Med, 2001; 174:247–253.

38.

Agency for Healthcare Research and Quality: Closing the Quality Gap: Revisiting the State of the Science Series: Improving Health Care and Palliative Care for Advanced and Serious Illness. Rockville, MD: U.S. Dept. of Health and Human Services Agency for Healthcare Research and Quality, 2012.

39.

Fried

, Bradley

, O'Leary

: Prognosis communication in serious illness: Perceptions of older patients, caregivers, and clinicians. J Am Geriatrics Soc, 2003; 51:1398–1403.

40.

Szmuilowicz

, el-Jawahri

, Chiappetta

, Kamdar

, Block

: Improving residents' end-of-life communication skills with a short retreat: A randomized controlled trial. J Palliat Med, 2010; 13:439–452.

41.

Back

, Arnold

, Baile

, et al.: Efficacy of communication skills training for giving bad news and discussing transitions to palliative care. Arch Intern Med, 2007; 167:453–460.

42.

Wilkinson

, Perry

, Blanchard

, Linsell

: Effectiveness of a three-day communication skills course in changing nurses' communication skills with cancer/palliative care patients: A randomised controlled trial. Palliat Med, 2008; 22:365–375.

43.

Fallowfield

, Jenkins

, Farewell

, et al.: Efficacy of a cancer research UK communication skills training model for oncologists: A randomised controlled trial. Lancet, 2002; 359:650–656.

44.

Razavi

, Delvaux

, Marchal

, et al.: Does training increase the use of more emotionally laden words by nurses when talking with cancer patients? A randomised study. Br J Cancer, 2002; 87:1–7.

45.

Williams

, Fisicaro

, Veloski

, Berg

: Development and evaluation of a program to strengthen first year residents' proficiency in leading end-of-life discussions. Am J Hosp Palliat Care, 2011; 28:328–334.

46.

Fineberg

: Preparing professionals for family conferences in palliative care: Evaluation results of an interdisciplinary approach. J Palliat Med, 2005; 8:857–866.