Internal Medicine Trainee Self-Assessments of End-of-Life Communication Skills Do Not Predict Assessments of Patients,Families,or Clinician-Evaluators

Abstract

Purpose

: To investigate the strength of association between trainees' self-assessments of the quality of their end-of-life communication skills and the assessments of their patients, patients' families, and clinician-evaluators.

Methods

: As part of a randomized trial, pre-intervention survey data were collected at two sites from internal medicine trainees and their patients, patients' families, and clinician-evaluators. In this observational analysis, comparisons using regression analysis were made between (1) trainees' scores on a scale of perceived competence at communication about end-of-life care and (2) patients', families', and clinician-evaluators' scores on a questionnaire on the quality of end-of-life communication (QOC). Secondary analyses were performed using topic-focused subscales of these measures.

Results

: Internal medicine trainees (143) were studied with both self-assessment and external assessments. No significant associations were found between trainee perceived competence scores and primary outcome measures (p>0.05). Of the 12 secondary subscale analyses, trainees' self-ratings were significantly associated with external assessments for only one comparison, but the association was in the opposite direction with increased trainee ratings being significantly associated with decreased family ratings on “treatment discussions.” We also examined the correlation between ratings by patients, family, and clinician-evaluators, which showed significant correlations (p<0.05) for 7 of 18 comparisons (38.9%).

Conclusions

: Trainee self-evaluations do not predict assessments by their patients, patients' families, or their clinician-evaluators regarding the quality of end-of-life communication. Although these results should be confirmed using the same measures across all raters, in the meantime efforts to improve communication about end-of-life care should consider outcomes other than physician self-assessment to determine intervention success.

Introduction

The Institute of Medicine has determined that the quality of end-of-life care in America is poor, resulting in avoidable pain and distress among dying patients and their families, and that “the education and training of physicians and other health care professionals fail to provide them the attitudes, knowledge, and skills required to care well for the dying patient.” ¹ In response to this shortcoming, instruction on end-of-life care is required of many training programs by the Accreditation Council for Graduate Medical Education.² An important component of these training programs is an emphasis on improving trainee communication skills. Successes and failures in communication generate more gratitude and complaints than any other aspect of end-of-life care,,^3–4 and successful communication regarding end-of-life care is associated with improved quality of life and reduced intensity of care at the end of life.^5–6 Past interventions to improve the quality of communication about end-of-life care have used providers' self-assessments as outcome measures to gauge intervention success or failure,^7–9 yet no published study has examined the correlation between providers' self-assessment of clinical competency and patient- or family-centered outcomes. Thus it is unknown whether provider self-assessments accurately represent the assessments of their patients and patients' families.

As part of a five-year, randomized trial to evaluate the effectiveness of an experiential workshop to improve communication about end-of-life care among internal medicine trainees, we collected pre-intervention data in which trainees assessed their competence at communication about end-of-life care. In this paper, we conduct an observational analysis in which trainee pre-intervention self-assessments were compared with corresponding pre-intervention assessments by their patients, patients' families, and clinician colleagues. We hypothesized that trainee self-assessment would be positively associated with assessments by external evaluators. In an effort to characterize the agreement among external raters, we also examined the correlations among patients, families, and clinician-evaluators.

Methods

Study design

Data for this study were drawn from pre-intervention surveys completed during the first three years of the Improving Clinician Communication Skills (ICCS) study, a five-year randomized trial of an experiential training program designed to improve clinicians' end-of-life communication skills. All study procedures were approved by the institutional review boards of all involved institutions.

Participants

Trainees

The study population included trainees in internal medicine residency and select medicine specialty fellowships at the University of Washington or the Medical University of South Carolina. All internal medicine residents in postgraduate years (PGY) 1-5 as well as fellows from the following medicine specialties were invited to participate: geriatrics, nephrology, oncology, palliative medicine, and pulmonary and critical care medicine. Trainees and all evaluators were approached for participation with three recruitment attempts for nonrespondents.

Patients

All patients in the study had either inpatient or outpatient clinical encounters with enrolled trainees and were identified by screening the medical records of all patients cared for by an enrolled trainee. Encounters occurred between trainees and patients in the setting of their primary care clinic or during their rotations through various primary inpatient services (e.g., general medicine, medical intensive care unit (MICU), hematology-oncology). Eligible patients were those whom we expected to have a 50% or greater chance of mortality in the upcoming 12 months and who would therefore be appropriate candidates for communication about end-of-life issues. Patients were potentially eligible for the study if they met any of the following criteria: a diagnosis of a life-limiting illness, e.g., metastatic or Stage IV cancer, oxygen-dependent chronic obstructive pulmonary disease (COPD) or restrictive lung disease with a total lung capacity less than 50%, Stage III or IV heart failure and/or Childs' Class C liver disease;, comorbidities suggesting severe illness, i.e., a score of ≥5 on the Charlson Comorbidity Index;¹⁰ documentation of end-of-life care or communication, i.e., a hospice or palliative care consult or the presence of a do-not-resuscitate (DNR) order; MICU stay ≥72 hours; or age ≥80 years with hospital stay ≥72 hours. We also defined as an eligibility criterion the patient remembering the trainee well enough to evaluate his or her communication skills.

Study materials included a picture of the trainee to ensure that the patient recognized the trainee as someone who had provided clinical care to him or her. Both in-person and mail-based recruitment procedures were utilized with a total of three contacts for nonrespondents.¹¹

Families

Family members were identified through one of three recruitment methods: (1) participating patients identified family members involved with their medical care; (2) family members of noncommunicative hospitalized patients were identified during a patient's hospital stay; and (3) family members of patients who died were identified from the medical record. As with patients, family study materials included a photo of the trainee to ensure that the family member recognized the trainee as someone who had provided care to the patient and remembered the trainee well enough to assess his or her communication skills.

Clinician-evaluators

Clinician-evaluators included nurses and physicians who observed patient care provided by trainees and felt they knew the trainee well enough to assess their communication skills Nurse-evaluators were identified either through screening patient medical records, review of unit schedules, or by nurse supervisors as nurses familiar with the clinical care of the participating trainee. The identification of nurse clinician-evaluators was focused on units where patients meeting inclusion criteria were likely to receive care (e.g., oncology, intensive care). Physician-evaluators were faculty members that were identified through either patient medical records or institutional physician clinical schedules. Attending physicians who supervised trainees and co-signed trainee progress notes were eligible.

Data collection

Surveys were administered to participants between July 2007 and July 2010 prior to the trainee's commencement of the communication intervention. Some patients, family members, and clinician-evaluators completed evaluations for multiple trainees; to ensure independence of measurement, we selected one evaluation per evaluator for use in analyses. When a single evaluator provided more than one evaluation, we guided our selection of the survey to produce evaluations for the largest number of trainees; where selection between surveys had no effect on the size of the trainee sample, we used random selection.

Measures

Outcomes

This study's main outcomes were scale scores from the validated Quality of Communication (QOC) questionnaire completed by patients, family members, and clinician-evaluators. The two QOC scales, the General Communication (QOC-GEN) and End-of-Life Communication (QOC-EOL) scales, were scored according to published guidelines and ranged from 0-10, with 10 indicating “absolutely perfect.”¹²

Secondary outcomes were four topic-focused subscales derived from items in the QOC and items in a second questionnaire, the Quality of End-of-Life Care (QEOLC) that was also completed by patient, family, and clinician evaluators. The QEOLC is a validated questionnaire assessing physician skill at end-of-life care incorporating five domains of physician skills: communication, symptom management, affective skills, patient-centered values, and patient-centered systems.^13–14 For the purposes of these analyses, we selected two to eight items per subscale from the QOC and QEOLC with content similar to questions included in trainees' self-ratings. The subscales were: Bad News Discussions, Treatment Preferences Discussions, Discussions about End-of-Life Care and Dying, and Symptom Management Discussions. Scores were computed as item means and ranged from 0-10 with 10 indicating “absolutely perfect.” Instruments are available in the accompanying appendix and online at http://depts.washington.edu/eolcare/.

Predictors

The primary predictor was the Perceived Competence questionnaire, a 17-item scale with which trainees assessed their own perceived competence with end-of-life care and communication.¹⁵ Items were scored on a five-point Likert-type scale with labels supplied for three of the five points (1=not very competent; 3=somewhat competent; 5=very competent). A total score was computed as an average (“Perceived Competence”) across all valid items, and the score was rescaled to a range of 0-10 to match the scale of the outcome variables. Secondary predictors were created using items from the Perceived Competence scale that were topic-focused and matched items on the QOC and QEOLC. The four subscales contained from one to three items. Scores were averages of all valid responses and were rescaled to range from 0–10.

Confounders

We also examined a number of possible confounders. For trainees, we examined postgraduate year, sex, and racial/ethnic minority status. For evaluators, we examined sex and racial/ethnic minority status.

Data analysis

We used linear regression models with robust estimators to explore associations between the predictors, trainees' Perceived Competence total and subscale scores, and evaluators' outcomes including the QOC-EOL, QOC-GEN, and topic-focused subscales. Ratings from multiple external evaluators of a given evaluator type (patient, family, or clinician) for a single trainee were aggregated for the trainee, with the mean values used as outcomes in regression models.

We tested 18 unique models, regressing each outcome (n=6) for each respondent-type (n=3) on the relevant Perceived Competence predictor, with covariate adjustment for confounders. To identify confounders we first examined the bivariate associations between each of the 18 predictor-outcome combinations. Then for each predictor-outcome pair we ran five two-predictor models, each model including adjustment for one of the following covariates: trainee sex, trainee race, postgraduate year, proportion of evaluators who were female, and proportion of evaluators who were racial/ethnic minorities. Any covariate that changed the coefficient for the predictor of interest by 15% or more from that obtained for the bivariate association was included in the final model. Because of the exploratory nature of these analyses, we chose a liberal p≤0.05. Analyses were performed using STATA (version 11) (Stata Corporation, College Station, TX).

In order to explore whether associations existed between parallel evaluations provided by different external evaluator types, we examined Spearman's rho between paired ratings (patient-family, patient-clinician, family-clinician), aggregated by trainee, for the six external evaluations.

Results

There were 449 eligible trainees, of whom 254 (56.6%) enrolled as study participants. Of the enrolled participants, 196 completed the Perceived Competence scale (77.2% of those enrolled; 43.7% of those eligible). Twenty-nine trainees withdrew from the study due to difficulty scheduling the intervention, 13 of whom had completed baseline surveys. Of the 183 remaining trainees with completed baseline surveys, 143 of these had sufficient information to be included in analyses with at least one patient, family, or clinician-evaluator assessment available for comparison; these trainees constitute the sample for these analyses. Patient surveys were returned for 123 (67%) of the trainees; family surveys for 83 (45%); and clinician-evaluator surveys for 100 (55%).

Because some evaluators were asked to evaluate more than one trainee, response rates were calculated with survey as the unit of analysis. The response rate was defined as the number of useable surveys returned by eligible evaluators divided by the number of surveys distributed to eligible evaluators (the eligibility of nonresponding evaluators estimated from eligibility rates among responders). Response rates were 63% for patients (376/599); 84% for family members (182/217), and 69% for clinicians (506/738).

Completed surveys evaluating the 143 trainees were received from 362 patients, 180 family members, and 324 clinician-evaluators. Patients were evenly split by sex, whereas family members and clinicians were more likely to be female (Table 1).

Table 1.

Participant Characteristics

Characteristics	UW ^a	MUSC ^b	Total
Trainees: total n (% of row total)	90 (63)	53 (37)	143 (100)
Female, n (% of column total)	52 (58)	25 (47)	77 (54)
Minority race/ethnicity, n (% of column total)	15 (17)	14 (26)	29 (20)
Training year, n (% of column total)
PGY 1	63 (70)	16 (30)	79 (55)
PGY 2	10 (11)	17 (32)	27 (19)
PGY 3	11 (12)	10 (19)	21 (15)
PGY 4-8	6 (7)	10 (19)	16 (11)
Patient evaluators: total n (% of row total)	204 (56)	158 (44)	362 (100)
Female, n (% of column total)	91 (45)	71 (45)	162 (45)
Minority race/ethnicity, n (% of column total)	54 (26)	73 (46)	127 (35)
Family evaluators: total n (% of row total)	70 (39)	110 (61)	180 (100)
Female (% of column total)	51 (73)	79 (73)	130 (73)
Minority race/ethnicity, n (% of column total)	14 (20)	46 (43)	60 (34)
Clinician evaluators: total n (% of row total)	211(65)	113(35)	324 (100)
Female (% of column total)	133 (63)	68 (60)	201 (62)
Minority race/ethnicity, n (% of column total)	54 (26)	15 (14)	69 (22)

University of Washington.

Medical University of South Carolina.

Mean trainee questionnaire scores (Perceived Competence and four subscales, scaled to a 0-10 range) as well as mean patient, family, and clinician questionnaire scores (QOC-GEN, QOC-EOL, and four subscales) are shown in Table 2.

Table 2.

Competence and Communication Scores, All Participant Groups

	Trainees		Patients		Families		Clinicians
	Valid n	Mean ^a (SD)	Valid n	Mean (SD)	Valid n	Mean (SD)	Valid n	Mean (SD)
Perceived competence	143	5.2 (1.7)
QOC-EOL^b			331	4.2 (3.3)	165	4.5 (3.6)	291	6.9 (2.1)
QOC-GEN^c			348	8.5 (1.9)	174	8.6 (1.7)	321	7.5 (1.8)
Bad News discussions	143	6.1 (1.7)	358	7.8 (2.0)	178	7.9 (1.9)	321	7.2 (1.9)
Treatment discussions	143	4.6 (2.3)	360	7.3 (2.1)	178	7.7 (2.0)	315	7.4 (1.9)
End-of-Life discussions	143	4.9 (2.1)	359	5.7 (2.6)	177	6.2 (2.5)	321	7.3 (1.9)
Symptoms discussions	142	5.7 (2.5)	321	8.1 (2.1)	161	8.4 (1.9)	296	7.7 (1.9)

The trainee-perceived competence predictor was rescaled to 0–10 to make it comparable in range to the evaluators' scores.

QOC-EOL, Quality of Communication Questionnaire, End-of-life care.

QOC-GEN, Quality of Communication Questionnaire, General communication.

Association between self-assessed competence and patient/family/clinician ratings: Standard Quality of Communication subscales

Regression models, adjusted for trainee postgraduate year and racial-ethnic group membership, did not show any significant associations between the trainee Perceived Competence score and either of the primary outcome measures (Table 3). Figure 1 demonstrates the absence of any association between trainee Perceived Competence and patient QOC-EOL scores.

FIG. 1.

Scatterplot showing association between trainee perceived competence and patient assessment of quality of end-of-life communication. EOL, end of life; QOC, quality of communication.

Table 3.

Associations ^a between Trainee's Perceived Competence in Discussing End-of-life Care and Assessments of Communication Skills by Patients, Family Members, and Clinician-Evaluators

	Patients				Families				Clinician-Evaluators
	n ^b	β	R ² incr ^c	P	n ^b	β	R ² incr ^c	P	n ^b	β	R ² incr ^c	P
Primary outcomes^d
QOC-EOL^e	121	−0.02	<0.01	0.87	83	−0.31	0.04	0.10	94	0.04	<0.01	0.72
QOC-GEN^f	122	0.01	<0.01	0.86	83	−0.21	0.06	0.07	96	0.03	<0.01	0.77
Secondary outcomes
Bad News discussions^g	123	0.03	<0.01	0.73	83	−0.15	0.03	0.08	96	0.02	<0.01	0.83
Treatment discussions^h	123	−0.11	0.02	0.06	83	-0.20	0.09	<0.01	96	0.05	0.01	0.40
End-of-Life discussionsⁱ	123	0.08	<0.01	0.21	83	−0.06	<0.01	0.62	100	0.04	<0.01	0.51
Symptoms discussions^j	120	0.03	<0.01	0.65	79	−0.08	0.02	0.24	99	0.01	<0.01	0.91

Parameter estimates and P-values based on robust linear regression models with a single predictor of interest, adjusted for confounders; all models were adjusted for trainee postgraduate year.

Number of trainees with valid data on both the aggregated outcome and the predictor/covariates.

Increment in R² obtained when adding the predictor of interest to a model that included only the covariates.

Predictor for primary analyses: trainee's score on the 17-item perceived competence scale rescaled to 0–10 range.

QOC-EOL: End-of-life care subscale from Quality of Communication questionnaire. Patient, clinician models adjusted for external evaluators' minority racial-ethnic status.

QOC-GEN: General communication subscale from Quality of Communication questionnaire. Patient, clinician models adjusted for external evaluators' minority racial-ethnic status.

Predictor for bad-news-discussion evaluation: trainee's score on the bad-news subscale of the competence rating. Clinician models adjusted for clinician-evaluators' minority racial-ethnic status.

Predictor for treatment-discussion evaluation: trainee's score on the treatment subscale of the competence rating. Patient, clinician models adjusted for external evaluators' racial-ethnic minority group membership; patient model adjusted for patient sex.

Predictor for end-of-life discussion evaluation: trainee's score on the end-of-life subscale of the competence rating. Patient model adjusted for patients' minority racial-ethnic status; family model adjusted for trainee sex, trainee minority racial-ethnic status, family-evaluators' sex.

Predictor for symptom-discussion evaluation: trainee's score on the symptom subscale of the competence rating. Patient model adjusted for minority racial-ethnic status; family model adjusted for family-evaluators sex; clinician model adjusted for trainee sex, clinician-evaluators' sex.

Association between self-assessed competence and patient/family/clinician ratings: Topic-focused subscales

Of the 12 adjusted analyses of subscale assessments made between (1) trainee and (2) patient, family, or clinician-evaluator, only one demonstrated significant associations (Table 3). On the Treatment Discussions subscales, trainee self-assessments were significantly–but negatively–associated with family assessments (p<0.01), suggesting that higher self-assessments were associated with lower family assessments. These trainee self-assessments of skill at conducting treatment discussions accounted for 9% of the variance in family assessments.

Association between patient, family, and clinician-evaluator ratings

Seven of the 18 correlations between evaluators were significant and positive (p<0.05) (Table 4) including: (1) five of six associations between patient and family member evaluations and (2) two of six associations between patient and clinician-evaluator evaluations. None of the associations between family members and clinician-evaluators were significant.

Table 4.

Correlations between Aggregated Evaluations from Three Groups of External Evaluators

	Patient-Family			Patient-Clinician			Family-Clinician
Evaluation	n	ρ^a	P	n	ρ^a	P	n	ρ^a	P
QOC-EOL	78	0.31	<0.01	80	0.11	0.35	56	−0.25	0.06
QOC-GEN	77	0.37	<0.01	83	0.16	0.15	57	−0.10	0.48
Bad News discussions	78	0.34	<0.01	83	0.19	0.08	57	0.01	0.95
Treatment discussions	78	0.31	<0.01	83	0.14	0.22	57	−0.20	0.14
End-of-Life discussions	78	0.29	0.01	83	0.30	<0.01	57	−0.02	0.89
Symptoms discussions	74	0.19	0.11	83	0.22	<0.05	54	0.03	0.82

Spearman's rho.

Discussion

In this two-center observational study of internal medicine trainees, we found no relationship between standard measures of trainees' perceived competence and patient, family, or clinician-evaluator assessments of the quality of their communication about end-of-life care. In additional analyses comparing trainee perceived competence on topic-focused subscales, we also found no positive associations between trainee self-perception and patient, family, or clinician-evaluator assessments. By contrast, we found some significant associations between patient and family surveys and patient and clinician-evaluators surveys suggesting that for some raters, there are small but positive associations. Because trainees and evaluators used different scales to assess communication, we were unable to determine whether self-evaluations are systematically more or less positive than external evaluations.

This is the first published study examining the association between health care providers' self-assessment of competency in communication about end-of-life care and patient- and family-centered outcomes. This finding is consistent with prior studies showing weak or absent associations between clinician self-assessments and external evaluations in fields of pediatric hospice care,¹⁶ ICU care,¹⁷ prevention-oriented counseling,¹⁸ and end-of-life care practices,¹⁹ though none of these prior studies used patient or family assessments as external measurements. Our results are also consistent with a 2006 systematic review of physician self-assessment studies that found physicians to have limited ability to accurately self-assess their clinical skills.²⁰

Improving the quality of communication about end-of-life care has been declared a priority by the Institute of Medicine¹ and the Accreditation Council for Graduate Medical Education.² Communication about end-of-life care encompasses diverse aspects of care, including giving bad news, advance care planning, discussing treatment options, and assessing and managing symptoms. Communication is central in all conceptualizations of quality of end-of-life care, including those provided by the Institute of Medicine¹ as well as those derived from qualitative studies of patients and families.^21–24 However, studies assessing physician communication about end-of-life care have found substantial shortcomings, including communication about prognosis,^25,26 advance care planning,^27–29] and shared decision making.³⁰ Efforts to improve physician communication about end-of-life care require reliable, valid, and responsive measures of the quality of that communication.

An important question raised by our study is whether the lack of correlation between self-assessment of communication skills and ratings by patients, families, and clinician-evaluators may be due to inadequacies of the communication measurement instruments. The Perceived Competence scale¹⁵ has shown evidence of content and construct validity and increases with year of training and experience with end-of-life care. The QOC scale was developed through qualitative studies of patients, family members, and clinicians,^13,31,32 and has been shown to have good measurement characteristics and construct validity, correlating with ratings of overall quality of care, number of discussions with physicians about end-of-life care, physician's awareness of patient's end-of-life preferences, and type and amount of life support discussions.³² In addition, the QOC has been shown to be responsive to a communication intervention in a recent randomized trial.³³ It remains possible that more precise measures of self-assessment or ratings by others would identify a significant correlation. In the absence of better survey-based instruments for measuring the quality of end-of-life communication than those used in this study, our findings identify an important shortcoming in the use of self-assessments of competence in communication.

Prior interventions to improve end-of-life communication have used clinician self-assessment as an outcome measure.^7–9 Our findings call into question the significance of interventions that measure efficacy by clinician self-assessment, as these do not seem to correlate with patient- and family-centered outcomes. Although self-assessments may have value for other purposes, they should not be seen as a replacement for patient and family assessments. Future efforts to improve the quality of communication about end-of-life care should consider outcomes other than physician self-assessment to determine intervention success.

Our study has several important limitations in addition to the question of accuracy of the communication measures. First, it is possible that our inability to identify associations between trainees' self-assessments and ratings of external evaluators is a function of differences in both the precise questions and the response options used to construct the scores for the two groups. Unfortunately, a validated measure of communication about end-of-life care for both self-assessment and external assessment is not available. Further, these differences between the self- and external-ratings made it impossible to reliably assess whether trainee evaluations were systematically higher or lower than external evaluators. Second, lack of correlation in an association study may reflect a true lack of association between the measured qualities, or it may reflect inadequate sample size. Although sample size may have limited our power to detect small associations, our sample is larger than those used in prior interventions with self-assessment as an outcome measure.^7–9 Moreover, if the “true” association between self-assessment and external assessments is so subtle that it requires a larger sample size than ours to detect, its clinical significance is likely modest. Third, in order to ensure the feasibility of securing evaluations of trainee communication skills, we collected assessments during routine clinical practice and teaching rather than following a specific communication event. It is possible that if trainees and evaluators assessed a single common communication event, we may have found an association. Fourth, the trainee status of physicians may limit the generalizability of our findings by lowering the accuracy of self-assessment, though this limitation is a speculative one. The relationship between stage of training/practice and accuracy of self-assessment is not known. A 2002 literature review actually suggests that older physician age is associated with less accuracy of self-assessment compared to external physician-review of performance.³⁴ Fifth, the extent of each trainee's involvement in each patient's end-of-life care was likely variable, and our ability to detect associations may have been limited by the inclusion of trainees without substantial direct involvement (e.g., conducting a family conference). Finally, the communication skills of participating trainees may not be representative of nonparticipating trainees, which could conceivably introduce selection bias.

Internal medicine trainees' self-assessments were not associated with the assessments of their patients, patients' families, or their clinician-evaluators regarding the quality of communication about end-of-life care. Future efforts to improve communication about end-of-life care should consider outcomes in addition to physician self-assessment to determine intervention success. Further research is needed to develop and validate feasible and responsive measures of the quality of physician communication about end-of-life care.

Footnotes

Author Disclosure Statement

No competing financial interests exist.

Acknowledgments

Funded by a grant from the National Institute of Nursing Research at the National Institutes of Health (NINR R01NR009987).

Appendix

References

Field

, Cassel

. Approaching Death: Improving Care at the End of Life. Washington, DC: National Academy Press, 1997.

Weissman

, Block

. ACGME requirements for end-of-life training in selected residency and fellowship programs: A status report. Acad Med, 2002; 77,4:299–304.

Abbott

, Sago

, Breen

, Abernethy

, Tulsky

. Families looking back: One year after discussion of withdrawal or withholding of life-sustaining support. Crit Care Med, 2001; 29,1:197–201.

Hanson

, Danis

, Garrett

. What is wrong with end-of-life care? Opinions of bereaved family members. J Am Geriatr Soc, 1997; 45,11:1339–1344.

Detering

, Hancock

, Reade

, Silvester

. The impact of advance care planning on end of life care in elderly patients: Randomised controlled trial. BMJ, 2010; 340:c1345.

Wright

, Zhang

, Ray

, Mack

, Trice

, Balboni

, Mitchell

, Jackson

, Block

, Maciejewski

, Prigerson

. Associations between end-of-life discussions, patient mental health, medical care near death, and caregiver bereavement adjustment. JAMA, 2008; 300,14:1665–1673.

Baile

, Kudelka

, Beale

, Glober

, Myers

, Greisinger

, Bast

Jr , Goldstein

, Novack

, Lenzi

. Communication skills training in oncology. Description and preliminary outcomes of workshops on breaking bad news and managing patient reactions to illness. Cancer, 1999; 86,5:887–897.

Yacht

, Suglia

, Orlander

. Evaluating an end-of-life curriculum in a medical residency program. Am J Hosp Palliat Care, 2006; 23,6:439–446.

Montagnini

, Varkey

, Duthie

Jr.

Palliative care education integrated into a geriatrics rotation for resident physicians. J Palliat Med, 2004; 7,5:652–659.

10.

Charlson

, Pompei

, Ales

, MacKenzie

. A new method of classifying prognostic comorbidity in longitudinal studies: Development and validation. J Chronic Dis, 1987; 40,5:373–383.

11.

Dillman

. The design and administration of mail surveys. Ann Rev Sociology, 1991; 17:225–249.

12.

Engelberg

, Downey

, Curtis

. Psychometric characteristics of a quality of communication questionnaire assessing communication about end-of-life care. J Palliat Med, 2006; 9,5:1086–1098.

13.

Curtis

, Wenrich

, Carline

, Shannon

, Ambrozy

, Ramsey

. Understanding physicians' skills at providing end-of-life care perspectives of patients, families, and health care workers. J Gen Intern Med, 2001; 16,1:41–49.

14.

Engelberg

, Downey

, Wenrich

, Carline

, Silvestri

, Dotolo

, Nielsen

, Curtis

. Measuring the quality of end-of-life care. J Pain Symptom Manage, 2010; 39,6:951–971.

15.

Billings

, Curtis

, Engelberg

. Medicine residents' self-perceived competence in end-of-life care. Acad Med, 2009; 84,11:1533–1539.

16.

Amery

, Lapwood

. A study into the educational needs of children's hospice doctors: A descriptive quantitative and qualitative survey. Palliat Med, 2004; 18,8:727–733.

17.

Johnson

, Cujec

. Comparison of self, nurse, and physician assessment of residents rotating through an intensive care unit. Crit Care Med, 1998; 26,11:1811–1816.

18.

Hoppe

, Farquhar

, Henry

, Stoffelmayr

. Residents' attitudes towards and skills in counseling: Using undetected standardized patients. J Gen Intern Med, 1990; 5,5:415–420.

19.

Buss

, Alexander

, Switzer

, Arnold

. Assessing competence of residents to discuss end-of-life issues. J Palliat Med, 2005; 8,2:363–371.

20.

Davis

, Mazmanian

, Fordis

, Van Harrison

, Thorpe

, Perrier

. Accuracy of physician self-assessment compared with observed measures of competence: A systematic review. JAMA, 2006; 296,9:1094–1102.

21.

Steinhauser

, Clipp

, McNeilly

, Christakis

, McIntyre

, Tulsky

. In search of a good death: Observations of patients, families, and providers. Ann Intern Med, 2000; 132:825–832.

22.

Steinhauser

, Christakis

, Clipp

, McNeilly

, McIntyre

, Tulsky

. Factors considered important at the end of life by patients, family, physicians, and other care providers. JAMA, 2000; 284,19:2476–2482.

23.

Singer

, Martin

, Kelner

. Quality end-of-life care: patients' perspectives. JAMA, 1999; 281,2:163–168.

24.

Curtis

, Wenrich

, Carline

, Shannon

, Ambrozy

, Ramsey

. Patients' perspectives on physician skill in end-of-life care: differences between patients with COPD, cancer, and AIDS. Chest, 2002; 122,1:356–362.

25.

Fried

, Bradley

, O'Leary

. Prognosis communication in serious illness: Perceptions of older patients, caregivers, and clinicians. J Am Geriatr Soc, 2003; 51,10:1398–1403.

26.

Christakis

, Lamont

. Extent and determinants of error in doctors' prognoses in terminally ill patients: Prospective cohort study. BMJ, 2000; 320,7233:469–472.

27.

Tulsky

, Chesney

, Lo

. See one, do one, teach one? House staff experience discussing do-not-resuscitate orders. Arch Intern Med, 1996; 156,12:1285–1289.

28.

Tulsky

, Fischer

, Rose

, Arnold

. Opening the black box: How do physicians communicate about advance directives? Ann Intern Med, 1998; 129,6:441–449.

29.

Fischer

, Tulsky

, Rose

, Siminoff

, Arnold

. Patient knowledge and physician predictions of treatment preferences after discussion of advance directives. J Gen Intern Med, 1998; 13,7:447–454.

30.

White

, Braddock

3rd , Bereknyei

, Curtis

. Toward shared decision making at the end of life in intensive care units: Opportunities for improvement. Arch Intern Med, 2007; 167,5:461–467.

31.

Curtis

, Patrick

, Caldwell

, Greenlee

, Collier

. The quality of patient-doctor communication about end-of-life care: A study of patients with advanced AIDS and their primary care clinicians. AIDS, 1999; 13,9:1123–1131.

32.

Curtis

, Engelberg

, Nielsen

, Au

, Patrick

. Patient-physician communication about end-of-life care for patients with severe COPD. Eur Respir J, 2004; 24,2:200–205.

33.

, Udris

, Engelberg

, Diehr

, Bryson

, Reinke

, Curtis

. A randomized trial to improve communication about end-of-life care among patients with COPD. Chest, 2012; 141:726–745.

34.

Eva

. The aging physician: Changes in cognitive processing and their impact on medical practice. Acad Med, 2002; 77,10 Suppl:S1–6.