Hospice and Palliative Medicine: Curriculum Evaluation and Learner Assessment in Medical Education

Abstract

Major efforts have been pursued to improve palliative care education for physicians at all levels of their training. Such changes include the incorporation of palliative care curriculum and guidelines, an established process for competency-based evaluation and certification, faculty development, innovative educational experiences, the improvement of textbooks, and the establishment of accredited palliative medicine fellowships. Hospice and palliative medicine (HPM) has been clearly defined as a subspecialty and a crucial area of medical education. As innovative curricular approaches have become available to educate medical and other interprofessional trainees, this article aims to describe different models and methods applied in curriculum evaluation, tailoring such approaches to the field of palliative medicine. A stepwise process of curriculum development and evaluation is described, focusing on available curriculum evaluation competency-based tools for each level of learners. As HPM evolves and its educational programs grow, curriculum evaluation will provides invaluable feedback to institutions and programs in many ways.

Background

Hospice and Palliative Medicine (HPM) is the subspecialty that provides expert comfort care, particularly at the end of life (EOL), by prioritizing the patient's quality of life and emphasizing symptom and pain management, coordination of care, interdisciplinary team involvement, and patient and family communication.^1,2

Major efforts have been pursued to improve palliative care education for physicians at all levels of their education. Such changes include the incorporation of palliative care curriculum and guidelines, an established process for evaluation/certification/licensure, faculty development, new venues for education, the improvement of textbooks, and the establishment of accredited palliative medicine fellowships.^3–5

The steadily increasing implementation of HPM curriculum in medical schools has been shown to improve medical students' preparedness and knowledge.⁶ Studies have shown that medical school graduates exposed to 20 hours of coursework on palliative and EOL care reported learning palliative care basic objectives and symptom management and assessment.⁷ As expected, bedside teaching and clinical encounters positively contribute to communication skills training in palliative care.⁸ The increase in formal HPM curriculum in medical schools is encouraging; however, the negative perceptions from the “hidden curriculum” (that inferred from behaviors and implicit in medical culture) may impair learning.⁴ Because of its interprofessional nature, HPM education has the potential of educating medical learners on the advantages of interdisciplinary and transdisciplinary teams when caring for chronically and terminally ill patients.⁹

Most recently, medical schools began to provide required EOL instruction for accreditation by the Association of American Medical Colleges (AAMC) and the Liaison Committee on Medical Education (LCME), even at early stages of medical education (pre-clerkship).^10,11 These skills and competencies are to be integrated and evaluated into the medical school curriculum.

At the postgraduate level, primary care residency programs such as internal medicine have incorporated HPM into their curriculum, as mandated by their regulatory organizations several years ago.¹² Weissman and Block reviewed the EOL requirements of postgraduate training programs and found that the programs with most comprehensive EOL content were internal medicine, geriatrics, and neurology; most surgical specialties contained no EOL requirement except ethics.¹³ Palliative care requirements have been identified as important geriatric competency elements for family medicine and internal medicine residents and geriatric fellows.¹⁴ Surgical residency programs have most recently included HPM curricular elements into their required topics.¹⁵

HPM was established as a new subspecialty by the American Board of Medical Specialties (ABMS) in 2006.¹⁶ Several medical specialties boards have core competency requirements in HPM and want their diplomates to have access to subspecialty certification in the HPM field. Recognizing a shared interest in this subspecialty, 10 participating ABMS boards now co-sponsor the subspecialty (anesthesiology, emergency medicine, family medicine, internal medicine, obstetrics and gynecology, pediatrics, physical medicine and rehabilitation, psychiatry and neurology, radiology, and surgery). Simultaneously, the Accreditation Council for Graduate Medical Education (ACGME) began accrediting training programs preparing physicians for subspecialty in the HPM field.¹⁷ As of January 2010, there were 74 active programs offering 181 fellowship positions. A total of 73 programs have been accredited by the ACGME and 3 programs have been accredited by the American Osteopathic Association.¹⁸ The American Academy of Hospice and Palliative Medicine (AAHPM) partnered with ACGME to develop HPM core competencies for fellows. Such competencies have been disseminated to standardize curriculum development and evaluation of HPM fellows as this subspecialty.¹⁹

Pediatric palliative care is also an emerging field. Pediatric residents who have benefited from pediatric palliative care teaching stated that they have learned important skills in pediatric EOL care and that they have enhanced confidence in their ability to care for dying patients and their families.²⁰ Additionally, there have been initiatives such as the Initiative for Pediatric Palliative Care (IPPC), an education and a quality improvement effort aimed at enhancing culturally respectful, family-centered care of children living with life-threatening conditions. This initiative aims to educate interprofessional learners who interact with very ill and dying children.²¹ A group of pediatric palliative care leaders is working on the Pediatric Hospice and Palliative Medicine Competencies Project with the aim of creating a resource for pediatrics specific HPM competencies.²²

In summary, HPM education continues to grow impressively throughout all levels of medical education. As innovative curricular approaches have become available to educate our trainees, this article aims to describe different models and methods applied in curriculum evaluation, tailoring such approaches to our field. We will describe generalities of how to evaluate curriculum, focusing on aspects related to the learner's assessment in palliative care.

General Concept: What Is Curriculum As It Applies to Medical Education?

Harden defines curriculum as a sophisticated blend of educational strategies, course content, learning outcomes, educational experiences, assessment, the educational environment, and the individual student's learning style, personal timetable, and program of work.²³ Furthermore, Harden has introduced the innovative concept of curriculum mapping. Curriculum mapping is the process that describes what is taught (the content, the areas of expertise addressed, and the learning outcomes), how it is taught (the learning resources, the learning opportunities), when it is taught (the timetable, the curriculum sequence), and the measures used to determine whether the student has achieved the expected learning outcomes (assessment or evaluation).

The process of curriculum development and delivery in medical education requires higher levels of organizational skills. First, medical curriculum must have clear learning objectives and criteria.²⁴ Key players such as trainees and faculty must be previously aware of the objectives. Once taught, the trainees' evaluation must be based on what was taught and must be clearly documented. Lastly, trainees should receive feedback in a timely manner and a process should be in place to address the individual needs of trainees.²⁵

General Concept: Defining Curriculum Evaluation

The ultimate evaluation of a medical curriculum seeks evidence for developing practice, setting policy, planning standards for education, and making educational decisions.²⁶ Effective curriculum evaluation begins before the educational experience starts. Medical educators should determine the attitudes, knowledge, and skills that are necessary to bring about the desired behavior. The ultimate challenge is to design/implement medical curriculum and pedagogical strategies in a way that leads medical learners to increase their knowledge and apply it as reflected in clinical performance that enhances the quality of patient care, an outcome at the result level of Kirkpatrick's evaluation model as illustrated in Figure 1.^27,28 Fig. 1 Kirkpatrick described four levels of evaluation in which the complexity of the behavioral change increases as evaluation strategies ascend to each higher level.²⁷

FIG. 1.

The Kirpatrick model's four levels include: (1) learner's reaction to the educational experience or curricular component; (2) learning that refers to changes in attitudes, knowledge, and skills once the curricular experience is delivered; (3) behavior or changes in practice and the application of learning to practice; and (4) results to trigger change at the level of the learner and the organization.²⁷ When this model is adapted to health care education, additional factors are considered such as change in organizational practice as it refers to clinical skills and benefits to both learners and their patients.²⁹

Specific Concept: Common Types of Learner-Focused Curriculum Evaluation Used in Hospice and Palliative Medicine

In 1998, the ACGME and the ABMS initiated the Outcome Project to ensure and improve the quality of graduate medical education for all specialties.³⁰ The project envisions an accreditation process that assesses a program's actual accomplishments through educational outcomes or competencies, rather than just their potential to educate. Competencies are defined as groups of skills, behaviors, or knowledge that are identified as performance standards for a particular job^22,31 In medical practice, competencies are written as real-life skills represented in six areas of skill and knowledge required for effective professional medical practice (patient care, medical knowledge, practice-based learning and improvement, interpersonal and communications skills, professionalism, and systems-based practice). Following this competency-based model, HPM education experts published competencies to evaluate fellows during training.¹⁹ Once competencies are defined, different and multiple methods can be matched to evaluate such competencies and multiple methods for each competency, understanding that direct observation in clinical settings is the typical method of evaluation. Observation is a way of gathering data by watching behavior or events, or by noting physical characteristics in their natural setting.³²

Next, we will exemplify how the Kirkpatrick model and the ACGME competencies can be applied to evaluate the effect of an HPM educational activity in family meetings:

Level 1 Reaction: After a training session on family meetings participants are asked if they like the training. (ACGME competencies: professionalism and practice-based learning)

Level 2 Learning: Participants are evaluated in a formative observed structured clinical encounter (OSCE) setting to assess the degree to which participants acquired the knowledge and skills to effectively lead a family meeting (ACGME competencies: medical knowledge, interpersonal and communication skills).

Level 3 Behavior: Observing participants apply what they learned while leading a real family meeting and goals-of-care discussion, followed by learner reflection on performance and reinforcement of communication principles by the supervising faculty (ACGME competencies: practice-based learning, patient and family care, interpersonal and communication skills).

Level 4 Results: Evaluating the results as reflected in the satisfaction of the patient/family member with the leaner's work after a family conference. At a more macro level there are opportunities to incorporate a quality improvement project to track how behavior changed pre- and posteducational activity in family meetings as reflected in the documentation of family meetings and in patient/family satisfaction surveys (ACGME competencies: systems-based practice, patient and family care, interpersonal and communication skills).

Medical students

Primary palliative care refers to the basic skills and competencies required of all physicians and other health care professionals.³³ The LCME recognized the need for EOL education among medical students. Since the incorporation of primary palliative care principles into medical schools, several studies have shown multiple curriculum delivery methods when teaching palliative and EOL care to medical students. The ideal HPM curriculum for medical students has not been defined; however there is evidence in several studies that exposing students to clinical experiences during their clinical years leads to more significant learning in palliative care because learners get to practice under supervision the skills they learned (Kirkpatrick level 3).^4,34 Common teaching methods included didactic sessions, assigned readings and case reviews, diverse forms of clinical experiences, and journaling to explore reactions.⁶ Learner evaluation methods included different types of surveys, pre and post-tests,³⁵ 360-degree evaluations,³⁶ and comprehensive knowledge testing.³⁷ Specifically for patient encounters, evaluation methods tested (1) Interviewing skills (standardized patients, 360-degree evaluations, OSCE, and checklists); (2) teamwork (checklists, global ratings, and 360-degree evaluations); (3) patient education and counseling (360-degree evaluations, OSCE, checklists, and patient evaluation),¹⁵ and communication skills (unfolding cases).³⁸

However, one study found that 39% of students felt unprepared to address patient and family issues related to dying, and more than 40% of residents reported being unprepared to teach EOL courses to students. The report stated that less than 18% of students and residents had participated in a formal EOL curriculum.³⁹ There is a critical need for the establishment of standardized, required core competencies in palliative medicine and EOL care to ensure proficiency. Efforts to define such competencies have begun, particularly relating to care for elders. The AAMC in partnership with the John A. Hartford Foundation has established competencies in geriatric education, defined as minimum graduating medical student competencies needed to assure competent care to older patients by new interns. Within this set of competencies, palliative care learning outcomes have specifically been described: pain and symptom management; psychological, social, and spiritual needs of patients with advanced illness and their family members; and the importance of interdisciplinary care.⁴⁰

Residents

The ACGME Outcome Project is a long-term initiative to emphasize educational outcomes in the accreditation of residency education programs. ACGME identified six general competencies for residents. Such competencies allow programs to determine how physicians are adequately prepared to practice medicine in the changing health care delivery system.⁴¹ Moreover, the Outcome Project aims to develop tools for programs to use as part of an overall evaluation system. ACGME and the ABMS delivered the Toolbox of assessment methods, which includes descriptions and examples of instruments recommended for use by programs as they assess the outcomes of their educational efforts. The Assessment Toolbox proposes several evaluation methods, including chart reviews, patient assessments, 360-degree evaluations, OSCEs, checklists, simulation, and standard examination, among others.⁴²

EOL training for residents has been recommended by regulatory organizations such as the ACGME and the Residency Review Committee (RRC).³ Some studies have described successful educational experiences tailored to residents with the potential of meeting objectives in virtually all of the ACGME core competencies including patient care, medical knowledge, communication, lifelong learning, systems-based practice, ethics, and professionalism.^36,43 At this time the ACGME competencies are not specific to HPM, although there is a taskforce working on developing HPM competencies for residents.

Fellows

As a recognized subspecialty, HPM has quickly and efficiently developed pathways for board certification, fellowship standards, and fellowship accreditation to comply with regulatory entities such as the ACGME and the ABMS and to adapt to their competency framework. In 2007, Morrison and colleagues¹⁹ published HPM core competencies to evaluate progress as fellows advance through novice and advanced beginner levels, and move toward expert levels of competency.⁴⁴ The HPM competencies are the result of a collaborative effort among field experts, who based their work on the ACGME competency domains. Once developed, the HPM competencies document was widely distributed for feedback and comments. The final competencies document includes patient and family care, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice.¹⁹ Subsequently, competency-based assessment methods were carefully selected for each ACGME domain, and compiled in the HPM Toolkit of assessment methods.⁴⁵ A summary of HPM competency-based assessment methods is shown in Table 1. The development of competencies and assessment methods for HPM fellowships has been crucial in the rapid advancement of this field.

Table 1.

Summary of Proposed Competency Evaluation Methods among Hospice and Palliative Medicine (HPM) Fellowships ⁴⁵ (focused on learner assessment)

HPM competencies	Suggested assessment methods by ACGME domain
Patient and family care	1) Attending physician assessment of fellow
	2) Fellow self-assessment
	3) Chart review
	4) Chart abstraction checklist (pain, psychosocial assessment)
Medical knowledge	1) Multiple choice exam
	2) Attending physician assessment of fellow
Practice-based learning and improvement	1) Attending physician assessment of fellow
	2) Team assessment of fellow
Interpersonal and Communication Skills	1) Attending physician assessment of fellow
	2) Team/peer assessment of fellow
	3) Fellow self-assessment
Professionalism	1) Attending physician assessment of fellow
	2) Team/peer assessment of fellow
	3) Patient and family assessment of fellow
	4) Reflecting journaling self-care exercises
Systems-based practice	1) Attending physician assessment of fellow
	2) Team assessment of fellow
Multidomain tools	1) 360° evaluation
	2) Academic portfolio
	3) Palliative medicine structured portfolio
	4) Master assessment table

Toolkit of Assessment Methods from www.aahpm.org/fellowship/default/competencies.html

Note: These assessment toolkit instruments have for the most part not been psychometrically assessed for validity or reliability.

General Concept: Interprofessional Education and Curriculum Evaluation

The delivery of palliative care can be best accomplished in a collaborative environment integrating a bio-psychosocial-spiritual model of interprofessional team members from different fields such as medical, nursing, and psychology. HPM training of interprofessional learners is a natural way to begin collaboration from the early stages of their careers.^9,46 Interprofessional education occurs during those occasions when members (or students) of two or more professions learn with, from, and about one another to improve collaboration and the quality of care.⁴⁷

Interprofessional education aims to educate other colleagues and patients about one's own professional role, responsibilities, and the general scope of practice. It is a true example of collaboration among health professions to establish common goals, provide care for patients and their caregivers, and facilitate shared decision making.⁴⁸ Interprofessional education has been widely used in different settings.

Most of the evaluation methods used focus on measuring reactions to the innovative concept of interprofessional education,⁴⁷ change in attitudes and qualitative knowledge, and skills improvement.⁴⁹ The use of interprofessional education has showed positive outcomes in service delivery or patient care⁵⁰ and the development of quality improvement initiatives.⁵¹

Interprofessional education in palliative care is advancing; however, those advances are outside the scope of this article.

A systematic review of interprofessional education curricula found that Kirkpatrick's model was useful²⁷ when analyzing evaluation outcomes.⁴⁷ Table 2 summarizes levels and outcomes of interprofessional curriculum evaluation, with a specific example related to HPM education (learning to conduct a family conference to establish goals of care).

Table 2.

Levels and Outcomes of Interprofessional Curriculum Evaluation

Levels	Description	Interprofessional outcomes	Examples of HPM interprofessional education
Level 1	Learners react and communicate about the interprofessional nature of learning experience	Changes in perception and attitude and knowledge and skills	An algorithm to conduct a family conference as part of a team, to establish patient's goals of care
Level 2a	Modification of perceptions and attitudes	Changes in perception and attitude and knowledge and skills	Discussion about who is the leader, how to address the family, what is important to discuss
Level 2b	Changes in reciprocal attitudes or perceptions between participant groups. Changes in perception or attitudes toward the value and/or use of team approaches to caring for a specific patient/unit	Changes in perception and attitude and knowledge and skillsSelf-reported perceptions of changes	Team-based learning for better understanding and practice with self-reports (learners teaching learners)
Level 3	Acquisition of knowledge and skills including knowledge and skills linked to interprofessional collaboration	Changes in perception and attitude and knowledge and skillsQualitative observations to report improvement	Role-play of how roles would interact during a family conference
Level 4a	Change in organizational practiceWider changes in the organization and delivery of care.	Establishment of interprofessional quality improvement initiativesImprovement of team effectiveness (sharing information, listening, demonstrating flexibility and providing feedback)	Conducting a real family conference using a transdisciplinary approach
Level 4b	Benefits to patients/clientsImprovements in health or well-being of patients/clients	Decreased in morbidity/mortality of patientsImprovement of quality of care	Patients and families benefiting from interprofessional expertise and a united plan of care

Adapted from Hammick et al.⁴⁷ and Kirkpatrick.²⁷

Discussion

Medical education is perpetually reconfiguring itself in response to the needs of constantly changing scientific knowledge, technology, and health care delivery systems.⁵² There is a crucial need to redesign medical training. Nonetheless, curricular reform is a very complicated process.⁵³ Typically, leaders in medical education have chosen to emphasize scientific knowledge over clinical judgment, practical skills, and the development of cardinal virtues such as compassion and integrity.⁵⁴ Palliative care aims to improve the quality of life of patients and their families facing the problems associated with life-threatening illness, through the prevention and relief of suffering.² As the paths to reforming medical education evolve,⁵² more emphasis should be placed on compassionate care, symptom relief, family involvement, and working in interprofessional teams, which are the core principles of hospice and palliative medicine.

An educationally rich palliative care curriculum should include didactic and clinical experiences where learners observe the role-modeling of competent and compassionate palliative care and have supervised experiential opportunities.^5,34 Improvements within individual schools/residencies still largely rely on the presence of an effective palliative medicine champion who combines commitment and vision, leadership skills, education skills, and clinical skills.⁵⁵ To provide meaningful clinical experiences for learners and to mainstream palliative medicine it is crucial to provide faculty development in primary palliative care skills to non-HPM faculty to ensure care for the growing population of complex chronically and seriously ill people.^34,56

As HPM evolves and its educational programs grow, learner-based curricular evaluation provides invaluable feedback to institutions and programs in many ways. First, it is expected that the field will experience drastic curricular changes in a dynamic way. Such dynamic curricular changes have already begun with the establishment of HPM as a subspecialty and the need to educate clinical learners at different levels of training. When evaluating HPM curricula, educators need to look at the “big picture” and map out the different teaching and evaluation activities against established competencies. Using available models of competencies and evaluation tools^19,42,45 is most helpful and very time/cost-effective, avoiding “re-inventing the wheel.” Second, programs should also carefully consider their current strengths as their starting point in building and modifying curriculum, and should subsequently fill the gaps with new educational experiences rather than a total curricular reform.

The success of a program is measured by evaluating some of its outcomes such as learner competency. Such outcomes are greatly determined by the type of clinician educators who deliver the educational intervention. Preparing health care professionals for teaching is crucial to enhance teaching effectiveness.⁵⁷ Studies have shown that outstanding clinical teachers benefit from faculty development experiences to improve their skills, job satisfaction, and most importantly, the quality of the education they deliver, as evaluated by their students. Highly rated learning experiences included experiential learning, direct feedback, effective peer and colleague networking, and innovative educational methods within single interventions.⁵⁷ Lastly, all training programs must report to local and national regulatory entities. Previously described organizations such as the ACGME (residents/fellows) and the LCME (medical students) have sponsored specific competencies and tools to evaluate training outcomes. Programs are judged by their potential to educate when (1) they are compliant with program requirements; (2) they have established objectives and an organized curriculum; and on (3) have evaluation processes in place for learners, teachers, and the program itself. Additionally, all training programs must provide evidence of performance, objective accomplishment, and continuous improvement in their educational processes.⁵⁸

As the field of HPM continues to grow and the available HPM assessment tools are more widely used these tools could be tested for reliability and validity at the fellowship level. The challenge is how to foster organized interinstitutional collaboration to evaluate the tools. Secondary to the early integration of palliative care training focused on medical students and residents, there will be a need to evaluate the impact of such training as primary palliative care skills are developed among these learners.

Footnotes

Acknowledgments

The authors wish to thank Dr. Charles Von Gunten for his thoughtful review and Wendy Kirkpatrick for granting permission to use the Kirkpatrick Four Levels^™ Evaluation Model.

Author Disclosure Statement

No competing financial interests exist.

References

Getpalliativecare.org . Palliative Care Definition. www.getpalliativecare.org/whatis. 2012 January 17.

World Health Organization: Definition of Palliative Care. www.who.int/cancer/palliative/definition/en/. 2011 March 23.

Block

. Medical education in end-of-life care: The status of reform. J Palliat Med, 2002; 5:243–248.

Billings

, Engelberg

, Curtis

, Block

, Sullivan

. Determinants of medical students' perceived preparation to perform end-of-life care, quality of end-of-life care education, and attitudes toward end-of-life care. J Palliat Med, 2010; 13:319–326.

Morrison

. Bringing palliative care to scale in our nation's medical schools. J Palliat Med, 2010; 13:233–234.

Bickel-Swenson

. End-of-life training in U.S. medical schools: A systematic literature review. J Palliat Med, 2007; 10:229–235.

Ross

, Shpritz

, Hull

, Goloubeva

. Long-term evaluation of required coursework in palliative and end-of-life care for medical students. J Palliat Med, 2005; 8:962–974.

Back

, Arnold

, Baile

, Fryer-Edwards

, Alexander

, Barley

, Gooley

, Tulsky

. Efficacy of communication skills training for giving bad news and discussing transitions to palliative care. Arch Intern Med, 2007; 167:453–460.

Otis-Green

, Ferrell

, Spolum

, Uman

, Mullan

, Baird

, Grant

. An overview of the ACE Project-advocating for clinical excellence: Transdisciplinary palliative care education. J Cancer Educ, 2009; 24:120–126.

10.

Association of American Medical Colleges (AAMC) Recommendations For Clinical Skills Curricula For Undergraduate Medical Education. 2008. www.aamc.org/download/163788/data/recommendations_for_preclerkship_skills_education_for_ugme.pdf. 2011 March 22.

11.

Liaison Committee for Medical Education (LCME): Functions and structure of a medical school. Standards for accreditation of medical education programs leading to the M.D. degree. 2010. www.lcme.org/functions2010jun.pdf. 2011 March 22.

12.

American Board of Internal Medicine Committee on Evaluation of Clinical Competence: Caring for the Dying: Identification and Promotion of Physician Competency. Philadelphia: ABIM, 1996.

13.

Weissman

, Block

. ACGME requirements for end-of-life training in selected residency and fellowship programs: A status report. Acad Med, 2002; 77:299–304.

14.

Williams

, Warshaw

, Fabiny

, Lundebjerg Mpa

, Medina-Walpole

, Sauvigne

, Schwartzberg

, Leipzig

. Medicine in the 21st century: Recommended essential geriatrics competencies for internal medicine and family medicine residents. J Grad Med Educ, 2010; 2:373–383.

15.

American Board of Surgery. Specialty of General Surgery Defined. http://home.absurgery.org/default.jsp?aboutsurgerydefined. 2011 March 22.

16.

American Board Of Medical Specialties: ABMS Establishes New Subspecialty Certificate In Hospice and Palliative Medicine. 2006. www.abms.org/News_and_Events/downloads/NewSubcertPalliativeMed.pdf. 2011 March 22.

17.

ACGME Program Requirements in Hospice and Palliative Medicine. 2008. www.acgme.org/acWebsite/downloads/RCC_progReq/540_hospice_and_palliative_medicine_02122008.pdf. 2011 March 23.

18.

American Academy of Hospice and Palliative Medicine Fellowship Program Directory. http://www.aahpm.org/fellowship/default/fellowshipdirectory.html. 2011 March 22.

19.

Morrison

, Scott

, Block

. American Board of Hospice Palliative Medicine Competencies Work Group. Developing initial competency-based outcomes for the hospice and palliative medicine subspecialist: Phase I of the Hospice and Palliative Medicine Competencies Project. J Palliat Med, 2007; 10:313–330.

20.

Schiffman

, Chamberlain

, Palmer

, Contro

, Sourkes

, Sectish

. Introduction of a pediatric palliative care curriculum for pediatric residents. J Palliat Med, 2008; 11:164–170.

21.

Browning

, Solomon

. The initiative for pediatric palliative care: An interdisciplinary educational approach for healthcare professionals. J Pediatr Nurs, 2005; 20:326–334.

22.

American Academy of Pediatrics Section on Hospice and Palliative Medicine: Chair Newsletter. 2011. www.aap.org/sections/palliative/chairletter.html. 2011 September 15.

23.

Harden

. AMEE Guide No. 21: Curriculum mapping: a tool for transparent and authentic teaching and learning. Med Teach, 2001; 23:123–137.

24.

Bell

. How should the ACGME core competencies be measured? Acad Med, 2009; 84:1173.

25.

Kreutzer

. Competency-based evaluation (and competency in evaluation!) AAHPM Fellowship Summit: Chicago, Illinois, 2007.

26.

Tavakol

, Gruppen

, Torabi

. Using evaluation research to improve medical education. Clin Teach, 2010; 7:192–196.

27.

The Kirkpatrick Four Levels^™ Evaluation Model. 1950. www.kirkpatrickpartners.com. 2011 March 22.

28.

Hutchinson

. Evaluating and researching the effectiveness of educational interventions. BMJ, 1999; 318:1267–1269.

29.

Freeth

, Hammick

, Koppel

, Reeves

, Barr

. A critical review of evaluations of interprofessional education. 2002. www.health.heacademy.ac.uk/publications/occasionalpaper/occasionalpaper02.pdf. 2011 March 22.

30.

Accreditation Council for Graduate Medical Education: The Outcome Project. www.acgme.org/outcome/. 2011 March 22.

31.

Epstein

, Hundert

. Defining and assessing professional competence. JAMA, 2002; 287:226–235.

32.

Taylor-Powell

, Steele

. Collecting evaluation data: Direct observation. University of Wisconsin Cooperative Extension. 1996. http://learningstore.uwex.edu/assets/pdfs/G3658-5.pdf. 2011 March 22.

33.

Von Gunten

. Secondary and tertiary palliative care in US hospitals. JAMA, 2002; 287:875–881.

34.

Weissman

, Quill

, Block

. Missed opportunities in medical student education. J Palliat Med, 2010; 13:489–490.

35.

Porter-Williamson

, von Gunten

, Garman

, Herbst

, Bluestein

, Evans

. Improving knowledge in palliative medicine with a required hospice rotation for third-year medical students. Acad Med, 2004; 79:777–782.

36.

Liao

, Amin

, Rucker

. An innovative, longitudinal program to teach residents about end-of-life care. Acad Med, 2004; 79:752–757.

37.

Ross

, Keay

, Timmel

, Alexander

, Dignon

, O'Mara

, O'Brien

3rd . Required training in hospice and palliative care at the University of Maryland School of Medicine. J Cancer Educ, 1999; 14:132–136.

38.

Wittenberg-Lyles

, Goldsmith

, Ragan

, Sanchez-Reilly

. Medical students' views and ideas about palliative care communication training. Am J Hosp Palliat Care, 2010; 27:38–49.

39.

Sullivan

, Lakoma

, Block

. The status of medical education in end-of-life care: A national report. J Gen Intern Med, 2003; 18:685–695.

40.

Leipzig

, Granville

, Simpson

, Anderson

, Sauvigne

, Soriano

. Keeping granny safe on July 1: A consensus on minimum geriatrics competencies for graduating medical students. Acad Med, 2009; 84:604–610.

41.

ACGME Common Program Requirements: General Competencies. 2007. www.acgme.org/outcome/comp/GeneralCompetenciesStandards21307.pdf.. 2011 March 22.

42.

ACGME Outcome Project, American Board of Medical Specialties (ABMS). Toolbox of Assessment Methods. 2000. www.acgme.org/Outcome/assess/Toolbox.pdf[Last accessed.]

43.

Weissman

, Ambuel

, von Gunten

, Block

, Warm

, Hallenbeck

, Milch

, Brasel

, Mullan

. Outcomes from a national multispecialty palliative care curriculum development project. J Palliat Med, 2007; 10:408–419.

44.

Batalden

, Leach

, Swing

, Dreyfus

. General competencies and accreditation in graduate medical education. Health Aff (Millwood), 2002; 21:103–111.

45.

Hospice and Palliative Medicine Competencies Project. Toolkit of Assessment Methods. 2009. www.aahpm.org/resources/default/comptoolkit.html. 2011 March 22.

46.

Gelfand

, Baker

, Cooney

. Developing end-of-life interdisciplinary programs in universitywide settings. Am J Hosp Palliat Care, 2003; 20:201–204.

47.

Hammick

, Freeth

, Koppel

, Reeves

, Barr

. A best evidence systematic review of interprofessional education: BEME Guide no. 9. Med Teach, 2007; 29:735–751.

48.

Salvatori

, Mahoney

, Delottinville

. An interprofessional communication skills lab: A pilot project. Educ Health (Abingdon), 2006; 19:380–384.

49.

Tucker

, Wakefield

, Boggis

, Lawson

, Roberts

, Gooch

. Learning together: Clinical skills teaching for medical and nursing students. Med Educ, 2003; 37:630–637.

50.

Reeves

, Koppel

, Barr

, Freeth

, Hammick

. Twelve tips for undertaking a systematic review. Med Teach, 2002; 24:358–363.

51.

Morey

, Simon

, Jay

, Wears

, Salisbury

, Dukes

, Berns

. Error reduction and performance improvement in the emergency department through formal teamwork training: evaluation results of the MedTeams project. Health Serv Res, 2002; 37:1553–1581.

52.

Cooke

, Irby

, Sullivan

, Ludmerer

. American medical education 100 years after the Flexner report. N Engl J Med, 2006; 355:1339–1344.

53.

Pascoe

, Babbott

, Pye

, Rabinowitz

, Veit

, Wood

. The UME-21 project: Connecting medical education and medical practice. Fam Med, 2004; 36,Suppl:S12–S14.

54.

The Common Wealth Fund Task Force on Academic Health Centers: Training Tomorrow's Doctors: The Medical Education Mission of Academic Health Centers. The Common Wealth Fund. 2002. www.commonwealthfund.org/Content/Publications/Fund-Reports/2002/Apr/Taining-Tomorrows-Doctors–The-Medical-Education-Mission-of-Academic-Health-Centers.aspx. 2011 March 23.

55.

Weissman

, Blust

. Education in palliative care. Clin Geriatr Med, 2005; 21:165–175, ix.

56.

Periyakoil

. Declaration of interdependence: The need for mosaic mentoring in palliative care. J Palliat Med, 2007; 10:1048–1049.

57.

Steinert

, Mann

, Centeno

, Dolmans

, Spencer

, Gelula

, Prideaux

. A systematic review of faculty development initiatives designed to improve teaching effectiveness in medical education: BEME Guide No. 8. Med Teach, 2006; 28:497–526.

58.

Introduction to Competency-based Education Facilitator Manual. ACGME Outcome Project. 2006. www.acgme.org/outcome/e-learn/21M1_FacManual.pdf. 2011 March 22.