Critical Care Anesthesiology Milestones 2.0: A Framework for Competency Assessment in Subspecialty Fellowship

Abstract

Objective

The Accreditation Council for Graduate Medical Education (ACGME) introduced Milestones in 2013 to aid the assessment of residents and fellows within the framework of the six core competencies of practice. Feedback from program directors and trainees, clinical competency committee processes, and milestone reporting identified the need for modification. This study represents a consensus-driven educational quality improvement initiative which describes the development, revision process, and key features of Critical Care Anesthesiology (CCA) Milestones 2.0, and provides lessons learned in adapting competency-based assessment to a one-year subspecialty training program.

Methods

A Critical Care Anesthesiology (CCA) working group was convened by the ACGME in 2021 to revise the content, enhance relevance to training, clarify goals for program leadership, and develop supplemental resources to aid implementation. The group consisted of a diverse group of experienced educators and clinicians from CCA fellowship programs nationwide. A modified Delphi process, stakeholder feedback, Milestones performance data, and public comment were used to iteratively revise Milestones. In addition, a supplemental guide was developed to provide examples and resources to aid the Milestones assessment.

Conclusion

The Milestones 2.0 simplify the use, implementation, and interpretation of this competency framework for program directors, trainees, faculty, and the American Board of Anesthesiology. The resulting CCA Milestones 2.0 provide updated specialty-specific, competency-based behavioral anchors to guide the developmental progression of competencies and outcomes for CCA fellows and facilitate the design of curricula and the advancement of CCA training programs. Lessons learned in the development of the CCA Milestones 2.0 include the importance of simplification, behavioral specificity, and supplemental.

Keywords

graduate medical education milestones 2.0 ACGME evaluation feedback

Introduction

Competency-based medical education (CBME) has continued to evolve since the turn of the century through a collaborative effort by the Accreditation for Graduate Medical Education (ACGME) and the American Board of Medical Specialties (ABMS). Originally designed as the Outcomes Project, the goal was to transition physician education standards from a time-based model to outcomes-focused curricula with greater public accountability for cost and patient-centered care.¹ In 2013, the ACGME developed Milestones (“Milestones 1.0”) to guide trainees and educators in understanding the skills, knowledge, and behaviors expected at different education stages and provide a more transparent path for engagement in competency-based assessment.² The assessment framework of the Milestones is anchored in six core competency domains: patient care, medical knowledge, professionalism, interpersonal and communication skills, practice-based learning and improvement, and systems-based practice. Each Milestone is organized by general competency, sub-competency, developmental progression, and a description of the specific Milestones (Figure 1). In addition, the Milestones are coupled with the requirement that programs utilize Clinical Competency Committees (CCCs) to provide a holistic and longitudinal evaluation of the trainee that generates semi-annual recommendations toward graduation proficiency.³

Figure 1.

Milestone formatting and organization

The educational Milestones for Anesthesiology were first developed and implemented alongside a broad majority of specialties in 2014 through the efforts of the original Anesthesiology Milestones Working Group.⁴ A total of 25 Milestones were established, refined, and utilized until the second version of Milestones in 2021.⁵ Studies of the Milestones were performed through surveys with program directors in addition to longitudinal studies to ensure reliability.^6-9 A study of multiple training programs (emergency medicine, urology, and diagnostic radiology) found that use of the Milestones at specific time periods during residency maintained consistency in ratings. (6) Subspecialty Milestones in Critical Care Anesthesiology (CCA) were developed in 2016 as a joint initiative with the American Board of Anesthesiology (ABA).^10,11 These Milestones maintained the established framework for assessment across the six competency domains in the context of fellow participation in an ACGME-accredited critical care anesthesia (CCA) fellowship program. Before the CCA Milestone Project in 2016, there was no standardized assessment of trainee performance for this subspecialty.^10,11

The CCA Milestone framework for the six core competencies (common to all specialty Milestones) was divided into 17 sub-competencies (specifically for CCA), organized into a 5-level progression to illustrate a trainee’s progress from novice to expert. Based on this framework, the recommended level for a trainee ready for independent practice was 4.^10,11 These Milestones provided a shared tool that enabled trainees, program directors, and faculty to evaluate fellow performance from the beginning of fellowship to its completion.^12-14 They assisted in identifying areas for improvement and aided in developing targeted learning plans to address competency gaps.¹⁵ Implementing CBME via Milestones in CCA represented a novel shift that emphasized the demonstration of specific skills and abilities “as developmental markers on the path to independent practice.” (15, 16) In addition, they reduced the traditional emphasis on length of training as a marker for readiness for graduation.^15,16

In the subsequent years after the publication of the Milestones, the ACGME began a coordinated effort to collect information on Milestone implementation and performance from key stakeholders across various specialties using formal and informal methods (including focus groups, surveys, and even institutional grand rounds).^17,18 These efforts revealed several consistent criticisms of the Milestones, including concerns regarding the excessive number of sub-competencies, the complexity of the language used in describing the Milestones (referred to as “edu-speak” by focus groups), and an excessive amount of content within each set of Milestones. Some suggested that the use of complex educational terminology in the Milestones made it challenging for many faculty to comprehend, making it unwieldy to utilize in practical settings without additional training. Moreover, a case study across pediatric subspecialties revealed that program directors faced difficulties in selecting and interpreting the Milestone levels and matching the sub-competency labels with Milestone descriptions.^18-20 In addition, Research by Edgar et al conducted in preparation for the Milestones revision process illustrated that there were significant inconsistencies in the wording and application of the Milestones across specialties, particularly in competencies not related to patient care or medical knowledge.^17,20 As an example they discovered that across the 26 core ACGME specialties there were more than 230 different ways of describing Professionalism.²¹ Such inconsistency was felt to hinder collaboration between programs and a limitation to faculty development.

Prior studies demonstrated “straight-lining” and PGY-based scoring patterns, suggesting milestone ratings may reflect training level rather than competency This practice raised questions regarding the ability to utilize Milestones 1.0 as a measure of progression through training.

CCA is a one-year fellowship focused on management of complex, critically ill patients. The typical CCA fellowship includes experience in multiple intensive care unit settings, including surgical, medical, cardiac, and neurologic units. The expected proficiency following fellowship includes the ability to manage patients in these diverse intensive care unit settings.

With the time limitations of a one-year fellowship, CCA training programs have a short time period to assess growth and identify trainees that require early intervention. Milestones 1.0 provided an introduction to consistent assessment. The complexity of the Milestones specific to CCA, and limited guidance for both educators and trainees limited the efficacy for CCA. This limitation was noted in feedback to the ACGME, leading to the development of CCA. Milestones 2.0.

The CCA Milestones 2.0 workgroup focused on addressing these issues for the one-year critical care anesthesiology fellowship, with the goal of simplifying the identification and understanding of growth of trainees to prepare for independent clinical practice, with the focus on providing guidance with the limitation of the one-year training period.

Our aim was to review the current Milestones 1.0 framework to identify gaps, and address these with development of Milestones 2.0 to improve clarity, usability, and provide additional resources to enhance their applicability

Methods

In response to these concerns, and consistent with the idea that the Milestones would require revision to remain relevant, the ACGME began developing Milestones 2.0 with the continued goals of enhancing the quality of educational experiences and evaluations for trainees. This was based on user feedback, focus groups, and surveys. Methods included harmonizing non-patient care competencies across specialties, simplifying language, creating volunteer workgroups, and generating supplemental guides with clinical examples to improve usability. A specific objective of Milestones 2.0 was to ensure that each specialty would determine which sub-competencies were required and that each sub-competency would be limited to a maximum of three measurable skills. Each sub-competency is rated on a five-level scale (Level 1 = beginning fellow, Level 2 = advanced beginner, Level 3 = advancing learner, Level 4 = proficient/graduation goal, Level 5 = aspirational expert). The aim was to limit excessive content, minimize cognitive load and make use more practical.¹⁸ We use “developmental trajectories” to describe the pattern of change in a fellow’s performance within an individual sub-competency over time (for example, steady progression from Level 1 → 2 → 3, rapid advancement to Level 4, plateauing at a given level, or regression following insufficient opportunity or remediation). Trajectories may vary by sub-competency for a given fellow (i.e., different rates or directions of change across skills), and recognizing these patterns is intended to guide timely, targeted feedback and individualized learning plans within the one-year fellowship. The ACGME also sought to make the Milestones for subspecialties more applicable to a fellow level learner (as opposed to a resident), with Milestones level 1 reflective of a novice first-year fellow.²²

This was a retrospective evidence-based synthesis from an expert panel aimed at assessing current data on use of training criteria to create specific and actionable changes to current Critical Care Anesthesiology Milestones. We included current guidelines and feedback on both the specific milestones for our specialty in addition to other fellowships, along with data provided to the American College of Graduate Medical Education. Our study adhered to the STROBE checklist.²³

Harmonized Milestones

The ACGME’s analysis of data from Milestones 1.0 revealed variability with the four competencies of interpersonal and communication skills (ICS), practice-based learning and improvement (PBLI), professionalism (PROF), and systems-based practice (SBP) domains across specialties.²¹ It was determined that such variability may create differential expectations of trainees across specialties, complicate faculty use, and make sharing assessment tools difficult. The ACGME thus convened groups of inter-professional experts (physicians, nurses, allied health professionals, and educationalists) to develop “harmonized Milestones” for use across specialties for ICS, PBLI, PROF, and SBP. These groups created a common language for these competencies, focusing on reducing the complexity of individual Milestones and providing behavior-based descriptions and positive language to highlight trainee performance and expectations rather than focusing on deficiencies. These harmonized Milestones were provided to specialty Milestone 2.0 Working Groups as a basis for modification to meet specialty-specific needs.

Development of CCA Milestones 2.0

A CCA Milestones 2.0 workgroup was assembled in May 2021 with an open call for volunteers (Figure 2). The ACGME CCA Milestones workgroup members were selected to achieve diversity in practice environment and region, as well as years of experience and educational background. The author information includes practice affiliation. Experience ranged from 4 years to 24 years post fellowship training. The 6-member CCA Milestones 2.0 workgroup included one appointed representative from the American Board of Anesthesiology and one from the Residency Review Committee. After selection and before the first CCA working group meeting, members were provided ACGME-Milestones 2.0 materials for review. During CCA workgroup meetings, each prior Milestone was reviewed and evaluated for revision. A modified Delphi process was used with predefined consensus thresholds (≥80% agreement) within the work group. Multiple iterative rounds were conducted, based on agreement. The number of rounds ranged from 3-6, based on responses from the workgroup.

Figure 2.

The process of milestone 2.0 development

The CCA Milestones 2.0 working group agreed that the harmonized language was relevant to the practice of CCA.

Supplemental Guide

A common theme of feedback from the evaluation of Milestones 1.0 was difficulty in interpretation and relevance to practice. Hence, the CCA Milestones 2.0 working group developed a Supplemental Guide to provide greater detail and direction for the utilization of the CCA Milestones 2.0 by faculty and program leadership. The Supplemental Guide aims to provide practical illustrations that fulfill the essential prerequisites for specific proficiency levels.²⁴

Public Comment

The preliminary 2.0 version of the CCA Milestones and the Supplemental Guide were made available online by the ACGME for public comment from March 23, 2022, through April 17, 2022, as per the prespecified Milestone 2.0 development plan.¹⁸ The Milestones and Supplemental Guide were posted on the ACGME milestone page, announced in the ACGME weekly e-communication, and sent to program directors and coordinators. Overall, there are over 60 CCA fellowship programs in the United States. Some will accept emergency medicine residents, in addition to anesthesiology residents.

Respondents to the public comment provided feedback on four statements for each Milestone (19 total in the 6 core competencies) using a Likert scale (Strongly Agree, Agree, Disagree, Strongly Disagree). The four statements were: “1. This Milestone set represents a realistic progression of knowledge, skills, and behaviors; 2. This Milestone set discriminates between meaningful levels of competency; 3. I know how to assess this Milestone set effectively; and 4. The Supplemental Guide was a useful resource for understanding this Milestone set.”

Results

A side-by-side comparison of the CCA Milestones 1.0 and CCA Milestones 2.0 illustrates the revisions resulting from the available feedback, workgroup efforts, and public comments (Table 1).

Table 1.

Comparison of Sub-competencies in Milestones 1.0 Vs. Milestones 2.0 for Anesthesiology Critical Care

Milestones 1.0	Milestones 2.0
Patient Care	Patient Care
1. Patient Assessment & Development of a Care Plan	1. Patient Assessment & Development of a Care Plan
2. Crisis Management	2. Crisis Management
3. Procedural Skills/Technical Abilities/Interpretation	3. Procedural Skills/Technical Abilities/Interpretation
4. Management of Respiratory Failure	4. Respiratory Failure & Ventilation Management
5. Palliative Medicine/End-of-Life Care	5. Management of Organ Dysfunction and Shock
Medical Knowledge	Medical Knowledge
1. Pharmacology	1. Pharmacology
2. Medical Knowledge of Critical Care Medicine	2. Pathophysiology of Critical Illness
Systems-based Practice	Systems-based Practice
1. Interprofessional & Transitions of Care	1. Patient Safety & Quality Improvement
2. Incorporation of Patient Safety and Quality Improvement into Clinical Practice	2. System Navigation for Patient-Centered Care
3. Understanding of Health Care Economics – cost awareness and cost-benefit analysis	3. Physician Role in Health Care Systems
Practice-based Learning and Improvement	Practice-based Learning and Improvement
1. Self-directed Learning & Scholarly Activity	1. Evidence-Based & Informed Practice
2. Education of Team Members & Other Health Care Providers	2. Reflective Practice & Commitment to Personal Growth
Professionalism	Professionalism
1. Commitment to Institution, Department, and Colleagues	1. Professional Behavior & Ethical Principles
2. Receiving and Giving Feedback	2. Accountability/Conscientiousness
3. Responsibility to Maintain Personal Emotional, Physical, and Mental Health	3. Well-Being
Interpersonal & Communication Skills	Interpersonal & Communication Skills
1. Communication with Patients and Families	1. Patient-and Family-Centered Communication
	2. Complex Communication Around Serious Illness
	3. Interprofessional & Team Communication
	4. Communication within Health Care Systems

As can be seen in Table 1, the harmonized language of the ICS, PBLI, PROF, and SBP competencies were incorporated into the CCA Milestones 2.0. The sub competency descriptions were simplified to improve understanding by both educators and trainees. Additionally, a sub competency focused on “complex communication around serious illness” was added to the interpersonal and communication skills (ICS) to address questions raised with Milestones 1.0. This additional sub competency (ICS2) included themes of tailoring communication according to disease characteristics and trajectory, patient consent, patient’s family’s needs, and medical uncertainty. The number of sub-competencies increased from 17 to 19 in the CCA Milestones 2.0. with conceptual groupings within each sub-competency, limited to ≤3 domains to reduce cognitive load. This expansion reflects domain refinement (e.g., serious illness communication), rather than increased complexity per Milestone.

The CCA Milestones 2.0 also added assistance in identifying fellows with critical performance deficiencies and differentiates them from others who have not yet had the opportunity to complete specific rotations, which can limit sub-competency assessments. “Level 4’’ was clarified to represent a graduation goal but not specifically a requirement. Additionally, “Level 5’’ was updated to characterize what can be accomplished by a trainee who has shown exceptional expertise in a sub-competency, as opposed to that of an independent practitioner.

Supplemental Guide

The Supplement Guide which aims to provide practical illustrations of the CCA Milestones for specific proficiency levels, was divided into the following components²⁴:

1. Overall Intent: This segment describes the primary aim of the competency domain.

2. Exemplification: Concrete examples are provided for every level to illustrate the tangible applications of sub-competencies.

3. Assessment Models and Tools: A range of assessment methodologies and tools are recommended to evaluate performance within each competency area effectively.

4. Notes and Resources: Relevant additional information, notes, and resources are shared to enhance understanding and facilitate implementation.

Incorporating well-defined behavioral examples and recommended assessment techniques, the Supplemental Guide offers an organized framework for evaluating competencies. Faculty can use the guide to match milestone levels with real-time clinical observations to ensure uniform evaluations across evaluators. In addition, by linking evaluations to actual clinical performance promotes personalized feedback, encourages fellows to become more autonomous over time, and improves the impartiality and usefulness for CCC discussions.

The guide was designed to be a flexible resource that can adapt to differences in training programs as well as the evolving landscape of critical care education. Importantly, it is provided as an editable document, allowing programs to customize it to their specific needs.

Public Comments

A request for public comment was sent to the 60 CCA programs and which collectively include 223 fellowship positions annually in the US.

The public comment timeframe resulted in responses from twelve program directors and coordinators. Most categories had 80% or greater consensus for agreement (Strongly Agree or Agree) with the statements. The themes from public comments included: improved clarity and usability, the value of the provided behavioral examples, the need for better tools to assess well-being and ongoing challenges in interpreting levels. Specific feedback was provided for

Patient Care Milestone 3: Procedural skills received comments about wording of one skill, which was revised.

The Well-Being Milestone within Professionalism received comments regarding the difficulties in assessing this area and how challenging this is for educators to accomplish with current evaluative methods.

Future evaluation and feedback on Milestones 2.0 will take place with the annual program evaluations. This will provide data on the impact and effectiveness of the revised Milestones, and guide future updates.

Comparison of CCM Milestones 2.0 With Other Critical Care Subspecialties Milestones 2.0

Comparison of CCA Milestones 2.0 to other critical care subspecialty Milestones 2.0 reveals similarities in medical knowledge, systems-based practice, professionalism, and interpersonal and communication skills.^11,25-27 (Table 2). The approach to Patient Care competency varies, with surgical critical care outlining multiple specific organ system-based sub competencies dissimilar to the other specialty sub competencies. In contrast, the other specialty sub competencies focus on patient assessments. All of the specialties include procedural sub competencies within the Patient Care competency. SBP, within the critical care specialties’ Milestones 2.0, focuses on safety and quality improvement and the role physicians play in health care systems. The PBLI competency across critical care specialties’ Milestones 2.0 focuses on evidence-based practice, practice reflection, and engagement in personal evolution. Team dynamics and communication about serious illness are sub competencies of ICS shared by all specialties. These similarities were anticipated secondary to the aim to harmonize the Milestones of ICS, PBLI, PROF, and SBP. Health equity was emphasized through the revised competencies emphasizing communication, systems awareness, and bias mitigation.

Table 2.

Critical Care Subspecialty Milestone 2.0 Comparison

Milestone	Critical care anesthesiology	Critical care medicine	Surgical critical care	Pediatric critical care
Patient Care	Patient Assessment & Development of a Care Plan Crisis Management Procedural Skills/Technical Abilities/Interpretation Respiratory Failure & Ventilation Management Management of Organ Dysfunction and Shock	History and Physical Examination Disease Management in Critical Care Pre-Procedure Assessment Procedures (Invasive and Non-Invasive)	Respiratory Failure Nutritional Support Shock/Resuscitation Renal Disorders of Critically Ill Patients Trauma and Thermal Injury Cardiovascular Disorders of Critically Ill Patients Neurologic Disorders of Critically Ill Patients Gastrointestinal (GI) Disorders fo Critically Ill Patients Infectious Diseases of Critically Ill Patients Procedural Competence	History and Physical Exam Organization and Prioritization of Patient Care Patient Management Pre-Procedure Assessment Procedures
Medical Knowledge	Pharmacology Pathophysiology of Critical Illness	Clincial Reasoning Scientific Knowledge of Diseases and Therapeutics	Pathophysiology, Pharmacology, and Therapeutics Prognosis in Critical Care Conditions Clinical Reasoning	Foundational Knowledge Clinical Reasoning
Systems Based Practice	Patient Safety and Quality Improvement System Navigation for Patient-Centered Care Physician Role in Health Care Systems	Patient Safety and Quality Improvement Coordination and Transition of Care Population Health Physician Role in Health Care Systems	Patient Safety and Quality Improvement System Navigation for Patient-Centered Care Community and Population Health Physician Role in Health Care Systems	Patient Safety Quality Improvement System Navigation for Patient Centered Care-Coordination of Care System Navigation for Patient-Centered Care-Transitions in Care Population and Community Health Physician Role in Health Care Systems
Practice-Based Learning and Improvement	Evidence-Based & Informed Practice Reflective Practice & Commitment to Personal Growth	Evidence-Based and Informed Practice Reflective Practice and Commitment to Personal Growth	Evidence-Based and Informed Practice Reflective Practice and Commitment to Personal Growht	Evidence-Based and Informed Practice Reflective Practice and Commitment to Personal Growth
Professionalism	Professional Behavior & Ethical Principles Accountability/Conscientiousness Well-Being	Professional Behavior and Ethical Principles Accountability Well-Being ad Resiliency	Ethical Principles Professional Behavior and Accountability Self-Awareness and Help-Seeking	Professional Behavior Ethical Principles Accountability/Conscientiousness Well-Being
Interpersonal and Communication Skills	Patient-and Family-Centered Communication Complex Communication Around Serious Illness Interprofessional & Team Communication Communication within Health Care Systems	Patient-and Family-Centered Communication Interprofessional and Team Communication Communication within Health Care Systems Complex Communication Around Serious Illness	Paitent-and Family-Centered Communication Interprofessional and Team Communication Communication within Healthcare Systems	Patient-and Family-Centered Communication Interprofessional and Team Communication Communication within Health Care Systems Communication Around Serious Illness, Including End-of-Life Care

Discussion

In this manuscript, we have described the methodology and considerations of our working group, guided by the ACGME, in revising the Milestones for CCA. In addition, our CCA Milestones 2.0 workgroup created a supplemental guide to better link the Milestones to CCA-specific scenarios to create a shared mental model between stakeholders. Guiding principles employed through the writing process included: 1) defining Milestones such that they could be assessed independently, without overlap with other Milestones; 2) providing a clear developmental progression from one Milestone to the next; 3) basing the Milestones on the unique skills expected of CCAs; and 4) incorporating health equity as a top priority for all stakeholders.

The original Milestones 1.0 framework was a marked paradigm shift from a global, subjective, and normative-referenced fellow assessment to a specific, objective, and criterion-referenced one. Milestones 2.0 maintains this progressive assessment paradigm but simplifies it, improving interpretability and overall subspecialty relevance. These updated Milestones were written with prioritization of their utility to fellows and programs as well as consideration of their documentation burden.

Final versions of the CCM Anesthesiology Milestones 2.0 and Supplemental Guide were implemented in July 2022 following a period of public comment and subsequent revision as described above. The CCA Milestones 2.0 strived to develop simpler and more applicable Milestones, adapting them for the multidisciplinary nature of critical care and the rapidly changing clinical aspects of care in our specialty. There was a deliberate effort to remove education-related technical jargon in addition to using simpler language, uniformity of positive language, and a reduced number of sub-competencies to aid CCA faculty educators in assessing fellows in training. Given the multidisciplinary nature of the specialty, the supplemental guide offers clarification and examples for each milestone as well as guidance for implementation and assessment. The public comments, while limited, indicated that CCA Milestones 2.0 demonstrated promise for improvement compared to the prior version.

The structure of one-year fellowships in comparison to longer fellowships lead to difficulty with accurate assessment and provision of meaningful feedback in order to allow trainees to benefit from these assessments. The time for trainee interactions is limited, and assessment is based on the shortened performance period in one-year fellowships, leading to less time for feedback cycles. With this limited opportunity for assessment, early identification of the need for educational interventions, and resources to guide these processes are an essential need that has been noted in other fellowships including sleep medicine and gastroenterology.^28,29 The Milestones 2.0 have been designed to better address these issues of assessment and remediation if needed., Harmonized Milestones, in addition to active simplification of Milestones language by the Milestones workgroup aimed to address the challenges of both fellowship directors and trainees. As the goal of assessment and feedback is to assess growth, the simplified and easily interpreted Milestones 2.0 are designed to define and support growth instead of simply documenting performance. The CCA Milestone 2.0 workgroup created the supplemental guide to provide useful examples, guide assessors, and include resource lists for each competency.²⁴ With the supplemental guide, the aim is a greater focus on growth in understanding and management of critical care during the training period. It is hoped that the harmonized Milestones and sub-competencies will better allow CCA educators to further training methods and drive education innovations.

The Milestones 2.0 workgroup utilized the feedback received by the ACGME prior to revising Milestones 1.0 to learn what issues existed that required modification. This information with workgroup collaboration led to the recognition of need to simplify the description of the Milestones and limit the sub competencies for a one-year fellowship. The allowed the workgroup to focus on how our recommended changes could aid educators to better understand the growth trajectory of their trainees, and the need to provide the supplemental resources with examples and direction needed.

In summary the process of development of the CCA Milestones 2.0 has resulted in important lessons for those developing frameworks for competency assessment. First is that simplification improves usability as faculty and trainees will be less inclined to use an assessment tool that is cumbersome or difficult to understand. Second specific behavioral anchors enhance feedback as they can be objectively measured. Third supplemental materials that provide examples of the competency assessment are valuable for promoting understanding. Finally, one-year programs require specific attention given the unique time-based assessment challenges and opportunities for remediation.

Barriers to Adoption and Proposed Solutions

Unique challenges exist when applying competency-based training principles, such as Milestones to a 1-year CCA fellowship. First, compared to a multiyear training program, the shorter duration of the CCA fellowship limits the opportunities for assessment, and therefore, a reduced number of data points are available for mapping the trajectory of fellows and providing remediation if needed.

Second, there is considerable variability in critical care exposure during core residency training programs. This is particularly true given the potential for trainees from different backgrounds, including anesthesiology, surgery, emergency medicine, and obstetrics/gynecology, to pursue fellowship training in CCA. As such, CCA programs should not expect all fellows to begin training at the same milestone level. Some fellows may require additional time during early fellowship training to acquire foundational knowledge and skills.

Third, it is essential for CCA programs to understand that milestone data reported to ACGME review committees is used to inform curricular changes and assessment metrics within specialties. The milestone data is not used for accreditation purposes.³⁰ Therefore, it is imperative to assess the fellows honestly over time so as to provide formative feedback to the trainee, as well as, when analyzed, provide opportunities for training improvements at the program level and national levels.

Regular formative feedback is an essential element of a CBME paradigm.³¹ However, critical deficiencies exist in the process of providing feedback to trainees. For example, a recent study found that only a minority of trainees reported receiving regular feedback and that when provided, it was frequently unclear or lacked actionable guidance for improvement.³² Faculty often endorse discomfort with feedback delivery due to lack of training and experience, particularly when feedback involves discussing areas needing improvement³³ Faculty development efforts will be necessary to train and engage CCA faculty for successful competency-based educational training. In Geriatric Medicine, a practical, four-step process has been suggested to engage faculty and fellows and successfully implement Milestones 2.0, which other specialties might adopt: (1) Train faculty in the use of the Milestones 2.0, including providing an overview of the background and updates; (2) Map the Milestones 2.0 to existing assessments; (3) Educate fellows about the Milestones 2.0; and (4) Present and Discuss the Milestones 2.0 at Clinical Competency Committee meetings.³⁴ This structured approach is intended to promote the development of a shared mental model for fellow assessments. While the Milestone 2.0 working group focused on innovations and trends anticipated to affect the practice of CCA in the next 5 to 10 years, it is impossible to predict these changes with certainty. Discoveries in medicine, advances in medical education, and changes in society will necessitate future revisions and additions to the Milestones 2.0. Additional work is needed to demonstrate that the comprehensive assessment picture intended by the Milestones leads to reliable, effective, and safe patient-centered care.

Limitations

This study has important limitations which include the small public comment sample which could lead to potential response bias and limit generalizability. Additionally, there is a lack of outcomes-based validation.

Conclusions

CCA Milestones 2.0 represents the next step in improving readiness assessment for independent practice of CCA fellows. Overall, the CCA Milestones 2.0 are more practical and behaviorally explicit as an educational tool to assess fellows’ progress toward achieving expertise while also helping the trainees identify their strengths and areas for continued growth. Although Milestones 2.0 CCA is on the right path toward competency-based assessment training, continued evaluation is needed to determine impact on trainee development and patient outcomes.

Supplemental Material

Supplemental material - Critical Care Anesthesiology Milestones 2.0: A Framework for Competency Assessment in Subspecialty Fellowship

Supplemental material for Critical Care Anesthesiology Milestones 2.0: A Framework for Competency Assessment in Subspecialty Fellowship by Edward A. Bittner, Maurice F. Joyce, Charles R. Sims, Babar Fiza, Gretchen A. Lemmink, Laura Edgar, EDd, CAE, Bhargavi Gali in Journal of Medical Education and Curricular Development

Footnotes

Acknowledgements

The authors thank their co-members of the Critical Care Anesthesiology Milestones 2.0 Work Group and the Accreditation Council for Graduate Medical Education for its extensive support of their committee work.

ORCID iD

Bhargavi Gali

Author Contributions

EA, MJ, CS, BF, GA, LE, BG all participated in the Critical Care Anesthesiology Milestones 2.0 group, planned the manuscript, and contributed to writing the manuscript. All authors read and approved the final manuscript.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Declaration of Conflicting Interests

The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Bhargavi Gali: Data monitoring committee member for Takeda.

Supplemental Material

Supplemental material for this article is available online.

References

Frank

Snell

Cate

, et al. Competency-based medical education: theory to practice. Medical Teacher. 2010;32(8):638-645. doi:10.3109/0142159X.2010.501190.

Nasca

Philibert

Brigham

Flynn

. The Next GME Accreditation System — Rationale and Benefits. New England Journal of Medicine. 2012;366(11):1051-1056. doi:10.1056/NEJMsr1200117.

Holmboe

ESL

William

. ACGME Assessment Guidebook. Chicago, IL: World wide web. ACGME; 2020. Accessed 12/12/2022.

Schartel

Kuhn

Culley

Wood

Cohen

Group

AMW

. Development of the anesthesiology educational milestones. J Grad Med Educ. 2014;6(1s1):12-14.

Ambardekar

Walker

McKenzie-Brown

, et al. The Anesthesiology Milestones 2.0: An Improved Competency-Based Assessment for Residency Training. Anesthesia & Analgesia. 2021;133(2):353-361. doi:10.1213/ane.0000000000005499.

Park

Hamstra

Yamazaki

Holmboe

. Longitudinal Reliability of Milestones-Based Learning Trajectories in Family Medicine Residents. JAMA Netw Open. 2021;4(12):e2137179. doi:10.1001/jamanetworkopen.2021.37179.

Ross

Metro

Beaman

, et al. A first look at the Accreditation Council for Graduate Medical Education anesthesiology milestones: implementation of self-evaluation in a large residency program. J Clin Anesth. 2016;32:17-24. doi:10.1016/j.jclinane.2015.12.026.

Sun

Park

Chen

Hogan

Tanaka

. Longitudinal Reliability of Milestones Learning Trajectories during Anesthesiology Residency. Anesthesiology. 2025;142(5):918-928. doi:10.1097/aln.0000000000005388.

Hamstra

Yamazaki

Barton

Santen

Beeson

Holmboe

. A National Study of Longitudinal Consistency in ACGME Milestone Ratings by Clinical Competency Committees: Exploring an Aspect of Validity in the Assessment of Residents' Competence. Acad Med. 2019;94(10):1522-1531. doi:10.1097/acm.0000000000002820.

10.

The Critical Care Anesthesiology Milestone Project . A Joint Initiative of The Accreditation Council for Graduate Medical Education & The American Board of Anesthesiology. 2016. 2016. https://www.med.upenn.edu/anesthesia-critical-care/assets/user-content/documents/CriticalCareAnesthesiologyMilestones_Jan2016.pdf. Accessed 10 May 2023.

11.

ACGME . Critical Care Anesthesiology Milestones. 2022. https://www.acgme.org/globalassets/pdfs/milestones/criticalcareanesthesiologymilestones.pdf

12.

Camac

Stewart

Santhosh

Carlos

Denson

Heath

. Best Practices for Remediation in Pulmonary and Critical Care Medicine Fellowship Training. ATS Scholar. 2022;3(3):485-500. doi:10.34197/ats-scholar.2022-0007RE.

13.

Frank

Koenig

Srivastava

. Connecting milestones to observable clinical performance through standardized pediatric cardiology rotation evaluations: Report and reflections on a novel pilot project. Progress in Pediatric Cardiology. 2017/03/01/. 2017;44:11-15. doi:10.1016/j.ppedcard.2016.12.005.

14.

Yuan

Prince

Oliver

3rd Abbott

Nee

. Implementation of nephrology subspecialty curricular milestones. Am J Kidney Dis 2015;66(1):15-22. doi:10.1053/j.ajkd.2015.01.020.

15.

ACGME. Milestones Guidebook for Residents and Fellows . Accreditation Council for Graduate Medical Education. 2020. 2020. Accessed Mayo 10, 2023. https://www.acgme.org/globalassets/pdfs/milestones/milestonesguidebookforresidentsfellows.pdf.

16.

Colbert

Dannefer

French

. Clinical Competency Committees and Assessment: Changing the Conversation in Graduate Medical Education. J Grad Med Educ. 2015;7(2):162-165. doi:10.4300/jgme-d-14-00448.1.

17.

Edgar

Roberts

Holmboe

. Milestones 2.0: A Step Forward. J Grad Med Educ. 2018;10(3):367-369. doi:10.4300/jgme-d-18-00372.1.

18.

McLean

Edgar

Harsy

. The Milestones 2.0 Development Process. J Grad Med Educ. 2021;13(2 Suppl):4-7. doi:10.4300/jgme-d-20-00909.1.

19.

Current milestones assessment practices, needs, and challenges of program directors: a collective case study in a pediatric hospital setting. Cureus. 2021;13(14585) doi:10.7759.

20.

Berlin

KEK

Gray

Myers

, et al. A new era of assessment for neonatal-perinatal medicine trainees: milestones 2.0. Journal of Perinatology. 2023;43:1506-1512. doi:10.1038/s41372-023-01683-8.

21.

Edgar

Roberts

Yaghmour

, et al. Competency Crosswalk: A Multispecialty Review of the Accreditation Council for Graduate Medical Education Milestones Across Four Competency Domains. Acad Med. 2018;93(7):1035-1041. doi:10.1097/acm.0000000000002059.

22.

Sawyer

Gray

Chabra

, et al. Milestone Level Changes From Residency to Fellowship: A Multicenter Cohort Study. J Grad Med Educ. 2021;13(3):377-384. doi:10.4300/jgme-d-20-00954.1.

23.

von Elm

Altman

Egger

Pocock

Gøtzsche

Vandenbroucke

STROBE Initiative . The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. The Lancet. 2007;370(9596):1453-1457. doi:10.1016/S0140-6736(07)61602-X.

24.

Supplemental Guide . Critical Care Anesthesiology. Accreditation Council for Graduate Medical Education 2022. 2022. chrome-extension://efaidnbmnnnibpcajpcglclefindmkaj/. https://www.acgme.org/globalassets/pdfs/milestones/criticalcareanesthesiologysupplementalguide.pdf. Accessed 9 August 2023.

25.

Adams

Chiu

Daley

, et al. Surgical Critical Care Milestones. 2021. Accessed December 4. 2024. https://www.acgme.org/globalassets/pdfs/milestones/surgicalcriticalcaremilestones.pdf

26.

Boyer

Burns

Czaja

, et al. Pediatric Critical Care Milestones. 2023. Accessed December 4, 2024. https://prep.acgme.org/globalassets/pdfs/milestones/pediatriccriticalcaremilestones2.0.pdf

27.

Addrizzo-Harris

Ashton

Buckley

, et al. Critical Care Medicine Milestones. 2020. Accessed December 4, 2024. https://www.acgme.org/globalassets/pdfs/milestones/criticalcaremedicinemilestones2.0.pdf

28.

Donnangelo

Shah

. What Faculty and Fellows Should Know About Milestones 2.0. Clinical Gastroenterology and Hepatology. 2022;20(4):720-722. doi:10.1016/j.cgh.2021.12.017.

29.

Dredla

Edgar

Samman

, et al. Sleep Medicine Milestones 2.0: designed for our field. Journal of Clinical Sleep Medicine. 2021;17(3):499-503. doi:10.5664/jcsm.8962.

30.

Use of Individual Milestones Data by External Entities for High-Stakes Decisions – A Function for Which they Are not Designed or Intended American College of Graduate Medical Education; 2022. https://www.acgme.org/globalassets/pdfs/milestones/useofindividualmilestonesdatabyexternalentitiesforhighstakesdecisions.pdf

31.

Burgess

van Diggele

Roberts

Mellis

. Feedback in the clinical setting. BMC Medical Education. 2020;20(2):460. doi:10.1186/s12909-020-02280-5.

32.

Shafian

Ilaghi

Shahsavani

, et al. The feedback dilemma in medical education: insights from medical residents’ perspectives. BMC Medical Education. 2024;24(1):424. doi:10.1186/s12909-024-05398-y.

33.

McCutcheon

Duchemin

. Faculty Self-Evaluation of Experiences with Delivering Feedback to Trainees Across Academic Ranks. Med Sci Educ. 2021;31(2):355-358. doi:10.1007/s40670-020-01196-5.

34.

Streiter

Foley

Bennett

, et al. Looking back, moving forward: A practical guide to implementing the updated ACGME geriatric medicine Milestones 2.0. J Am Geriatr Soc. 2023;71(5):1610-1616. doi:10.1111/jgs.18280.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.18 MB

0.00 MB