Process Evaluation for Improving K12 Program Effectiveness: Case Study of a National Institutes of Health Building Interdisciplinary Research Careers in Women's Health Research Career Development Program

Abstract

Introduction:

Process evaluation is an important tool in quality improvement efforts. This article illustrates how a systematic and continuous evaluation process can be used to improve the quality of faculty career development programs by using the University of Minnesota's Building Interdisciplinary Research Careers in Women's Health (BIRCWH) K12 program as an exemplar.

Methods:

Data from a rigorous process evaluation incorporating quantitative and qualitative measurements were analyzed and reviewed by the BIRCWH program leadership on a regular basis.

Results:

Examples are provided of how this evaluation model and processes were used to improve many aspects of the program, thereby improving scholar, mentor, and advisory committee members' satisfaction and scholar outcomes.

Conclusions:

A rigorous evaluation plan can increase the effectiveness and impact of a research career development plan.

Introduction

Research career development programs are important for developing the necessary knowledge and skills for a junior faculty member to become an independent scientist. To maintain high quality programs that are effective in achieving the goal of preparing independent researchers and team science leaders, a comprehensive program evaluation must consider both scholar outcomes and the specific processes involved in achieving these outcomes. Yet, there is limited literature that provides guidance on how to best evaluate faculty career development programs and how to use the information gained from an evaluation to improve the quality of the program.

Traditional evaluation efforts are narrowly focused on scholar outcomes and can lead to missed opportunities to systematically collect data to inform changes necessary to improve a training program throughout its development and implementation, whereas a focus on continuous program improvement using process evaluation can ultimately maximize scholar outcomes. Process evaluation entails the systematic collection of information to document and assess how a program was implemented and operates.¹ This form of evaluation can be done as a one-time assessment or conducted on a continuous basis. Results help stakeholders see how the program outcomes were achieved, identify the types of problems encountered when delivering the program, and design interventions for improving the program.² Process evaluation differs from outcome evaluation, in that the latter seeks to determine worth or merit of the program by comparing outcomes or measured changes observed between participants and others with similar characteristics who did not participate in the program intervention. Process and outcome evaluations are not mutually exclusive; in fact, process monitoring and evaluation activities often provide foundational data for interpreting outcome findings.

The Centers for Disease Control recommends that a process evaluation design should have a strong, a priori conceptual framework supporting the plan and should include a program logic model to achieve effective integration of evaluation procedures in the proposed program plan.¹ The initial application for the University of Minnesota's (UMN) Building Interdisciplinary Research Careers in Women's Health (BIRCWH) K12 program included a logic model and process evaluation plan that has been implemented throughout 10 years of funding. This article describes the benefits and procedures of process evaluation as a tool for quality improvement of research career development programs by using the UMN BIRCWH program as an exemplar.

Case Study

Program description

The national BIRCWH initiative is a mentored career development program (K12) funded by the Office of Research on Women's Health at the National Institutes of Health (NIH). The program matches junior faculty to senior faculty mentors with shared research interests in women's health and sex differences research.

The goal of the UMN BIRCWH program is to improve the health of all women across the life span and, by extension, to improve the health of their families and communities in Minnesota, the nation, and beyond. To accomplish the program goals, the UMN BIRCWH program simultaneously supports four scholars who seek to develop successful independent research careers in women's health and sex differences research. Like all BIRCWH programs across the country, the UMN BIRCWH program requires each scholar applicant to have a robust research proposal, an interdisciplinary mentoring team, and an individual career development plan that will advance them toward an independent research career. Program benchmarks and scholars' individual career development plans are used to ensure successful progression through the program. The UMN BIRCWH program requires scholars to attend weekly career development seminars, lead a journal club session, participate in leadership and grant-writing workshops, present their research at local and national conferences, complete three online NIH Sex and Gender Difference courses, and attend the Annual BIRCWH Meeting at the NIH. Formal coursework and other short courses may be taken as part of scholars' individual career development plans.

Each scholar works with an interdisciplinary mentoring team comprising a minimum of three mentors who cover content expertise, statistical and/or methodological expertise appropriate to the scholar's research aims, and home department promotion and tenure expectations. Each scholar must meet with their primary mentor weekly and secondary mentors once or twice a month. Scholars are required to meet with their mentoring team as a whole at least twice a year, although the recommendation is to meet three to four times per year. The scholar and their mentoring team meet with the BIRCWH leadership twice a year for a progress review. The UMN BIRCWH program's Interdisciplinary Advisory Committee (IAC) provides oversight of the program, including the selection and annual reappointment of scholars.

Since the initiation of the UMN BIRCWH program, 14 faculty members have received scholar awards. Eleven of the 14 were women and three were men. All but one male scholar continued in academic health science careers. Three of the women were from groups who are under-represented in academic health sciences.

Overview of program evaluation

As a component of our original grant application, we designed a logic model in accordance with the BIRCWH program plan that has guided the implementation and evaluation of the program since its inception (see Fig. 1 for the current version of the logic model). Our evaluation program incorporated quantitative and qualitative measures for continuous monitoring of program satisfaction, quality, and outcomes. Output data from the evaluation are examined twice a year and have been used over a period of 10 years to improve the program (process evaluation) and, thereby, enhance scholar and mentor satisfaction and scholar outcomes (outcome evaluation). The logic model includes our overarching program goal, institutional resources directed to the program (inputs), components and activities, critical products (outputs), and short- and long-term outcome goals. Outputs and outcomes are evaluated with specific evaluation tools. Table 1 ^3

–6 provides a brief description of each evaluation tool and the frequency with which it is administered. Although there were multiple rating scales of varying lengths, our scholars were conscientious about completing the scales. The scholars found the BIRCWH program to be a valuable component of their career development and understood the importance of program evaluation outcome data to support future grant renewal submission.

FIG. 1.

Building Interdisciplinary Research Careers in Women's Health logic model and evaluation tools.

Table 1.

Description of University of Minnesota's Building Interdisciplinary Research Careers in Women's Health Evaluation Tools

Evaluation instruments used	Description	Administered
Confidence survey^a,4 –6	A 92-item questionnaire that assesses scholars' perceptions of their ability to apply research knowledge and skills to their ongoing research. (10-point scale; 0 = no confidence, 10 = total confidence).	Pre- and post-BIRCWH program
Culture survey^a	A 70-item questionnaire that measures scholars' perceptions of research culture, productivity, and support of their department using a five-point Likert scale (1 = strongly disagree; 5 = strongly agree).	Pre- and post- BIRCWH program
Scholar progress report	Completed by scholars to update mentors and leadership on their progress and outcomes, including grants, publications, and presentations.	Semiannual
Seminar evaluation survey	A multi-item questionnaire to evaluate seminar sessions. (length varies depending on the content)	Postspring and fall semesters
Scholars' satisfaction with mentors survey^a	A 17-item questionnaire capturing scholars' satisfaction with each of their mentors.	Semiannual
Mentors' satisfaction with scholars survey^a	A seven-item questionnaire capturing mentors' satisfaction with scholar's progress and ability to accept feedback.	Semiannual
Scholar review sessions	Meeting of scholar and their mentoring team with the BIRCWH leadership to assess progress and future directions.	Semiannual
Scholar exit interviews	Semistructured interview performed by a graduate student who was not involved in the BIRCWH program	Post-BIRCWH program
Interdisciplinary advisory committee meetings	Leadership presents scholar progress and programmatic accomplishments/opportunities for improvement based on program evaluation results.	Semiannual (external members annually)

These tools were adapted from the study by Bland et al.³ and other sources as indicated.

BIRCWH, Building Interdisciplinary Research Careers in Women's Health.

Using the logic model as the basis of our process and outcome evaluation gave us the opportunity to reassess our program resources, activities, outputs, and short- and long-term outcomes on a regular basis. At each evaluation point, the Principal Investigator and Program/Research Director meet to review findings, identify areas for improvement, and determine changes that should be made to the program. In the case study that follows, we give examples of how we used our process evaluation methods to identify problems and substandard program or scholar performance so that the issues could be corrected in a timely manner.

Student's t-test was used to compare baseline and postprogram scores for the Departmental Research Culture Survey and for the Scholar Confidence Survey. The statistical analysis was completed using Microsoft Excel.

Process evaluation

Mentoring program

Semiannually, mentors and scholars individually fill out the Mentors' Satisfaction with Scholar Survey and the Scholars' Satisfaction with Mentors Survey. Based on their feedback and discussions with mentors, we took the following actions to improve the program: (1) designed mentoring agreement forms for mentors and scholars that clearly specified the expectations for their roles; (2) improved our orientation and mentoring workshop by emphasizing the importance and benefits of interdisciplinary mentoring; (3) clarified the value and expected use of individual career development plans; and (4) created a checklist of recommended discussion topics for each mentoring session to help mentors keep scholars on track with meeting program expectations. All changes were well received by mentors and scholars.

Using results of the Mentors' Satisfaction with Scholar Survey combined with discussions at the semiannual scholar review sessions, we found that some scholars felt they did not receive the guidance they desired from their mentors. Thus, we assigned an additional person to some mentoring teams—someone who was either more available or was a better match with the research focus of a scholar's work. We also added a requirement that each mentoring team have a mentor from the scholars' home department as these mentors could best advise the scholars on issues related to department-specific career development procedures, policies, and resources available to meet research, teaching, and professional service obligations as well as promotion and tenure requirements. These adjustments resulted in increased mentorship satisfaction scores from scholars.

We surmise that adding a mentor from the scholar's home department was responsible for the significant improvement in scholars' perceptions of being supported by their department colleagues (p = 0.008) as assessed by the Culture Survey (Table 2). The subscale, colleagues are supportive, addressed the question of whether home department colleagues were available and interested in discussing research work. Overall, scholars reported significantly higher career satisfaction and more support for their research at the end of their BIRCWH training compared with their baseline ratings (Table 2).

Table 2.

Comparison of Pretest and Post-Test Scores for Culture Survey As Reported by Scholars (N = 10)

	Pretest	Post-test
Thematic areas	Mean (SD)	Mean (SD)	Difference in mean scores	t Statistics (df = 9)	p Value
Adequate time for research	3.3 (1.4)	4.6 (0.5)	1.3	2.9	0.008
Adequate resources for research	3.8 (0.5)	3.8(0.6)	0	0.2	0.419
Colleagues are supportive	3.6 (0.7)	4.1(0.6)	0.5	2.2	0.029
Departmental commitment to research	4.3 (0.8)	4.5 (0.6)	0.2	1.4	0.097
Personal commitment to research	4.2 (0.6)	4.8 (0.2)	0.4	3.7	0.002

Survey scores range from 0 (strongly disagree) to 5 (strongly agree).

Total number of items in the survey = 70.

Bold values denote p < 0.05.

SD, standard deviation.

Seminar content and quality

We used results from our Scholar Confidence Survey combined with qualitative and quantitative feedback from our Seminar Evaluation Survey to tailor the topics in a seminar series to meet the scholars' needs. For instance, the data indicated that scholars lacked confidence in funding a research study, planning and managing a research study, and working effectively as a team. After delivering seminar sessions that focused on these topics and through coaching from the mentoring teams, we demonstrated improvements in scholars' confidence scores in these areas (Table 3). In fact, there was significant improvement in all 10 thematic areas of the Confidence Survey (Table 3). Feedback on the Seminar Evaluation Survey also helped us work with seminar speakers on their content and delivery format.

Table 3.

Comparison of Pretest and Post-Test Scores for Confidence Survey As Reported by the Scholars (N = 11)

	Pretest	Post-test
Thematic area	Mean (SD)	Mean (SD)	Difference in mean scores	t Statistics (df = 10)	p Value
Conceptualizing a study	7.5 (1.4)	9.0 (0.7)	1.5	4.2	0.001
Designing a study	7.3 (1.4)	9.0 (0.9)	1.7	4.5	0.001
Collaborating with others	6.9 (1.8)	9.0 (0.9)	2.1	3.7	0.002
Funding a study	6.6 (1.4)	9.1 (1.0)	2.5	6.2	<0.001
Planning and managing research study	6.7 (1.8)	9.0 (0.9)	2.3	4.5	0.001
Responsible conduct of research	7.4 (1.4)	9.1 (0.8)	1.7	4.4	0.001
Collecting, recording, and analyzing data	7.4 (1.7)	9.1 (0.7)	1.7	3.9	0.001
Interpreting data	7.9 (1.5)	9.2 (1.1)	1.3	2.5	0.015
Reporting a study	7.9 (1.3)	9.3 (0.6)	1.4	3.9	0.001
Presenting the study	8.4 (1.4)	9.5 (0.5)	1.1	2.7	0.011

Survey scores range from 0 (no confidence) to 10 (complete confidence) in current ability to perform the task.

Total number of items in the survey = 92.

Bold values denote p < 0.05.

Applicant pool

During a meeting with our IAC, members raised concerns that the size and quality of the applicant pool were not as robust as they should be to ensure a competitive application process. In response, we arranged individual and group meetings with research deans in our Academic Health Center, each Medical School department head, and selected research deans outside the Academic Health Center to request their support in actively recruiting junior faculty for the UMN BIRCWH program. Subsequently, applications for each open slot increased dramatically from a low of three applications per scholar slot to 11 applications per slot.

Interdisciplinary advisory committee

An important component of our scholar outcome evaluation was the annual assessment of scholars by IAC members. Initially, the BIRCWH leadership provided written and oral reports on scholar progress along with oral reports from the two external advisory committee members who met individually with each scholar before our IAC meeting. However, internal IAC members told us that they found it difficult to provide evaluative feedback to scholars whom they did not meet in person. As a result, we implemented scholar presentations at IAC meetings. This resulted in increased engagement of IAC members in assessing scholar progress in research and in career development, which ultimately improved the quality and relevance of their feedback to scholars and the leadership team. After each IAC meeting, we recorded all recommendations; at subsequent meetings, we reported actions taken on their recommendations. IAC members voiced their satisfaction with the process when they subsequently volunteered to assist in semiannual scholar review sessions, mock grant reviews, and to present at seminars, which ultimately enhanced the program.

Scholar transition out of the program

Through our scholar exit interviews, we identified scholars who reported feeling ill-prepared to transition out of the UMN BIRCWH program. They expressed a general lack of confidence in their ability to continue their research careers without the 75% salary support provided by the BIRCWH grant. With that information, we instituted a formal transition planning process for the end of the scholar's second year in the program. We required scholars to develop a plan, discuss it with their mentoring teams, and then present the plan to the BIRCWH leadership team during the semiannual scholar review sessions. In addition, we added a seminar where prior BIRCWH scholars talked about the lessons they learned in transitioning back to regular faculty appointments and maintaining their research portfolios. Last, we have added an exit meeting with the department chair and primary mentor to ensure that the chair is aware of the scholar's progress during the program and supports the transition plan. In addition, new resources and opportunities for continuing the scholar's career development are identified.

Scholar outcomes of the UMN BIRCWH program

Academic productivity

Scholars who have completed the program have been highly productive as illustrated in Table 4. This table shows the cumulative number of grants and publications received by the 14 scholars from the time they entered the BIRCWH program through September 2016. The data are based on biannual data collection from active scholars, using the scholar progress report, and annual review of the current curriculum vitae of past scholars. As of September of 2016, scholars have published 225 peer-reviewed articles and all scholars continued to publish after completing the BIRCWH program. All scholars who remained in academia have obtained NIH funding, with a total of 14 grants where the scholar serves as the principal investigator and 19 grants where the scholars serve as coinvestigators. We have no comparison group to demonstrate that frequent process evaluation has been an important contributing factor to this success. However, data from the process evaluation indicate that the BIRCWH program, as implemented, aligns with the logic model presented in the program plan of the original grant application. Furthermore, we assert that scholars' successes are, at least in part, due to successful implementation of the program and use of this evaluation model.

Table 4.

University of Minnesota's Building Interdisciplinary Research Careers in Women's Health Scholar Productivity

			Research grants awarded
Current position	Appointment period	Total peer-reviewed publications (first author + coauthor)	Total # (NIH, industry, foundation, and other)	Total NIH grants	# PI	# Co-I
Promoted to Associate Professor	2008–2011	28 (16 + 1)	21	10	5 (1 RO1, 2 RO3, 1 R21, 1 R56)	5 (4 RO1, 1 UO1)
Promoted to Associate Professor	2008–2011	22 (3 + 19)	5	3	1 (U54 Pilot)	2 (1 R01, 1 T03)
Promoted to Associate Professor	2008–2011	28 (19 + 9)	16	6	0	6 (1 R01, 1 R18, 1 R21, 1 HRSA, 1 CDC, 1 P20)
Promoted to Clinical Professor	2008–2011	26 (9 + 17)	15	3	2 (1 R01, 1 R03)	1 (NSF)
Assistant Professor	2010–2011	12 (8 + 4)	6	4	0	4 (1 U01, 3 R01)
Promoted to Associate Professor	2010–2013	18 (4 + 14)	15	10	3 (2 R01, 1 R03)	7 (1 R01, 3 U01, 1 CDC contract, 2 NHLBI Contract)
Promoted to Associate Professor	2011–2014	48 (31 + 17)	9	4	2 (1 R03, 1 CMS)	2 (1 CMS, 1 HRSA)
Promoted to Associate Professor	2011–2014	9 (5 + 4)	6	1	1 (R03)	0
No longer in academics	2011–2014	9 (5 + 4)	None	None	None	None
Promoted to Associate Professor	2013–2016	15 (4 + 11)	8	2	0	2 (1 R01, 1 R24)
Promoted to Associate Professor	2014–2016	10 (6 + 4)	4	1	1 (NIH Loan Repayment)	0

NIH, National Institutes of Health; CDC, Centers for Disease Control and Prevention; CMS, Centers for Medicare and Medicaid Services; HRSA, Health Resources and Services Administration; NHLBI, National Heart, Lung, and Blood Institute; NSF, National Science Foundation.

Program satisfaction

In addition to demonstrating positive outcomes in terms of research productivity, the evaluation data suggest that scholars were satisfied with the training they received (Table 5). Strong scholar outcomes and high scholar satisfaction with the program have encouraged Associate Deans for Research and other senior faculty at our university to recommend the program to junior faculty.

Table 5.

Scholar Satisfaction with Program Components (N = 12 Scholars)

	Preparation for research	Career development preparation	Mentoring team support	Departmental support	Usefulness of semiannual reviews
Median (range)	6 (6)	6 (4–6)	6 (5–6)	6 (5–6)	5.5 (4–6)

N, number of scholars.

Discussion

Although most faculty research career development programs use evaluation methods to track scholar outcomes, and likely use data from their evaluations for program improvement, few have offered details regarding how they have used data from a process evaluation for quality improvement of a training program. To date, many publications that discuss faculty career development programs emphasize mentoring programs that improve skills in teaching^7,8 or research productivity.^9
–11 Typical of many career mentoring programs, evaluation relies on assessing scholarly productivity or measurable outcomes related to numbers of grant submissions, awards, and research publications.^9
–11 Programs also report using mixed, quantitative, and qualitative methods to assess and better understand the bidirectionality of mentoring relationships as well as secondary effects of mentoring on faculty job satisfaction, perceptions of organizational commitment to scholars, and self-efficacy and confidence in research skills.^12

–18

The importance of a program evaluation is emphasized in a 2012 review of BIRCWH research career development programs, which found that 28 of 29 programs incorporated a structured evaluation of mentoring and training components.¹⁵ The authors emphasized evaluation as a significant strength of the national program. Several faculty mentoring programs gather feedback from scholars regularly to monitor their performance and so that course corrections can be made in a timely manner. A midpoint or interim survey implemented to gather opinion on program effectiveness offers frequent opportunities to improve program quality and to detail lessons learned in discussions of program outcomes.^14,19
–21 Thorndyke et al.²² proposed a multilevel evaluation of mentoring whereby mentors can be matched to scholars based on their specific needs and expertise. Law et al.²³ offered a checklist of recommendations for implementing and evaluating mentoring programs that includes assessing the intent, structure, processes, resources, and evaluation of the program. However, scant attention has been given in academic journals to describing process evaluation methods and how process evaluation output data are used to identify, strategize, and implement program improvements.

We have added to the literature by demonstrating the intentional use of output data from a process evaluation guided by a logic model as a tool for continuous quality improvement of a research career development program. Examples highlighted the tools and methods used to evaluate whether each component of the program was implemented with fidelity and quality. The evaluation results were used to address program shortcomings in a timely manner. This process increased scholar, mentor, and advisory board members' satisfaction with and commitment to the program. We posit that this had a positive impact on scholar outcomes.

The NIH requires frequent outcome evaluations of scholars in training programs. Any faculty development program could enhance their overall program functioning by incorporating a process evaluation as illustrated by the UMN BIRCWH program exemplar. Adding a rigorous process evaluation to the outcome evaluation already required by the NIH can increase the effectiveness and impact of a research career development program.

Footnotes

Acknowledgments

This work was supported by a Building Interdisciplinary Research Careers in Women's Health Grant (K12HD055887) from the Eunice Kennedy Shriver National Institute of Child Health and Human Development, the Office of Research on Women's Health, and the National Institute on Aging.

Author Disclosure Statement

No competing financial interests exist.

References

U.S. Department of Health and Human Services Centers for Disease Control and Prevention. Office of the Director, Office of Strategy and Innovation. Introduction to program evaluation for public health programs: A self-study guide. Atlanta, GA: Centers for Disease Control and Prevention, 2011 October, 96 p. Available at: https://cdc.gov/eval/guide/cdcevalmanual.pdf Accessed March 16, 2018 .

U.S. Department of Health and Human Services Centers for Disease Control and Prevention. Division of Health Communication and Marketing. What we know about…. Available at: https://cdc.gov/healthcommunication/pdf/evaluationplanning.pdf Accessed March 16, 2018 .

Bland

, Center

, Finstad

, Risbey

, Staples

. A theoretical, practical, predictive model of faculty and department research productivity. Acad Med, 2005; 80:225–237.

Bland

, Schmitz

, Stritter

, Henry

, Aluise

. Successful faculty in academic medicine: Essential skills and how to acquire them. New York: Springer Publishing Co., 1990:167–170.

Berbert

, Brown

, Bland

, Risbey

, Trotman

. Planning for the generational turnover of the faculty: Faculty perceptions and institutional practices. In: Clark

, Ma

, eds. Recruitment, retention, and retirement in higher education: Building and managing the faculty of the future. Northampton, MA: Edward Elgar Publishing, 2005:80–100.

Millikin

, Bakken

, Betz

. Assessing research self-efficacy in physician-scientists: The clinical research appraisal inventory. J Career Assess, 2007; 15:367–387.

Lord

, Mourtzanos

, McLaren

, Murray

, Kimmel

, Cowley

. A peer mentoring group for junior clinician educators: Four years' experience. Acad Med, 2012; 87:378–383.

Leslie

, Baker

, Egan-Lee

, Esdaile

, Reeves

. Advancing faculty development in medical education: A systematic review. Acad Med, 2013; 88:1038–1045.

Howell

, Kravet

, Kisuule

, Wright

. An innovative approach to supporting hospitalist physicians towards academic success. J Hosp Med, 2008; 3:314–318.

10.

Morrison

, Lorens

, Bandiera

, et al. Impact of a formal mentoring program on academic promotion of department of medicine faculty: A comparative study. Med Teach, 2014; 36:608–614.

11.

Thompson

, Schucker

, Kent

, et al. Reviving the vascular surgeon scientist: An interim assessment of the jointly sponsored lifeline foundation/national heart, lung, and blood institute william j. von liebig mentored clinical scientist development (K08) program. J Vasc Surg, 2007; 45 Suppl A:A2–A7.

12.

Bagramian

, Taichman

, McCauley

, Green

, Inglehart

. Mentoring of dental and dental hygiene faculty: A case study. J Dent Educ, 2011; 75:291–299.

13.

Kohn

. A mentoring program to help junior faculty members achieve scholarship success. Am J Pharm Educ, 2014; 78:29.

14.

Thorndyke

, Gusic

, George

, Quillen

, Milner

. Empowering junior faculty: Penn state's faculty development and mentoring program. Acad Med, 2006; 81:668–673.

15.

Guise

, Nagel

, Regensteiner

, Building interdisciplinary research careers in women's health directors. Best practices and pearls in interdisciplinary mentoring from Building Interdisciplinary Research Careers In Women's Health Directors. J Women's Health, 2012; 21:1114–1127.

16.

Jackevicius

, Le

, Nazer

, Hess

, Wang

, Law

. A formal mentorship program for faculty development. Am J Pharm Educ, 2014; 78:100.

17.

Dutta

, Hawkes

, Kuipers

, Guest

, Fear

, Iversen

. One year outcomes of a mentoring scheme for female academics: A pilot study at the Institute of Psychiatry, King's College London. BMC Med Educ, 2011; 11:13.

18.

Tracy

, Jagsi

, Starr

, Tarbell

. Outcomes of a pilot faculty mentoring program. Am J Obstet Gynecol, 2004; 191:1846–1850.

19.

Gadbury-Amyot

, Smith

, Overman

, Bunce

. Faculty development at one midwestern dental school: A program evaluation. J Dent Educ, 2015; 79:1177–1188.

20.

Files

, Blair

, Mayer

, Ko

. Facilitated peer mentorship: A pilot program for academic advancement of female medical faculty. J Women's Health, 2008; 17:1009–1015.

21.

Bryant

, Aizer Brody

, Perez

, et al. Development and implementation of a peer mentoring program for early career gerontological faculty. J Nurs Scholarsh, 2015; 47:258–266.

22.

Thorndyke

, Gusic

, Milner

. Functional mentoring: A practical approach with multilevel outcomes. J Contin Educ Health Prof, 2008; 28:157–164.

23.

Law

, Bottenberg

, Brozick

, et al. A checklist for the development of faculty mentorship programs. Am J Pharm Educ, 2014; 78:98.