Abstract
Growth in the demand for public health services, along with limited funding, makes workforce collaboration and capacity building imperative. The faculty and staff of the Midwestern Public Health Training Center, with two Robert Wood Johnson Public Health Nurse Leaders, postulated that training could be more effective, and public health workers more effective in the field, if workers contributed to training format and content. The learning paradigm was tested on diabetes prevention and self-management programs. Public health professionals were surveyed on infrastructure, practices, roles, and gaps in diabetes-related services. Responses influenced the format and content of a one-day diabetes summit training program. Participants submitted evaluations immediately afterward. Eight months postsummit, participants were surveyed to self-assess behavioral changes attributed to the training. Using the Kirkpatrick model for evaluation, participants (
Keywords
Demand for public health services has never been greater, and it is still growing (Trust for America’s Health, 2018). At the same time, funding has been slashed (Ye, Leep, & Newman, 2015). Resources are chronically less than what is needed to provide foundational services (Bekemeier, Marlowe, Squires, Tebaldi, & Park, 2018). Collaboration of public health systems, services, and resources is imperative at all levels—national, regional, state, and local.
Background and Literature Review
In response to the growing needs and demands, the Health Resources and Services Administration (HRSA) of the U.S. Department of Health and Human Services created 10 regional public health training centers (PHTCs) serving all 50 states (HRSA, n.d.).
The Midwestern Public Health Training Center (MPHTC), designated Region VII, includes Iowa, Kansas, Missouri, and Nebraska. Each of the regional PHTCs is assigned a unique topic by HRSA to develop content and training area to include in their work to support the overall network. The MPHTC content area was diabetes. Diabetes is a chronic, costly, potentially catastrophic disease, requiring significant involvement with public health systems (Centers for Disease Control and Prevention [CDC], 2018c). Diabetes prevention programs (DPPs) and diabetes self-management education (DSME) are two of the most important services promoted in public health systems, with trained professionals instructing diabetic and prediabetic individuals about lifestyle changes and disease management (American Association of Diabetes Educators, 2018; American Diabetes Association [ADA], n.d.; CDC, 2018a; Diabetes Prevention Program Research Group, 2002). The CDC (2018b) recognizes DPPs, with a supporting evidence base for program effectiveness. The American Association of Diabetes Educators, with the ADA, developed the DSMEs.
Past research has concluded that training in best practices should improve the health professionals’ knowledge and skills in the delivery of the DPP and the DSME services (Carroll et al., 2015; DiBenedetto, Blum, O’Brian, Kolb, & Lipman, 2016). In addition, adult learning theory asserts that training is most effective when users play a role in course design and content (eLearning Industry, 2011-2019). Challenges to the implementation of training include travel cost and time to learn from national experts, limited evaluation of training effectiveness, and translating knowledge into improved practice and increased program funding (Rattay & Rosenthal, 2014).
The MPHTC’s mission is capacity building of the technical, scientific, educational, and managerial functions of the public health workforce as well as to identify and prepare potential leaders. Leaders must act on the frontline, in a complex system with limited funding. Capacity building at strategic points in public health systems is a form of leadership (Heifetz, Linsky, & Grashow, 2009; O’Malley & Cebula, 2015).
The Kirkpatrick Model of program evaluation is used to assess the effectiveness of training (Kirkpatrick & Kirkpatrick, 2006; Nesbit et al., 2018). It uses four levels that assess differing training parameters. These are as follows:
Level 1: Assesses reactions to and satisfaction with the content of the training just received.
Level 2: Assesses acquisition of knowledge, improvement in skills, and the workers’ attitude toward the training. The model can include self-ratings to assess the learning of objectives and the probability that the workers will apply the new knowledge in practice (MacVarish et al., 2018).
Level 3: Assesses changes in behavior attributed to the training.
Level 4: Assesses organizational changes, impact of the training, and outcomes beyond the individual.
Level-3 and Level-4 evaluations may be seen as especially resource and labor intensive, limiting research and reporting in the medical literature (Paull, Whitsed, & Girardi, 2016; U.S. Office of Personnel Management, 2011; Wu, Roychowdhury, & Khan, 2017). Recently, however, public health preparedness centers have demonstrated the need for this level of evaluation data and have procured investments in training programs and evaluation frameworks using Levels 1 to 4 (Hites et al., 2014; Public Health Foundation, n.d.).
In seeking to develop diabetes prevention training and to implement Level-3 and Level-4 evaluation methods, MPHTC had a unique opportunity to partner with two Robert Wood Johnson Public Health Nurse Leaders (RWJ-PHNL; Center for Creative Leadership, 2018). RWJ-PHNL fellows are required to complete a team-based leadership project, which results in meaningful outcomes (Fralic & Morjikian, 2006). The RWJ-PHNL fellows from the 2015-2017 cohort, who are already public health nurses in the MPHTC Region, agreed to focus on their leadership project as a team and with MPHTC, which included support for the development of a diabetes intervention training summit and its evaluation.
This partnership allowed for more in-depth evaluation processes for DPPs and DSMEs, with the hope that enhanced evaluation would reveal progress in learning and impact among trainees who completed these programs. This meant understanding the training needs of professionals and clinics/agencies that could use or refer to DPPs and DSMEs. These health professionals would benefit from both content knowledge and training support to develop skills in networking and collaboration.
The MPHTC regional training summit was chosen as the optimal training method for the project because of the MPHTC’s specialty area of public health workforce development focusing on diabetes, including its prevention and management. Thus, regionally, overall content of the training was pertinent, regardless of the state in which participants resided.
Methods
In 2015, the MPHTC formed a National Diabetes Advisory Council of nationally recognized experts and key stakeholders in diabetes prevention and management (Table 1). These subject matter experts provided general guidance in overall training and program delivery needs for MPHTC in the region.
MPHTC National Diabetes Advisory Council Organizations
NOTE: MPHTC = Midwestern Public Health Training Center.
Includes Region VII diabetes planning team members.
While the council supported a larger program of work for MPHTC, it also served as a sounding board for RWJ-PHNL and MPHTCs plan to develop training around DPP and DSME programs. The council specifically oversaw the development, application, and evaluation of assessment tools and made recommendations to enhance training. While the advisory council oversaw the program, the two RWJ-PHNL fellows executed and coordinated the data collection and reporting within the advisory process (Figure 1).
Diabetes Prevention Capacity Development Model
Gap Assessment
An electronic survey about DPP and DSME services was administered first to inform training development. This survey was sent to a large network of public health partners including but not limited to primary care providers, current DPP providers, and potential new DPP providers. E-mails were sent through various list serves in the region, resulting in 122 responses from an undetermined possible number of respondents. The questionnaire consisted of approximately 21 questions using fixed and open-ended responses.
Most of the questions focused on identifying the agencies’ infrastructure, practices, roles, and gaps related to DPP- and DSME-related services. For example, one question stated, “Which of the following individuals in your clinic discuss the risk of prediabetes with patients? Select all that apply.” Responses to this question included nurse, nurse practitioner, registered dietitian, diabetic educator, certified medical assistant, health coach/health coordinator, physician, physician assistant, pharmacist, and other. All questions included a list of possible responses for the participant (e.g., yes/no/unsure or check all that applied with open-text boxes for “other”). Information from the survey provided significant guidance in developing a training summit. The respondents to this survey were also expected to be the primary audience for the summit training. This included the following survey results:
Diabetes prevention programs: Screening for prediabetes usually occurred during wellness exams, chronic visits, and acute care visits (62%). Clinics indicated that they refer patients to DPP programs about 50% of the time. In those clinics, a majority of the individuals making those referrals are direct service providers (e.g., nurse/nurse practitioner = 30%, physician/physician assistants = 35%). Common clinic barriers identified to such screening include the following: lacking policies and procedures in places (43%), no screening tools in place (19%), and no DPPs available (13%). Common patient barriers to receiving services included the patients not understanding that such services are necessary (27%), having health insurance that did not cover DPP (25%), or other financial barriers (24%). Common opportunities to screen and refer patients to DPPs included wellness programs (27%), weight management programs (24%), hospital discharge (23%), and WIC (Women, Infants, and children) programs (15%).
Diabetes self-management education: About 77% of the participants who work in clinics indicated that they routinely talk with diabetic patients about the need to participate in DSME programs, while the remaining 23% of clinic participants indicated that this did not occur in their clinics. Most common roles among those who routinely assess and talk about DSME included nurses/nurse practitioners (29%), physicians/physician assistants (27%), certified diabetes educators (14%), and registered dieticians (13%). The patient settings where these assessments and talks occurred included chronic care checkups (30%), annual physicals (28%), every visit (18%), and acute care visits (16%). For 67% of the respondents, local DSME programs existed to which patients could be referred from clinics. Common barriers for patients getting necessary DSME services included patients not understanding that such services are necessary (25%), having health insurance that did not cover DSME (22%), and other financial barriers (31%).
Regional Diabetes Summit
Findings of the gap analysis were used to guide content and frame strategy discussions for a regional diabetes summit. The advisory council, with the RWJ-PHNL fellows, supported the creation of a Region VII diabetes planning team from multidisciplinary agencies formed within three of the four states (Iowa, Kansas, and Missouri) in Region VII, to coordinate the summit. Representation included professionals from the MPHTC, state and local public health, ADA, the YMCA, the Association of Public Health Nurses, and nonprofit health care organizations (Table 1). The summit agenda was approved by the MPHTC advisory council.
The 1-day summit in November 2016 was conducted at a site centrally located in each of the three states that participated (Nebraska had recently had a training session in the topics of interest). Specifically, the gap analysis indicated that content training was needed, which included best practices and current trends in diabetes management. After a didactic morning session, the summit format also included state-specific afternoon small groups with priority training activities determined by each state.
There were 178 participants from throughout Region VII, representing a variety of health care settings: hospitals, primary care practices, diabetes centers, governmental health agencies, academic institutions, and others. There was a variety of health care professionals: certified diabetes educators, clinicians, health coaches, dieticians, nurse practitioners, physicians’ assistants, and others.
The morning session included interactive video conferencing of national speakers. Presentations took place live in Iowa and were live streamed to other sites and included the following:
“The Role of the Department of Health and Human Services in Addressing Diabetes Prevention and Control,” presented by Shary Jones, PharmD, MPH, BCPS, U.S. Department of Health and Human Services.
“Current Trends and Social Impacts of Prediabetes and Diabetes,” presented by Patricia Schumacher, MS, RD, CDC.
“Innovative Approaches to Implementing National Diabetes Prevention Programs,” presented by Julissa Molina Soto, NDPP Life Style Coach, Bilingual Master Trainer, and Monica Chavez-Singleton, NDPP Life Style Coach, Bilingual Master Trainer, ADA.
“The Value of Diabetes Self-Management Education and Diabetes Education and Support Programs,” presented by JoEllen Condon, RD, LD, CDE, ADA.
In the afternoon session, participants worked in breakout groups by state to integrate strategies learned in the morning and to develop next steps and recommendations for their own systems. For Iowa, participants were split into two groups: one focused on DSME strategies and the other on DPP strategies to address an assigned barrier/challenge identified through the gap analysis. After completing the procedure, the groups switched tasks. Missouri participants attended presentations on face-to-face and virtual NDPPs, DSME sites, diabetes accreditation standards for pharmacists, and a content-expert panel on how to become an NDPP or DSME. Kansas participants attended one of two breakout sessions on diabetes advocacy or the state of diabetes prevention and control in Kansas.
Results
Postsummit Level-1 and Level-2 Evaluation
Evaluations were completed following the summit on November 18, 2016, by 112 of the 178 participants across three states. Evaluation data gathered at the summit included qualitative and quantitative questions as well as participant consent for follow up. At the summit, Level-1 evaluation (documenting participant reactions to training) and Level-2 evaluation (measuring participant learning) were conducted. All program quality ratings by participants were indicated on a 5-point Likert-type scale (1 = strongly disagree, 2 = somewhat agree, 3 = neither agree nor disagree, 4 = somewhat agree, 5 = strongly agree).
Level-1 Reactions
Quantitative results included reactions of the summit (n = 112). This resulted in mean ratings above 4.0 for all indicators (Table 2).
Qualitative responses indicated what the participants (n = 112) found most valuable about the training. Convergent themes focused on the breakout afternoon sessions that increased networking and collaboration and understanding the existing infrastructure in their state.
Additional themes included understanding current data and trends, as well as a more detailed understanding of DPP and DSME programs.
Level-1 Participant Reactions to Summit Training
Level-2 Learning
Quantitative results included learning from the summit (n = 112). This also resulted in mean ratings above 4.0 for all indicators (Table 3).
Qualitative responses indicated what the participants (n = 109) learned from the summit that could be implemented immediately. Themes included networking with state partners and obtaining resources and tools for setting up, promoting, and engaging partners with DPP and DSME programs.
Participants also included topics that they would have liked to have covered. Convergent themes included success stories by DPP and DSME programs, funding and payment of programs, community coordination and educational awareness of programs, and more direct patient care and prevention interventions.
Level-2 Participant Learning From Summit Training
Postsummit Level-3 and Level-4 Follow-up Evaluation
In the Level-1 and Level-2 postsummit survey, participants were asked if they could be contacted for a 6-month follow-up survey about the summit, and 49 participants agreed. Thirty (61%) of the 49 participants completed the 6-month follow-up survey from three states (Iowa = 36.36%, Kansas = 21.21%, and Missouri = 42.42%). Of these participants, 30.30% (n = 9) indicated that they did work for a primary care clinic (PCC) with patients who may have prediabetes or diabetes. The remaining participants (69.70%, n = 21) indicated that they worked for an organization/agency that did not provide primary care but did provide other services related to prediabetes or diabetes. Participants connected with DPPs in the following ways: 40% offer DPP classes or were thinking about it in their areas, and 60% did not but provided other services related to prediabetes or diabetes (n = 30). Participants were also asked questions regarding “what changes occurred directly as a result of the summit.” This included behavioral changes of the participants (Level 3) and organizational results/outcomes reported by the participants (Level 4).
Level-3 Behavior Change
Participants indicated how their diabetes prevention efforts have grown in ways outside of direct services, which resulted in 40% expanding their professional network, establishing partnerships with other providers and community businesses, as well as offering new diabetes-related services (n = 30).
Of the participants who worked in PCCs, 66.67% were conducting more routine screenings for prediabetes while 33% were not (n = 9).
Level-4 Organizational Change
The participants who “offered DPP classes or were thinking about it” indicated how their organizational status for becoming a CDC-recognized DPP provider changed on a continuum as a result of the summit (Prochaska & Velicer, 1997). ● 1 = Lacking awareness of diabetes prevention programs ● 2 = Contemplating the idea of starting a program ● 3 = Committing to the idea of starting a program ● 4 = Organizing a program and training staff ● 5 = Beginning classes in your community ● 6 = Gathering and submitting data for CDC recognition ● 7 = Achieving full CDC recognition as a DPP provider.
Sixty seven percent of the participants moved forward by one stage (n = 12).
Participants indicated how many DPP groups (yearlong cohorts) their program has the capacity to serve per year, estimating an average of 2.2 cohorts per year before the summit and increasing to 4.2 cohorts per year after the summit. Actions taken to increase capacity included working with primary care providers to get more referrals; partnering with other community organizations to attract more participants; addressing cultural barriers to participation such as languages spoken, food preferences, and customs and beliefs about health care; and putting more resources into publicizing the program (n = 5).
When considering all the PCC patient visits made for routine (nonemergency) health care services every month, 80% of the participants who worked in PCCs estimated a 10% to 20% increase in routine screening for prediabetes because of the summit (n = 5). Actions taken to increase screenings included the following: clinics have modified or adopted new screening tools and/or identified one or more DPPs to which to refer patients who screen positive for prediabetes. Reasons for not changing screening practices included the following: clinics had not modified or adopted any new screening tools or had not put any new policies and procedures for screening in place.
When considering all PCC patient visits every month for those who have prediabetes, participants who worked in PCCs (n = 4) estimated an average of 5 referrals per month before the summit and increased to an average of 12.5 referrals per month afterward. Actions taken to increase screenings include clinics becoming more familiar with DPPs and how they work, participants identifying local DPPs available to patients, and participants actively helping patients get enrolled in DPPs. Reasons for not referring to DPPs include health insurance does not cover DPPs, DPPs not available in participant area, patients do not have transportation for getting to DPPs in the area, or health insurance has too high of a copay or out-of-pocket expense for patients.
As a result of the training, 60% of PCCs (n = 9) have partnered with other programs in the community for screening and referring patients with prediabetes to DPPs. These include weight management programs, wellness programs, health coaches, and community programs.
Discussion
The training approach allowed the region to address training gaps in diabetes prevention and management work as well as evaluate potential progress. Data obtained through this project and the engagement of regional leaders, including the RWJ-PHNL fellows, have been instrumental in formulating continued strategies in addressing diabetes prevention and control within the region, states, and locally. State-level summits provided an immediate link with new knowledge and active discussion on moving the system (participants and their organizations) in the context of regional and national perspectives. This was much more than training in that it was an opportunity to engage with system partners to strengthen systems for prevention and care related to diabetes. The improved organizational outcomes on a variety of factors reported for the Level-4 evaluation provides the greatest support for this approach of this work. Therefore, evaluation methods are critically important to determine if and how public health system moves forward and how we can make a case for public health’s value to policymakers, funders, and the public at large (Kirkpatrick & Kirkpatrick, 2006; U.S. Office of Personnel Management, 2011).
Since the initial summit in November 2016, Iowa continues to have an active Diabetes Prevention and Control Coalition and has hosted annual summits. Four additional partnering agencies have joined since the original group developed. A formal Iowa Diabetes Prevention Action Plan 2018-2020 was developed through a collaboration between the Iowa Diabetes Control and Prevention Coalition, the Iowa Department of Public Health, the CDC, and the NACDD (Iowa Department of Public Health, 2017). In addition, Telligen has partnered with the Iowa Department of Public Health to offer local Lifestyle Coach Training to help increase the number of DPP programs offered in the state. An Iowa Diabetes Prevention Program Providers private Facebook group has also been established for members to interact with one another. The page offers members the opportunity to share barriers, best practices, and resources related to offering program services.
This progress shows promise for continuing and extending the project’s methods beyond diabetes prevention and management specifically. It models a relationship of training delivery and leadership development through the role of PHTC and individual leadership development programs. This not only aligned resources but also allowed for deeper consideration of the public health system coordination among and between learners. The RWJ-PHNL fellows’ engagement enhanced their own leadership skills by providing support with critical aspects of the summit development and evaluation process.
PHTC can engage frontline leadership to take ownership of their own system with appropriate knowledge, skills, and networks. This approach to training should also include greater efforts to include more Level-3 and Level-4 evaluation methods in training, which can provide more compelling return of investment support for the needs of the public health system.
Strengths and Limitations
There are known benefits and limitations to live and distance-based trainings. The diabetes program attempted to utilize the best of each modality to expand reach and deepen learning. The live webinars permitted real-time best-practice sharing and follow-up discussions, while also providing a recording open and free to the public. The summit provided opportunities for the region to test this model and support its utility as a future training model. Limitations of the posttest data collected include the use of self-report measures for all levels. In addition, Level-3 and Level-4 samples were rather small and could only provide qualitative data. The current process used a participant opt-in method for the follow-up survey, which may have also contributed to a low response rate and a response bias. Future approaches could benefit from using an opt-out method for data collection and using additional objective measures for Level-2 and Level-4 evaluations. With this said, the current findings are meaningful in that participants indicated that practical progress in diabetes prevention was a direct result of their learning from the summit on a range of indicators.
Conclusions
As public health leadership programs continue to provide opportunities for innovative projects by fellows, PHTCs can also find ways to support these initiatives while expanding their own work and outcomes. Continued synergy among multidisciplinary health care teams, such as the one formed through this project, is vital in addressing future diabetes control and prevention efforts.
This type of collaborative and multidimensional training initiative will be continued with MPHTC. Future diabetes summits will target individual states’ development needs, with the region providing support and context. To further the evaluation results of MPHTC trainings conducted to date, more diabetes professional development trainings and Level-1 to Level-4 evaluation–type strategies will be developed. This should include 6-month follow-up evaluation to determine training impact over time.
The value of public health training can lead to organizational improvement in practice. While enhanced training can take more time and funding, the favorable public impact should convince policymakers and funding programs that investing in public health training is truly cost-effective.
Footnotes
Authors’ Note:
The authors would like to recognize the following individuals for their contributions to the diabetes prevention and education work with the regional diabetes summit: from Kansas: Megan Skaggs from the Kansas Foundation for Medical Care, Charlene Wallace from the American Diabetes Association, and Amanda Bridges from the Kansas Department of Health and Environment Association; from Iowa: Ami Bolles from the American Diabetes Association, Kady Reese from the Iowa Healthcare Collaborative, Carol Hibbs from the Marshalltown YMCA–YWCA, and Katie Jones and Laurene Hendricks from the Iowa Department of Public Health; from Missouri: Glenn Studebaker and Teresa Goslin from the Missouri Department of Public Health and Senior Services; and from Nebraska: Brandon Grimm from the Office of Public Health Practice–College of Public Health, University of Nebraska Medical Center. Their experience brought meaningful insights and value to workforce needs. We would also like to thank Faith Coleman from the University of Iowa College of Public Health and writing consultant Theresa St. Romain for assistance with manuscript preparation. The Wichita State University Institutional Review Board determined that this study is within established regulations for human participant compliance (Number 4162, Review Category: Program Evaluation). This work was supported by the Health Resources and Services Administration of the U.S. Department of Health and Human Services, grant number UB6HP27879—Affordable Care Act public health training centers. In-kind support was provided by the Robert Wood Johnson Foundation.
