Abstract
In this commentary, we will describe our study and report results that will be of interest to information and education professionals and researchers. Evidence-based medicine requires health professionals to keep up to date with new research-based knowledge. Canadian physicians must now participate in Continuing Medical Education (CME) activities. CME strives to improve clinician performance as well as patient health outcomes. Our study was aimed to assess whether physicians who participated in a CME program and expected health benefits for their patients following an elearning activity were more likely to have higher participation in the program in subsequent years. Weekly treatment Highlights were delivered by email to practicing family physicians across Canada, who rated them using the Information Assessment Method (IAM). The number of expected benefits for patients reported by participants during 2016 was plotted against the number of instances of participation in 2017. Results show that the number of expected benefits in 2016 was correlated with the number of IAM ratings in 2017.
Introduction
The practice of evidence-based medicine (EBM) requires the integration of the best available research evidence with clinicians’ expertise and patient values (Karagiannis, 2019). It was originally proposed that practicing clinicians themselves should be capable of finding, appraising, and integrating research evidence at the point of care (Guyatt et al., 1992). The sheer volume of published research studies, often with contradictory findings, and time pressures of physicians make this goal untenable. Thus, electronic knowledge resources, expertly and regularly curated to contain the latest and best available evidence, have emerged as essential tools for the practice of EBM. As EBM progresses, there is an increasing role for systematic summaries of patient oriented evidence (Djulbegovic & Guyatt, 2017).
The use of online continuing medical education (CME) programs is promising as it allows for the large distribution of educational materials. Advantages of online CME include convenience and low cost. In terms of the impact of CME, studies have found that CME programs have a positive impact on physicians’ clinical practice and a moderate impact on patient outcomes (Cervero & Gaines, 2015). These positive effects of CME are greatest when CME programs focus on outcomes such as practice improvement, which are considered important by clinicians (Cervero & Gaines, 2015).
In the context of CME in primary health care, four levels of information outcomes have been proposed: the situational relevance, the cognitive/affective impact, the intention to use, and expected health-related outcomes of educational information (Pluye et al., 2013). As described in a companion paper in this special issue by Granikov et al., the Information Assessment Method (IAM) has been validated to systematically assess and document these information outcomes in multiple CME contexts. In terms of education, the IAM is considered as a reflective learning tool for clinicians (Leung et al., 2010). Moreover, the feedback generated through the IAM can help information managers and health information professionals (HIPs) to optimize the informational content presented to clinicians (Pluye et al., 2014).
Assessing and knowing more about the expected patient health benefits of information in the context of CME can contribute to enhance future participation in CME programs, which is relevant to HIPs, and especially those interested in education. HIPs continue to play an important role in the management of evidence-based health information used in CME in medicine (Ma et al., 2018). A recent scoping review on the roles of HIPs found that their involvement in CME and curriculum development are two of the most common types of professional duties for HIPs (Ma et al., 2018). In this setting, HIPs are “expected to be able to present and maintain user engagement in active online and in-person contexts” (Ma et al., 2018). It is thus important to equip HIPs with knowledge, tools and techniques in preparation for these roles. In other words, the systematic identification of information which is both relevant to clinicians and beneficial to patients may help HIPs to optimize the information content in CME programs and sustain clinicians’ continued participation (the base of CME outcomes) (Moore et al., 2009).
This commentary describes our study. Our general objective was to explore the association between physicians’ expected beneficial health outcomes for their patients arising from their reading and assessing educational information in a national CME program, and their subsequent participation in this program. Measuring these benefits can provide valuable insight into the clinical challenges faced in daily practice by thousands of physicians across a variety of settings. In this study we hypothesized that clinicians who reported more expected health benefits for their patients from information objects (such as educational emails) contained in a CME program would be more likely to have more instances of participation in the program in subsequent years. Thus, our exploratory specific research question was: among family physicians in Canada participating in a CME program based on weekly educational email alerts, is there a relationship between the number of expected patient benefits reported by a clinician using the IAM and their subsequent participation in this program during the following year?
Methods
Study design
This study was a cross sectional observational study.
Setting
The Highlights CME program was developed by the Canadian Pharmacists Association in collaboration with the College of Family Physicians of Canada (CFPC) and the Information Technology Primary Care Research Group (McGill University). Family physicians and pharmacists across Canada received a weekly email containing an evidence-based therapeutic summary (a Highlight) excerpted from e-Therapeutics+, an online drug and therapeutic resource published by the CPhA (now RxTx,
Participants
Family physicians across Canada who evaluated at least one Highlight using the IAM between January 1, 2016 and December 31, 2016 were included. Participation was voluntary, and family physicians could submit their first IAM at any point during the study period.
Data sources and variables
Participants’ responses for each IAM item were automatically recorded with a date and time stamp, linked by an anonymized identifier, and aggregated per participant per year. The number of IAM questionnaires completed per participant per year was assessed through the number of entries of each participant in the yearly database of IAM ratings. The outcome under study is the number of IAM questionnaires completed by each family physician between January 1, 2016 and December 31, 2017. The number of expected patient benefits was calculated for each physician in 2016 by counting all instances of ‘Yes’ and one-half of all instances of ‘Possibly’ responses to the question: “Do you expect any health benefits [for a specific patient] as a result of applying this information?”. This number is equivalent to the numerator of the Benefit of Information Index, normally calculated for each information object (Pluye et al., 2016).
Bias
This study included only family physicians who voluntarily enrolled in the program and chose to complete an IAM questionnaire upon reading a Highlight, at least once during 2016. The number of weekly recipients of the Highlight emails and the number of recipients who opened and read the email are unknown to us. This selection bias precludes any generalization of our results to all family physicians in Canada. No adjustment was made for physicians’ demographic variables or for the type of information contained in the rated Highlights.
Statistical methods
Descriptive statistics on the number of ratings by year and by participant were computed. A smooth scatter plot was created using the smoothScatter function of R (version 3.5.2) to explore the relationship between the number of expected benefits reported by clinicians in 2016 and the number of completed IAM questionnaires in 2017. A generalized linear model using Poisson error distribution and a log link was fit using cubic smooth B-splines to assess the statistical significance of the observed relationship.
Results
A total of 4527 family physicians (or residents in Family Medicine) rated at least one Highlight using the IAM between January 1, 2016 and December 31, 2016. Demographic questionnaires were completed by 4517 participants. Half of participants (2276/4517, or 50%) were female; ages ranged from 20 to 87 years, with the average age being 46.2 years (95% CI 45.9–46.6) (Fig. 1).
Participant characteristics of 4527 family physicians who completed at least one IAM rating in 2016, with 4517 completed demographics questionnaires. A. Distribution of participants’ gender (50.4% female, 49.6% male); B. Participants’ age in years in 2017, average 46.2 years (95% CI 45.9–46.6), 179 missing values; C. Distribution of the number of expected benefits reported per participant in 2016, median 1; D. Distribution of the number of completed IAM ratings in 2017, median 1. IAM: Information Assessment Method; FP: Family physician.
In 2016 these participating family physicians completed a total of 40,267 IAM ratings of 120 Highlights, in which they reported 11,750 expected health benefits. Twenty-nine percent of participants reported zero expected benefits, and three quarters expected two or less health benefits for a patient. The median value was one expected health benefit per participant in 2016 (IQR 0–2, range 0–54; average of 2.6, 95% CI 2.4–2.8). Among the 4527 family physicians who completed at least one IAM rating in 2016, 2585 of them completed 31,182 IAM ratings of 60 Highlights in 2017, whereas 1941 participants from 2016 did not rate any Highlights in 2017. The median number of IAM ratings per person in 2017 was 1 (IQR 0–8, range 0–52; average 6.9 ratings per person, 95% CI 6.5–7.2) (Fig. 1).
The scatter plot of the number of expected benefits per person in 2016 in relation to the number of completed IAM ratings per person in 2017 revealed a positive correlation between these two variables: a higher number of expected benefits in 2016 was related to a higher number of completed IAM ratings in 2017. The GLM model revealed that this relationship is statistically significant (
The number of expected health benefits for patients reported by family physicians in 2016 was correlated with the number of IAM ratings completed in 2017. Smoothed-density scatterplot with a cubic smoothing spline (8 degrees of freedom) showing the relationship between the number of expected health benefits for patients reported by family physicians in 2016 and the number of IAM ratings they completed in 2017. IAM: Information Assessment Method.
These results suggest that the number of instances of participation in an online CME program for family physicians is correlated with the number of expected health benefits reported by participants in the previous year. This relationship modelled as a Poisson family generalized linear model was statistically significant. While our results show an overall decrease in participation in the CME program between 2016 and 2017, physicians who reported a higher number of expected benefits in 2016 tended to have a higher participation in 2017. This finding suggests that family physicians will participate more often in a voluntary CME program if they have previously found the information to be potentially beneficial for at least one patient under their care.
Our study faces three main limitations. First, it is an exploratory study and our results should be interpreted with caution. We did not adjust for any demographic and professional characteristics of participating family physicians. Second, the number of expected benefits reported in 2016 depends on the participation rate of family physicians in 2016 (the number of expected benefits being necessarily a function of the number of IAM ratings completed). Thus, it is possible that a subgroup of physicians that displays a higher-than-average participation, and has a higher probability of having more expected health benefits, will then participate more the following year regardless of the number of expected benefits observed. Third, the participation in the Highlights program and the completion of IAM questionnaires were voluntary. It is likely that many recipients of the Highlights never completed a single IAM questionnaire. Thus, it is impossible to consider family physicians who read and used the information in the Highlight, potentially leading to expected health benefits for patients, but who did not rate an IAM questionnaire.
All education programs are based on the willingness to participate, and our results only apply to physicians willing to participate in the Highlights program and complete at least one questionnaire. Our exploratory study can pave the way to further research that will comprehensively assess the role of expected patient health benefits on physician participation in CME programs. The confirmation of our results by other research can have implications for helping HIPs to sustain CME programs.
Conclusion
The systematic identification of information which is deemed relevant and beneficial for patients may help HIPs to optimize the educational information content in CME programs, and to improve clinicians’ continued participation. In turn, this can enhance CME positive effects on patient outcomes. Finally, HIPs may use IAM-based results to regularly identify the types of information that are expected to be the most beneficial for each type of participating clinician (e.g., family physicians, specialists, pharmacists), and eventually tailor educational information content to participants.