Design and Implementation of an Online Course on Research Methods in Palliative Care: Lessons Learned

Abstract

Background:

Research capacity in palliative and end-of-life care is less than some other fields of medicine where there is a longer track record of biomedical research. Palliative medicine clinicians often receive little or no formal research training during their postgraduate education; hence, education efforts may prove pivotal to increasing palliative care research capacity. To that end, our group established a national online training program on palliative care research methodologies, called Foundations of Palliative Care Research. This report describes the development and implementation of the course, and its evaluation. To inform decisions on the overall course objectives, length, design, and implementation, formal needs assessments were conducted through surveys of Canadian palliative medicine residency program directors and of Canadian palliative medicine residents.

Methods:

A 12-week, online, module-based course was designed. The first iteration of the course was offered to English-speaking palliative medicine residents from across Canada between October 2008 and March 2009. The course utilized Web-based communication methods, and was delivered using a combination of asynchronous and synchronous learning strategies and activities.

Results:

Ten palliative care residents from different parts of the country registered and all completed the course with passing marks. Participants evaluated the course through a post course survey. The formal evaluation of the course, along with successes, challenges, and lessons applicable to future ventures, are discussed.

Introduction

There is a critical shortage of well conducted clinical research in palliative and end-of-life care.^1–3 One potential contributing factor to this shortage is that palliative medicine clinicians often receive little or no formal research training during their postgraduate education.² As such, the overall pool of investigators and the number of research efforts in palliative care have remained small compared to other areas of medicine.⁴

A group of investigators met to explore ways to enhance research capacity in palliative care across Canada. We reasoned that it can take many years to increase research capacity within a large health care community, and that education was a pivotal starting point. We recognized that Canada's geography and sparse population made an online training program a strong option for achieving our goals. Herein, we describe the development and implementation of an online course, called Foundations of Palliative Care Research. This is to our knowledge the first such online course on palliative care research methodologies anywhere in the world. The formal evaluation of the course, along with successes, challenges, and lessons applicable to future ventures, are discussed. We present this information so that palliative care educators can benefit from and improve upon our approaches to increase research capacity through online research methods education initiatives.

Course development: Overview

Acumen in online learning has expanded rapidly over the past few years. Members of our group based at the University of Calgary had prior experience with distant and distributed instructional design and technologies,⁵ and from the start we elected to apply similar approaches to the design and delivery of the Foundations of Palliative Care Research course. We used four widely available, interdependent approaches to conduct the online course: a curriculum designed to be of interest to a group with a range of learning needs and interests; a course background document containing readings, assignments, and other learning materials; live web-based audio conferencing software (Elluminate®, Elluminate USA, Pleasanton, CA); and course management software (BlackBoard®, Blackboard, Inc., Washington, D.C.) to support asynchronous discussions. Central to our process of curriculum development was the six-step approach to curricular design outlined by Kern.⁶

With our understanding of the curriculum and intended delivery methods, we chose to limit the number of course participants to 20. We were aware there is in Canada a year-long nationally accredited training certification for palliative medicine consultants, with postgraduate residency training programs active in most medical schools across the country. We suspected that these residents would be motivated to enroll in the course. To inform our decisions on course objectives, length, design and implementation, we undertook formal needs assessments by surveying Canadian palliative medicine residency program directors and separately, palliative medicine residents.

Canadian palliative medicine residency program directors: Needs assessment

A 22-item survey was sent via e-mail to all Canadian palliative medicine residency program directors (n = 12). Respondents were asked to rank the importance of 18 potential course topics (e.g., ethical issues in palliative care research, proposal writing, research design, statistical analyses), using a 5-point Likert-like scale anchored by “not at all important” (1) and “very important” (5). The remaining four questions queried the availability of a similar course, as well as respondents' attitudes toward such a course offering for their residents. Eight program directors, from five Canadian provinces, completed the survey. All 18 potential course topics were ranked as “important,” “quite important,” or “very important” by all respondents (i.e., no topic received a ranking of little or no importance by any respondent; Appendix A). We interpreted this to mean that residency program directors were generally keen on their trainees developing competence in research methods, but were not entirely discerning as to exactly what that meant. Thus, we had latitude regarding the curriculum content.

All eight directors indicated they would encourage their residents to take a 12-week online research course, and five of them would require their residents to take the proposed course. We understood this to mean that most residency program directors would strongly support their trainees to participate and were likely willing to protect them from the many other competing demands on their time.

Canadian palliative medicine residents: Needs assessment

A convenience sample of seven Canadian palliative medicine residents from Alberta, British Columbia, and Ontario was surveyed to investigate the needs of residents regarding research training. An online survey was created using Zoomerang™ software (Zoomerang, San Francisco, CA). Residents were contacted via e-mail with a link to the survey. All residents completed the survey. Most respondents were very interested in learning about palliative care research methods and participating in a palliative care research project, but the majority felt such undertakings should be optional rather than mandatory in their residency programs. When asked to rate their level of comfort with various topics, the majority of respondents felt comfortable with ethical issues in palliative care, systematic reviews in health care, and research designs. Most respondents felt uncomfortable with or neutral about topics relating to the role of theory in palliative care research, the challenges of palliative care research, writing a research proposal, the scientific and ethical review process, critical evaluation of published literature, qualitative research methods, survey research, statistical analyses, and manuscript preparation.

Methods

Course implementation

A 12-week, online, module-based course was designed. The course contained five modules, each approximately 2 weeks in length: (1) Introduction to inquiry in palliative care; (2) Scientific Communication; (3) Research Design Part 1; (4) Research Design Part 2; and (5) Data analysis and interpretation.

The first iteration of the course was offered to English-speaking palliative medicine residents from across Canada between October 2008 and March 2009. Residents learned about the course from their program directors, who were provided with course information and registration instructions by e-mail. The course utilized Web-based communication methods, and was delivered using a combination of asynchronous and synchronous learning strategies and activities. Four synchronous Elluminate guest lectures were conducted in real time and saved to allow viewing at a later date. Each Elluminate session lasted 90 minutes and was scheduled so that it occurred in the early evening in western Canada and late evening on the East Coast to accommodate the 4-hour difference in time zones across Canada. Asynchronous learning activities included online narrated “Breeze” PowerPoint presentations (Adobe Presenter®, Adobe, San Jose, CA) and online discussions held in forums moderated by palliative experts, using BlackBoard learning management system. Participants were expected to submit one substantial posting per week to the online discussions. As well, required and suggested readings from the palliative medicine literature were provided to participants in the form of a paper-based course manual. It was expected that learners would spend 4–6 hours per week on online activities and required readings. Elluminate sessions and online discussions focused on issues and challenges in palliative medicine research. Course participants were provided with a textbook⁷ focused specifically on the unique aspects of conducting palliative care research, such as the ethical and practical difficulties of working with very ill patients, the heterogeneous patient population, and the range of research methods and outcome measures used to address physical, psychosocial, and spiritual issues of patients and their families.

Course fees for participants were based on a cost-recovery model, to cover the costs of the course manual and textbook ($175 CAD). While some programs covered these costs for their residents, most participants were responsible for paying their own course fees. A much greater expense was incurred in developing the course material and course manual, and delivering the formal and informal didactic program. Some of the course developers, and all online presenters and moderators, donated their time. Despite these factors, the actual cost incurred for course development and delivery was $40,760 CAD.

Each participant was expected to complete a pre- and postcourse survey, two assignments (submission of a research proposal, and critical review of a research paper), attend the Elluminate® guest lectures, and contribute at least one substantial posting per week to the online discussion forums.

Course evaluation tool

A postcourse survey was developed to evaluate the course. Surveys were sent by e-mail and fax and returned in a manner to support anonymity. The survey was comprised mostly of 5-point Likert-like scale questions that assessed the following topics: learners' demographics, comfort level with online learning technology, prior research experience and training, attitudes toward palliative care research, comfort level with research topics, online course learning experience, and quality of course modules. Three open-ended questions solicited narrative comments from the residents: What worked well in the course? What did not work well in the course? How would you improve the course?

Results

Learners' demographics

Ten palliative care residents registered for the course. Three learners were between 20 and 30 years of age, four were between 31 and 40 years of age, and three were older than 40. Trainees lived in Calgary (2), Vancouver (3), Toronto (2), Ottawa (2), and Halifax (1). Overall, the trainees had received limited formal didactic teaching in research methods and statistics prior to enrolling in the course. Eight had participated in some kind of research project in the past, and only two were ever involved in palliative care research. Six participants were required to take this course by their residency program directors.

Course evaluation

All 10 residents completed the course and obtained a passing mark.

Comfort level with online learning technology

Participants rated their comfort level with learning via asynchronous and synchronous computer mediated conferencing using a 5-point Likert-like scale ranging from “very uncomfortable” (1) to “very comfortable” (5). Participants' mean comfort level with asynchronous learning methods was higher after the course (4.2 ± 0.2) compared to precourse scores (3.0 ± 0.2). Participants' mean comfort level with synchronous learning methods also improved after the course (scores increased from 3.0 ± 0.3 before the course to 3.9 ± 0.3 after the course).

Attitudes toward palliative care research

There was greater agreement with offering this course as an optional part of palliative medicine training rather than a mandatory component, yet there was general agreement that palliative medicine should be evidence-based and that research is essential to the advancement of the discipline (Table 1). Most participants intended to conduct palliative care research after completion of their residency program.

Table 1.

Attitudes of Course Participants Toward Palliative Care Research ^a

Statement	Agreement score
Palliative care practice should be evidence based	4.0 ± 0.2
Palliative care research is essential to the advancement of the specialty	4.4 ± 0.2
My palliative care residency program/fellowship should include a mandatory research methods course	3.3 ± 0.3
My palliative care residency program/fellowship should include an optional research methods course	4.1 ± 0.4
Conducting a research project should be mandatory in my palliative care residency program/fellowship	3.1 ± 0.3
Conducting a research project should be optional in my palliative care residency program/fellowship	3.8 ± 0.4
All palliative care physicians should be involved in palliative care research	2.8 ± 0.2
I am very interested in learning more about palliative care research methods	4.0 ± 0.3
I am very interested in participating in a palliative care research project	4.2 ± 0.2
I intend to keep involved in palliative care research when I finish my residency/fellowship program	4.0 ± 0.2
I will only participate in palliative care research if I have to (to meet program requirements)	2.7 ± 0.4

Participants rated their agreement with each statement using a 5-point Likert scale anchored by “completely disagree” (1) and “completely agree” (5). Values are mean agreement scores ± standard errors for each statement (n = 10 participants).

Comfort level with research topics

Comfort levels were higher for ethical issues, the challenges of palliative care research, and randomized controlled trials, than for most topics relating to statistical analysis (Table 2).

Table 2.

Participants' Rankings of Level of Comfort with Research Topics ^a

Research topic	Comfort level ranking
The role of theory in clinical research	3.8 ± 0.2
Ethical issues in palliative care research	4.3 ± 0.2
The main challenges of palliative care research	4.4 ± 0.2
The scientific and ethics review process	3.5 ± 0.3
Defining the research question	3.8 ± 0.1
Writing a research proposal	3.7 ± 0.2
Systematic reviews	3.8 ± 0.2
Critical evaluation of the published research	3.8 ± 0.1
Reporting research results	3.1 ± 0.3
Principles of measurement (reliability and validity)	3.1 ± 0.3
Sampling	3.0 ± 0.3
Causes and causal inference	3.1 ± 0.2
Bias and confounding	3.7 ± 0.2
Qualitative research methods	3.7 ± 0.2
Survey research	3.6 ± 0.2
Cross-sectional studies	3.5 ± 0.2
Cohort studies	3.4 ± 0.2
Case-control studies	3.8 ± 0.1
Randomized controlled trials	4.1 ± 0.1
Measures of disease frequency	3.6 ± 0.3
Measures of association	3.1 ± 0.3
Data collection methods	3.2 ± 0.4
Data management	3.1 ± 0.4
Descriptive statistics and measures of variability	2.9 ± 0.3
Statistical inference	2.6 ± 0.3
Comparison of group means: the t test	2.8 ± 0.3
Analysis of variance	2.9 ± 0.3
Multiple comparison tests	2.5 ± 0.4
Nonparametric tests of significance	2.6 ± 0.3
Correlation and regression	3.0 ± 0.2
Analysis of frequencies: χ²	3.0 ± 0.3
Statistical measures of reliability	3.0 ± 0.3
Multivariate analysis	2.9 ± 0.3

Participants rated their level of comfort with each research topic using a 5-point Likert scale anchored by “very uncomfortable” (1) and “very comfortable” (5). Values are mean comfort level scores ± standard errors for each topic (n = 10 participants).

Online course learning experience

Overall, participants rated the online learning experience, moderators, and guest speakers very highly (Appendix B). Despite some technical problems with the instructional media and a heavier than expected workload, the majority of participants found online learning to be a useful learning method and generally had a positive experience.

Quality of course modules

Participants rated the quality of each course module (reading materials, independent self-learning online modules, Elluminate sessions, and moderation) using a 5-point Likert-like scale ranging from “very poor” (1) to “very good” (5). With few exceptions, the quality of all five course modules was rated as good or very good (data not shown).

Narrative feedback

Comments regarding course quality were solicited through the use of open-ended questions. Several participants commented positively on the Elluminate and Breeze sessions and the quality of moderation. One respondent commented on the convenience of online learning and its accessibility from any location. One participant noted, “The mix of asynchronous and synchronous sessions was a good thing; it changes the rhythm of learning (keeps things “alive”) and increases the chances to learn something by exchanging experiences.”

Several comments suggested that the course was a greater undertaking than expected. Required readings were felt to be extensive and time consuming. Several respondents suggested a need for more time to complete the course—either by lengthening the time to complete the modules, by removing some modules, or by dividing the course into two smaller courses. A few participants expressed the need for protected time to complete the course.

Discussion

The planning, implementation, and evaluation of this online palliative care research methods course gave its organizers greater understanding of how to develop online courses for a community of palliative medicine residents dispersed across a country.

Course successes

We were pleased that all students successfully completed the course. After taking the course, participants' comfort level with online learning technologies improved; all participants expressed a greater understanding of palliative care research methods; nine of the ten participants indicated they felt more effective at conducting research; and eight felt more motivated to conduct palliative care research. The synchronous Web-based didactic sessions were particularly highly rated, and we were pleased to note that despite operating classes across four time zones (a technical challenge described by others in the literature⁸), these sessions were still well attended. There was widespread support for this course from palliative medicine residency program directors from across Canada, and residents were represented from programs coast to coast.

Course challenges

Curriculum development

Developing teaching materials for both synchronous and asynchronous components of this online course was resource intensive and the time and financial investment exceeded our expectations. We were also challenged with trying to balance comprehensiveness of content within a finite, 12-week timeframe. We aimed to focus course activities on topics highly relevant to palliative care research, with the value clearly explained, and a delineation of the time anticipated to complete the tasks.⁹

Delivery format

A course that was entirely online was believed most convenient for connecting learners from across Canada. However, we were aware that it can be challenging to stimulate deep cognitive engagement through entirely online discussions.¹⁰ We therefore explicitly employed a variety of principles to facilitate online learning including:

Hosting an introductory Elluminate session whereby participants got to know each other;

Posting frequent reminders of upcoming events through various routes;

Regular encouragement by course moderators to promote active participation;

Relating the content of the asynchronous discussions to the course manual and required readings, grounding the discussions in issues pertinent to palliative medicine;

Using high-profile palliative care research experts as course moderators and Elluminate guest lecturers to promote engagement, passion and applicability of the material.

We received feedback consistent with the published literature that providing written materials (in our case, a course manual with print copies of all readings) rather than having solely online material was of benefit.⁹

Technical challenges are inevitable with virtually all instructional media, and run the risk of deflating satisfaction and course participation.¹¹ Having technical support available for the synchronous activities was most helpful.

Learner engagement

Maintaining learner engagement was a factor critical to the success of the course. Postgraduate medical residents have busy schedules that may compete with course expectations. The literature attests to challenges in running a course entirely online over an extended period of time, such as 6 months or longer, with a substantial risk of drop off of participants.^8,9 The potential for “social isolation” can be an issue in a community of distributed learners.^10,11 We were well aware that learner interaction through discussion, debate, and other aspects of collaborative learning, is key to successful uptake of material in both traditional and online environments,⁹ and there is evidence that too little interaction is the largest source of dissatisfaction with online education.¹² Posing provocative questions within the online Blackboard discussions helped to foster discussion and debate. The inclusion of a few synchronous live sessions was also helpful in sustaining interest in the course.

Effective online tutoring is also important for promoting learning in an online environment.^13,14 Our online discussion moderators promoted active participation by providing regular feedback to participants and by posting questions to stimulate discussion. The time commitment of online moderators was considerable, similar to what has been described in the published literature.^8,15

Lessons learned and future directions

Although the course was offered to a relatively uniform group of learners, we observed an enormous diversity in learning styles. Some learners were more comfortable reading the materials as opposed to taking advantage of online asynchronous Web presentations. Others indicated they found the online presentations helpful. Overall, developing a course that will anticipate and respect the wide range of learning styles of adult learners is key to success.^8,16

Course participants strongly agreed that “palliative care research is essential to the advancement of the specialty,” but had much lower levels of agreement with the statement, “I will only participate in palliative care research if I have to.” We interpret these data to mean that participants understood palliative care research to be foundational to the advancement of the specialty, much aligned with a core belief we had which motivated our construction of the course. The group was divided on whether training in palliative care research should be a mandatory part of residency training, and we interpret this to reflect more complex social issues around right to self-determination. Perhaps the palliative care community is more likely to respond to invitations to participate in research training than it is to respond to coercion.

The biostatistics module proved the most difficult content for the learners. Presenting introductory statistics effectively in 2 or 3 weeks of instruction presents a challenge. Future iterations of the course will embed biostatistics content in relevant, problem-oriented discussions.

A few learners felt pressured to keep up and wondered if the course should be longer. There are substantial risks in running online courses longer than 6 months.^8,9 Learner fatigue is a prominent issue with lengthy online courses. We favor careful content review, strategies to maintain engagement, and a delivery timeframe of no more than 3 to 6 months.

In summary, we successfully provided an innovative learning opportunity to a geographically dispersed cohort of palliative medicine learners, engaging them in a curriculum focused on relevant research topics. In doing so, we believe we have provided a foundation from which they may become active in adding to the palliative medicine evidence base. Future educational endeavors to support palliative medicine research will build on this foundation. Although we specifically targeted palliative medicine residents for this first iteration of the course, we look forward to the vibrant online discussions that will take place when the course is made available to learners from a variety of palliative medicine disciplines.

Footnotes

Acknowledgments

We wish to thank the following individuals for their involvement in the development and teaching of this course: Carla Stiles, Jose Pereira, Gillian Fyles, Eduardo Bruera, Srini Chary, Penny Brasher, and Anne Bruce.

Funding to support the development and implementation of the course was provided through the CIHR New Emerging Team Grant for Difficult Cancer Pain PET69772.

Author Disclosure Statement

No competing financial interests exist.

Appendix A.

Canadian Program Directors' Rankings of the Importance of Potential Course Topics ^a

Course topic	Importance ranking
Role of theory in clinical research	4.1 ± 0.3
Quantitative vs. qualitative research paradigms	4.4 ± 0.2
Ethical issues in palliative care research	4.8 ± 0.2
Scientific review process and ethics	4.6 ± 0.3
Proposal writing	4.3 ± 0.3
Critical evaluation of published research and research proposals	4.6 ± 0.2
Principles of measurement (validity and reliability)	4.3 ± 0.3
Research design: question, sampling, data collection and management	4.4 ± 0.3
Research design: descriptive/exploratory studies	4.3 ± 0.3
Research design: experimental/intervention studies	4.1 ± 0.2
Research design: survey research	4.3 ± 0.3
Research design: epidemiologic research	3.9 ± 0.3
Causes and causal inference	3.8 ± 0.3
Bias and confounding	4.5 ± 0.3
Statistical analysis: data management	4.0 ± 0.3
Statistical analysis: descriptive statistics	4.3 ± 0.3
Statistical analysis: statistical inference	4.0 ± 0.2
Reporting results	4.4 ± 0.2

Each course topic was evaluated using a 5-point Likert scale anchored by “not at all important” (1) and “very important” (5). Values are mean importance rankings ± standard errors for each course topic (n = 8 program directors).

Appendix B.

Participants' Ratings of Their Online Learning Experience ^a

Statement	Agreement score
Overall, as a result of the course, I have a better understanding of palliative care research methods	4.0 ± 0.0
Overall, as a result of the course, I am more comfortable with critical appraisal of the published palliative care research literature	3.8 ± 0.4
Overall, as a result of the course, I feel I will be more effective in conducting palliative care research than before the course	4.1 ± 0.2
The course objectives were met	3.6 ± 0.3
The course content was relevant to my clinical practice	3.9 ± 0.2
Overall, this was a good learning experience	4.1 ± 0.1
I will recommend this course to future palliative care residents	3.6 ± 0.3
Online learning (as occurred in this course) offers a useful method of learning when there are geographic and other barriers to accessing learning	3.9 ± 0.3
Overall I learned a lot from the online discussions (asynchronous discussions—using Blackboard)	3.8 ± 0.3
The online discussions were interesting	3.8 ± 0.3
The questions posted for online discussion were relevant to the course content	3.9 ± 0.1
The discussions prompted me to reflect on research issues that I would not generally reflect upon by myself	3.4 ± 0.2
The workload of the online component of the course was more than I expected	4.3 ± 0.3
The amount of technical problems I experienced with Blackboard (asynchronous discussions) in this course was minor	3.5 ± 0.4
Overall, the technical support I received during the course was good	4.0 ± 0.3
The amount of technical problems experienced with Elluminate (live online discussions) in this course was minor	2.5 ± 0.4
The amount of technical problems experienced with Breeze (online self-learning modules) in this course was minor	4.0 ± 0.3
The Breeze presentations were a valuable learning resource	3.9 ± 0.3
The mandatory readings were a valuable learning resource	4.3 ± 0.3
The suggested readings were a valuable learning resource	3.9 ± 0.3
Overall, the course moderators responded to my questions in a timely manner	4.3 ± 0.2
Overall, the course moderators enhanced my learning experience	4.2 ± 0.1
Overall, the Elluminate guest speakers presented interesting work	4.2 ± 0.1
My main motivation for posting to the online discussions was to meet the course requirements of at least 2 substantive postings per module	3.9 ± 0.2
The evaluation rubric (i.e., marks for participating in the course activities) in this course is fair	3.7 ± 0.3
The assignment load of the course was reasonable	3.1 ± 0.4
The first assignment (critical review) was a useful learning experience	3.5 ± 0.5
The second assignment (research proposal) was a useful learning experience	3.5 ± 0.4
Overall, as a result of the course, I feel more motivated to conduct palliative care research	3.9 ± 0.3

References

Rinck

, van den Bos

, Kleijnen

, de Haes

, Schade

, Veenhof

. Methodologic issues in effectiveness research on palliative cancer care: A systematic review. J Clin Oncol, 1997; 15:1697–1707.

Kaasa

, Dale

. Building up research in palliative care: An historical perspective and a case for the future. Clin Geriatr Med, 2005; 21:81–92.

Zimmermann

, Riechelmann

, Krzyzanowska

, Rodin

, Tannock

. Effectiveness of specialized palliative care: A systematic review. JAMA, 2008; 299:1698–709.

Hagen

, Addington-Hall

, Sharpe

, Richardson

, Cleeland

. The Birmingham International Workshop on Supportive, Palliative, and End-of-Life Care Research. Cancer, 2006; 107:874–881.

Pereira

, Palacios

, Collin

, Wedel

, Galloway

, Murray

, Violato

, Lockyer

. The impact of a hybrid online and classroom-based course on palliative care competencies of family medicine residents. Palliat Med, 2008; 22:929–937.

Kern

, Thomas

, Howard

, Bass

. Curriculum Development For Medical Education: A Six-Step Approach. Baltimore, MD: The Johns Hopkins University Press, 1998.

Addington-Hall

, Bruera

, Higginson

, Payne

. Research Methods In Palliative Care. Oxford, UK: Oxford University Press, 2007.

Hall

, Cuttini

, Flemmer

, Greisen

, Marlow

, Schulze

, Smith

, Valls-i-Soler

, Truffert

, Conole

, de Laat

. European online postgraduate educational programme in neonatology—The way forward? Eur J Pediatr, 2009; 168:449–456.

Zimitat

. Designing effective on-line continuing medical education. Med Teach, 2001; 23:117–122.

10.

Cook

, McDonald

. E-learning: Is there anything special about the "E"? Perspect Biol Med, 2008; 51:5–21.

11.

Cook

. Web-based learning: Pros, cons and controversies. Clin Med, 2007; 7:37–42.

12.

Casebeer

, Kristofco

, Strasser

, Reilly

, Krishnamoorthy

, Rabin

, Zheng

, Karp

, Myers

. Standardizing evaluation of on-line continuing medical education: physician knowledge, attitudes, and reflection on practice. J Contin Educ Health Prof, 2004; 24:68–75.

13.

Curran

, Lockyer

, Kirby

, Sargeant

, Fleet

, Wright

. The nature of the interaction between participants and facilitators in online asynchronous continuing medical education learning environments. Teach Learn Med, 2005; 17:240–245.

14.

Wong

, Greenhalgh

, Russell

, Boynton

, Toon

. Putting your course on the Web: Lessons from a case study and systematic literature review. Med Educ, 2003; 37:1020–1023.

15.

Lockyer

, Sargeant

, Curran

, Fleet

. The transition from face-to-face to online CME facilitation. Med Teach, 2006; 28:625–630.

16.

Maley

MAL

, Harvey

, Bastiaan de Boer

, Scott

, Arena

. Addressing current problems in teaching pathology to medical students: Blended learning. Med Teach, 2008; 30:e1–e9.