Towards collective intelligence in a national community of physicians

1. Introduction

In 2006, one of the authors (RG) helped establish a Continuing Medical Education program based on POEMs, with support from the Canadian Medical Association (CMA). A POEM stands for Patient Oriented Evidence that Matters. POEMs are synopses of research articles or clinical practice guidelines reporting on health outcomes that matter for patients and their relatives, such as symptoms or mortality (Shaughnessy et al., 1994).

In 2019, about 25,000 physician members of the CMA receive the daily POEM via email. To document their learning, physicians complete a brief validated questionnaire for each POEM they read (Figs 1–3). On submitting a completed IAM questionnaire, physicians are taken to a page to confirm this submission and acknowledge their receipt of a continuing education credit. We call this process – using a validated questionnaire to reflect on and rate the daily POEM – the Information Assessment Method (IAM) (Badran et al., 2017). Presently, about 1,550 physicians rate each POEM with the IAM questionnaire (Grad et al., 2008). In our funded research, we have shown how this program promotes reflective learning (Leung et al., 2010).

OPEN IN VIEWER

OPEN IN VIEWER

OPEN IN VIEWER

The POEMs education program for physicians is remarkable for how popular and sustainable it has become. In the first quarter of 2018 alone, physicians submitted over 102,000 completed IAM questionnaires linked to POEMs. As such, we established a ‘mass movement’ involving a national community of physicians. Embedded within these 100,000 IAM questionnaires are more than 4,400 free-text comments. Thus, the IAM questionnaire stimulates physicians to comment on POEMs. Some of these comments provide insight into clinical issues not directly addressed in the POEM itself. Presently, we do not share these comments with physicians in our national community. Consequently, each comment represents an individual thought trapped in a silo.

We conceptualize free-text comments from physicians about the daily POEM as a type of mass peer review, analogous to commenting on web pages. These comments harvest professional experience and knowledge from thousands of physicians. Ideally, we would share these comments within the community to promote knowledge exchange at the collective level, i.e., enable collective intelligence that can benefit decision-making in clinical practice.

To illustrate the potential of this idea, we provide the following example. A physician read a POEM about a randomized controlled trial that tested the idea of screening women for ovarian cancer (Jacobs et al., 2016). In this trial, 200,000 women 50 to 74 years of age were randomized to cancer antigen 125 testing guided by a novel algorithm, ultrasonography, or no screening, and followed for a median of 11 years. No reduction in ovarian cancer mortality was found. In reflecting on this POEM information, an anonymous physician submitted the following comment through the IAM questionnaire: “This is the issue with many screening interventions; people still wish to be screened and find something. Risks of screening are rarely discussed.” The importance of this comment is in knowing about the risk of harm associated with cancer screening tests, such as those used to screen for ovarian cancer. Knowing more about these harms (such as false positive test results and their consequences) would be important information to share within a community of physicians, who in turn will help their patients make better-informed decisions, about screening for cancer.

The proposed project will nurture a longstanding interest of physicians in the clinical information contained within POEMs. Physicians recognize that adopting POEMs information in clinical practice could result in improved health care for patients. While this project is centered on physicians, information professionals can apply the underlying idea in multiple fields for different audiences using an appropriate version of the IAM questionnaire (IAM-clinician, IAM-manager, IAM-patient, and IAM-parent), or adapt an IAM questionnaire to meet their needs. The IAM is free for education and research (see www.mcgill.ca/iam).

We seek to enable ’collective intelligence’ for the continuing education of physicians in practice. Collective intelligence is shared or group intelligence that emerges from collaborative effort (Malone et al., 2010). We will harness collective intelligence through the specific tasks of producing and sharing constructive comments on POEMs, from and with the community of CMA members. In this way, we can begin to energize a discussion on POEMs for a second wave of learning, using constructive comments from the same physicians who engage in this educational program.

3. Proposal to harness collective intelligence

At www.iambeats.io, we have launched an online platform to help us harness collective intelligence from the ongoing POEMs Continuing Medical Education program. This online platform is inspired by the eSRAP platform developed at McGill University (Tang et al., 2015). The eSRAP platform now has three branches, respectively dedicated to the monitoring of emerging scientific literature, managing and facilitating the understanding of comments from a community, and community engagement. This platform will become a stimulating educational tool for the benefit of doctors in practice, analogous to researchers who benefit from peer review in science. This type of peer-to-peer interaction is characterized by learning from others on the clinical information in POEMs. To harness collective intelligence from a community of physicians, we propose to stimulate a second level of reflection on POEMs. We will accomplish this in two steps: (a) the selection of comments by one of us (RG) for posting on a collective intelligence platform, and (b) the synthesis of physicians’ comments.

4. Comment selection

We will offer physicians an opportunity to read and share their comments on POEMs, with other physicians. After reading a POEM, a physician opts to complete the IAM questionnaire, at the end of which a comment can be submitted. Once submitted, an acknowledgement page will be displayed. We will add a link to the acknowledgement page that reads “Would you like to see other doctors’ comments on this POEM?”. Clicking on this link will take the physician to the collective intelligence platform where organized comments on that POEM will be displayed. Once on the platform, a physician would be able to see and respond to others’ comments.

Content will be systematically organized into types of constructive feedback, as described below and in a prior paper (Pluye et al., 2014). This new content (and its organization into meaningful categories of constructive feedback) will enable a second wave of learning that follows a first wave of reflective learning achieved through the IAM questionnaire. The following two collective intelligence features will characterize our platform: presentation and exchange.

5. Comment synthesis

Presently, we can identify constructive feedback comments from thousands of physicians who rate POEMs. However, these comments need to be organized in a meaningful way before they can be presented to the community. Before they are posted, we will organize constructive comments into meaningful categories of constructive feedback. Then, we will display these comments for the benefit of the community. Compared to a rudimentary process that does not leverage comments submitted through the IAM questionnaire, this project will enrich our community by enabling peer-to-peer discussion on a more interactive platform.

On selected topics, an editor will extract insights from physicians discussing POEMs and create a synthesis for sharing with the community. This process will require further technical development to: (a) establish a more organized classification scheme for comments, and (b) create a data transfer protocol for transferring raw comments linked to a POEM to a collective intelligence platform. Presently, our system supports a closed, back-end process whereby ‘raw’ feedback comments from the IAM questionnaire are viewable, but only by us. In a concrete practical manner, we propose four major revisions of the existing POEMs program, which will lead to create a new improved program.

First, we will import physician comments from the IAM questionnaire into the new system. Then, the interface will allow an “editor” or community leader to easily sort and review each comment. The editor can rate individual comments using above-mentioned categories of constructive feedback, with the most interesting being selected by tagging and adding notes. The next step in our plan will be to use a data transfer protocol to connect these processed back-end comments to the community-facing collective intelligence platform.

Second, with the above components in place, we will offer the community an enhanced opportunity to read and share comments on POEMs with other physician members. In our workflow, we will systematically organize physician comments according to the concept of constructive feedback. The potential value of collective intelligence resides within these selected comments. From there, the collective intelligence process will further unfold, as physicians start to interact around the topic of the POEM, as opposed to simply submitting comments that will never be read by others.

Third, for our project to be successful, diffusion will be the broadest strategy to encourage participation and involves raising awareness of the platform. We will begin by announcing the ’birth’ of our community. This announcement will be located on the page that displays the IAM questionnaire linked to the daily POEM read by 25,000 physicians. In addition, dissemination is more active and targeted, and consists of reaching out to specific physicians. To be even more proactive, we will consider contacting any physician who has previously submitted comments on POEMs. The CMA can help us identify these persons. It makes sense to target this group, as these physicians have already demonstrated an interest in our work, by actively commenting on POEMs.

Finally, the following four elements will help to facilitate interactions between community members and enable a second wave of learning around POEMs: The ability to display (auto-populate) members’ names alongside their responses to selected comments; the ability to follow discussions, e.g., physicians may opt to receive alerts when others post comments about a specific POEM of interest; the ability of users to respond directly to questions and answers triggered by selected comments and members’ responses (discussion structured by threaded comments); the ability to attach files and images to questions and answers.

6. Conclusion

To our knowledge, at a national level, we implemented the first educational program to promote reflective learning on synopses of clinical research. This has raised awareness of research-based clinical information of high relevance to health care and the health of patients. On a new collective intelligence platform, we will leverage the benefits of the constructive feedback we receive from thousands of physicians who already read and rate research-based clinical information on an individual basis. This project will mobilize physicians through the sharing of constructive feedback on the most relevant clinical research of our time, uniting and inspiring physicians on health information that matters. Knowing about this project may be of benefit to a community of information professionals, who seek to improve their use of evidence and experience-based information in their daily work.