Evaluation of a guidelines website capitalizing on Finnish content and Belgium interface: A pilot study in French general practice

Abstract

No reference point-of-care, web-based medical compendium is available in general practice in France. We have then conducted the experimentation of EBMPracticeNet, a Belgian website of guidelines translated and adapted from the Finnish EBM Guidelines. We collected data from three sources: (i) the website logbook; (ii) a search-specific assessment questionnaire; (iii) a global assessment questionnaire. A cumulative number of 262 (62.8%) physicians performed at least one search on the website and clicked on average 5.9 times per month. Physicians globally got an accurate answer (74.2%). They found the information provided by the website reliable (92.2%) and useful for practice (78.6%). They perceived the website ergonomics as good. The main reported barriers were the time and effort required to find an accurate answer and the uneven relevance of the information retrieved. Improvements should focus on guidelines indexing and their adaptation to the French context, and training physicians to search medical databases.

Keywords

France general practice point-of-care resources practice guidelines

Introduction

Finding the appropriate current evidence that a busy general practitioner needs for decision-making during consultation implies developing resources that filter expanding literature according to reliability and relevance, and presenting it in an easily and quickly accessible way.¹ A variety of resources are available to physicians, including clinical practice guidelines, medical compendia and computerised decision-support systems.²

Efforts have been made to develop high-quality guidelines internationally, yet their methodological rigor and editorial independence often remain poor³ and they are rapidly outdated.⁴ Even if they are trustworthy, they can be complex⁵ or difficult to apply,³ and their format makes them difficult to consult at the point of care.^6,7 In France, their utility is well recognised, but their dissemination and implementation in practice have been neglected.^8,9

Clinical decision support systems are increasingly considered the best tools for guidelines implementation, as they provide the appropriate recommendations according to the patient’s clinical situation.² However, their effectiveness to improve patient care is still disappointing and they can contribute to alert fatigue and to physician burnout due to electronic health records.¹⁰

Point-of-care, web-based medical compendia are resources specifically designed to deliver rapidly accessible, pre-digested, comprehensive, periodically updated, evidence-based information to clinicians,¹¹ which could facilitate the use of guidelines.¹² More than 20 systems have been developed worldwide, like BMJ Best Practice in the UK, or Dynamed and UpToDate in the USA, which have been assessed as being in the top three according to their information volume, editorial quality and evidence-based methodology.¹¹ EBM Guidelines is one of these medical compendia, produced for primary care by the Finnish medical company Duodecim. It is a collection of 1000 guidelines linked to 4000 evidence summaries in Finnish and English, mainly based on the Cochrane reviews.¹³ The process used to develop these guidelines has been accredited by the National Institute for Health and Care Excellence (NICE) in the UK, based on the AGREE criteria.¹⁴ After guidelines translation into Dutch and French,¹⁵ EBMPracticeNet, a consortium of scientific and medical organisations, adapted EBM Guidelines to the Belgian context.¹⁶ This compendium, funded by the Belgian public health insurance system (INAMI), has been made freely accessible for all Belgian physicians on the EBMPracticeNet website (now entitled ebpracticenet).¹⁷

Evaluating health information systems is required to insure their effectiveness and safety,¹⁸ as they should have an impact on clinical practice and patient health.¹⁹ Since health care practice is highly context-sensitive, however, their evaluation should be performed in their context of use in order to be meaningful.²⁰ According to two studies, EBM Guidelines scored in the top quartile for evidence-based methodology and above the median for volume coverage.^21,11 Their use in context had once been assessed in Finland.²²

As no such no such point-of-care, web-based medical compendium was available in France, we conducted an experimentation of the Belgian EBMPracticenet website, in the context of French general practice. The aim of this study was to assess physicians’ participation, search effectiveness and users’ global appraisal of EBMPracticeNet.

Methods

This study consisted in a global evaluation of the website’s content and interface, in its context of use in routine French general practice, through indicators based on physicians’ procedures and appraisal.

Physicians’ recruitment

From January to February 2017, we recruited 367 members of the French National College of General practice via its mailing list (including approximately 2000 general practitioners (GPs)), and 50 postgraduate trainees from the French medical schools of Lyon and Saint-Etienne during university meetings. Those 417 registered physicians received an email with detailed website login instructions and were encouraged to use it at the point of care from March to September 2017. They also received monthly email reminders to use the EBMPracticeNet website.

Data collection

At their first connection, the participating physicians were asked to complete profile data. A personal login identification password was assigned to each in order to analyse their frequency of use and search behaviour. Each search was recorded in the website logbook, which contained the following fields: user ID, timestamp(s), queries (words or phrases), title(s) of the guidelines opened. In their assessment of the EBM Guidelines in Finland, Jousimaa et al.²² collected log files from users to measure the frequency of their searches.

During the 7-month study, search-specific, self-developed, assessment questionnaires were periodically activated online to assess physicians’ clinical search effectiveness (see Appendix 1). In addition, at the mid-point and end of the study, participants were asked by email and by post to complete a global assessment questionnaire, self-developed. This global assessment concerned website ergonomics (usability) and content (reliability, relevance to the French context, usefulness for practice), and physicians’ global satisfaction (see Appendix 2). The ergonomics perception was evaluated with the System Usability Scale (SUS), an international standard widely used for assessing information systems because it is valid, reliable even for small samples, and quick to administer and score. It consists of a ten-item questionnaire with five response options, and is based on a score ranging from 0 to 100.²³

Data analysis

From the logbook, we calculated the frequency of use per participant, as an objective indicator of participation. It was defined as the number of specific clicks performed monthly on the website, either for entering a query or for opening a guideline. From search specific assessments, we calculated the proportion of accurate answers provided by the website to searches. From the logbook, we distinguished inaccurate answers with no guideline opened from guideline(s) opened but no answer found by the physician. We calculated the mean SUS score and interpreted it according to the adjective rating scale designed by Bangor et al.²⁴

We compared the number of clicks of the physicians who responded, and those who did not, to the search-specific and global assessments, using the Student t-test. We also compared the physicians’ global assessment at the mid-point and at the end of the study, using the Chi-square test for qualitative variables and the Student t-test for the SUS score. These statistical analyses were performed using Epi Info™ software (version 7). We qualitatively analysed the free comments left by the respondents in the questionnaires.

Ethical and regulatory aspects

This study was declared to the French Data Protection Authority (CNIL, N° neS20672351). The participating physicians were fully informed of the aims of the study before consenting to participate.

Results

Among the 417 physicians registered, 262 (62.8%) performed at least one search on the website during the study period, including 229 GPs (89.8%) and 26 trainees (10.2%) (Figure 1; Table 1). Most GPs were trainers (63.3%). The physicians performed 10,737 clicks on the website, corresponding to 5234 queries entered and 5503 guidelines opened. On average, users clicked into the website 5.9 times per month (standard deviation: 9.7), corresponding to 2.9 queries entered, and 3.0 guidelines opened, per month. The respondents to the search-specific questionnaire clicked, on average, 13.0 times per month on the website, as compared with 2.4 clicks for the non-respondents (p = 6.7 × 10⁻¹⁰). The respondents to the global assessment clicked, on average, 9.8 times per month on the website, compared with 1.6 clicks for the non-respondents (p = 3.4 × 10⁻¹³). The number of clicks decreased as the study progressed, from 2862 clicks in the first month, to 597 clicks in the final month. The number of monthly users decreased during the study period from 168 to 53. In the meantime, the mean number of clicks per monthly user decreased from 17.0 to 11.3 (Figure 2).

Figure 1.

Summary of physicians’ recruitment and participation to the study.

Table 1.

Physicians’ profile.^*

	n (%)
Medical status
GP trainer	145 (56.9)
Non-trainer GP	84 (32.9)
Trainee	26 (10.2)
Gender
Male	131 (51.4)
Female	124 (48.6)
Age (years)
26–29	33 (12.9)
30–39	90 (35.3)
40–49	19 (7.5)
50–59	64 (25.1)
60–67	49 (19.2)
Practice setting
Urban	138 (60.3)
Semi-rural	52 (22.7)
Rural	39 (17.0)
Type of practice
Solo practice	34 (14.8)
Group practice	127 (55.5)
Medical Home	43 (18.8)
Health Centre	25 (10.9)
Mean visit number (per week)
12–49	20 (8.7)
50–99	118 (51.5)
100–149	73 (31.9)
150–250	18 (7.9)

The number of answers differed from 262 owing to missing data.

Figure 2.

Evolution of monthly users and clicks per user* during the course of the study.

We collected 194 search-specific questionnaires from 85 physicians (32.4%). Most of them had performed their search out of the consultation (61.3%), found this process easy (94.2%) and fast (94.3%), and got an accurate answer (74.2%) (Table 2). We collected a global assessment from 158 physicians at the mid-point (60.3%), and from 103 physicians at the end of the study (39.3%). At mid-point, most responding physicians found the information provided by the website rather reliable (90.5%) and useful for practice (75.3%), were rather satisfied (55.7%), and wished to continue using the website (94.9%). At the end of the study, most responding physicians also found the information provided by the website rather reliable (92.2%) and useful for practice (78.6%), were rather satisfied (63.1%), and wished to continue using the website (95.1%). No change was observed for these variables between the mid-point and the end of the study (Table 3). The mean SUS score was of 73.9 at the mid-point, and 70.3 at the end of the study, with no change in the meanwhile (p = 0.75).

Table 2.

Search-specific assessment (n = 194).

	n (%)
Search time
During the consultation	75 (38.7)
Out of the consultation	119 (61.3)
Easy use of the website
Yes	184 (94.2)
No	10 (5.2)
Quick search
Yes	183 (94.3)
No	11 (5.7)
Search answer
Accurate	144 (74.2)
Inaccurate	50 (25.8)
Guideline(s) but no answer	35 (70.0)
No guideline	8 (16.0)
Uncertain	7 (14.0)
Clear information
Yes	162 (83.5)
No	32 (16.5)
Information useful for patient care
Yes	119 (61.3)
No	75 (38.7)

Table 3.

Global assessment.

	At mid-point (n = 158)				At the end (n = 103)				p value
	n (%)			Total	n (%)			Total	p value
Information reliability	Rather reliable	Medium	Rather unreliable	157 (100.0)	Rather reliable	Medium	Rather unreliable	103 (100.0)	0.62
Information reliability	142 (90.5)	14 (8.9)	1 (0.6)	157 (100.0)	95 (92.2)	5 (4.9)	3 (2.9)	103 (100.0)	0.62
Information adaptation to the French context	Rather adapted	Medium	Rather nor adapted	157 (100.0)	Rather adapted	Medium	Rather nor adapted	103 (100.0)	0.62
Information adaptation to the French context	131 (83.4)	21 (13.4)	5 (3.2)	157 (100.0)	82 (79.6)	17 (16.5)	4 (3.9)	103 (100.0)	0.62
Website usefulness for practice	Rather useful	Medium	Rather useless	158 (100.0)	Rather useful	Medium	Rather useless	103 (100.0)	0.54
Website usefulness for practice	119 (75.3)	25 (15.8)	14(8.9)	158 (100.0)	81 (78.6)	14 (13.6)	8 (7.8)	103 (100.0)	0.54
Global satisfaction	Rather satisfied	Medium	Rather not satisfied	158 (100.0)	Rather satisfied	Medium	Rather not satisfied	103 (100.0)	0.23
Global satisfaction	88 (55.7)	56 (35.4)	14 (8.9)	158 (100.0)	65 (63.1)	31 (30.1)	7 (6.8)	103 (100.0)	0.23

We obtained 56 comments from the search-specific assessment, and 127 from the global assessment. Respondents mainly reported problems concerning the use of the website during the consultation, connection to the website, query terms entry, relevance of guidelines retrieved, comprehensiveness of the answer and its consistency with French recommendations (Table 4). The main barriers to using the website during consultations were search slowness and guideline length.

Table 4.

Main comments.

Comments	Search-specific assessment (n = 56)	Global assessment (n = 127)
	n (%)	n (%)
Website difficult to access (login and password not saved)	4 (7.1)	9 (7.1)
Website difficult to use during the consultation	2 (3.6)	19 (15.0)
Appropriate query terms difficult to select	5 (8.9)	7 (5.5)
Relevant guidelines difficult to retrieve from queries	5 (8.9)	10 (7.9)
No full answer in the guideline(s) retrieved	7 (12.5)	2 (1.6)
Recommendations not in line with French guidelines	4 (7.1)	8 (6.3)
Total	27 (48.2)	55 (43.3)

Of the 1038 guidelines accessible on the website during the study, 803 different guidelines were opened at least once (77.4%) and 636 at least twice (61.3%). The 20 most consulted guidelines accounted for 14.8% of guidelines opened. The five most consulted guidelines were on Lyme borreliosis, dermatophytosis, vulvovaginitis, gout and osteoporosis (Table 5).

Table 5.

Top 20 most consulted guidelines.

Most consulted guidelines	Number of clicks (n = 5503)
	n (%)
Lyme borreliosis	83 (1.5)
Dermatophytosis	67 (1.2)
Vulvovaginitis	67 (1.2)
Gout	50 (0.9)
Osteoporosis	45 (0.9)
Low back pain	45 (0.8)
Hyperthyroidism	42 (0.8)
Scabies	41 (0.8)
Hemochromatosis	38 (0.7)
Swelling of the salivary glands	34 (0.6)
Hypothyroidism	33 (0.6)
Flu	33 (0.6)
Disorders of the shoulder rotator cuff	32 (0.6)
Cow’s milk allergy	31 (0.6)
Treatment of urinary tract infections	30 (0.6)
Hypertension	30 (0.6)
Dyslipidaemia treatment	29 (0.5)
Hypercalcemia and hyperparathyroidism	29 (0.5)
Viral infections of the oral mucosa	28 (0.5)
Trigger Finger	28 (0.5)
Total	815 (14.8)

Discussion

Main results

During our 7 months of experimentation of the EBMPracticeNet website, the frequency of queries entered and guidelines opened was less than one per week and per user on average. The number of users sharply decreased during the study, whereas the number of clicks per monthly user slightly decreased. Physicians performed most of their clinical searches out of the consultation. They mostly got an accurate answer and found the information provided by the website reliable and useful for practice. They perceived its ergonomics as good and wished to continue using it. The main reported barriers concerned the time and effort required to find an accurate answer, and the uneven relevance of the information retrieved.

Confrontation to literature

The under-weekly frequency of both queries entered and guidelines opened was low compared with the number of clinical questions usually raised by primary care physicians. It has been estimated that physicians usually raise between 7 and 185 clinical questions per 100 consultations, of which, half lead to a search.^25,26 In fact, it would be worth comparing the low frequency of use of the website with the usual rate of guideline consultation by physicians, which is unknown. In our study, the number of participants decreased more than the frequency of use by the persistent physicians. Overall, we observed two groups of physicians: a majority who stopped participating and a minority who persisted until the end of the study.

The low frequency of use of the website can be explained by both the lack of effectiveness and usability of the website, but also by the user behaviour and the context of use of the website. According to the mean SUS score, the website ergonomics was perceived as good, but not enough to recommend it to a colleague.^23,27 The main barriers to using the website were the effort and time required to find accurate information, especially during the consultation. This is consistent with the literature, which identified time issues as the main barrier to the seeking of information by physicians.^1,25,28,29 The participating physicians also reported difficulties selecting appropriate search terms and retrieving relevant guidelines. Physicians are increasingly aware of the various sources of medical information, but not specifically trained to search within medical databases.^28,30 In addition, guidelines indexation was probably incomplete and sometimes inappropriate in this version of EBMPracticeNet. When the physician thinks that the information does not exist in a source of medical information, he may give up the search.²⁶ Guidelines indexing is therefore a critical issue for the development of web-based medical compendia. These barriers may explain the low level of persistent participation of physicians in our study. Although searches were mostly performed out of the consultation, which was already observed with another web-based medical compendium,³¹ most were judged useful for patient care. Actually, not all clinical searches need to be performed during the consultation, and the appropriate time depends on several factors: urgency of the clinical problem, time constraints of the consultation and relationship with the patient.²⁹

Some 74% of searches performed on the website led to an accurate answer. This is consistent with data from a systematic review, indicating that various medical information sources answered 78% of physicians clinical searches²⁶; in an evaluation of the use of EBM Guidelines in Finland, 71% of physicians found complete answers.²² If the efforts to search the website may have discouraged some physicians, those who persisted and responded to our questionnaires found answers to most of their clinical questions.

The large variety of guidelines consulted shows the scope and the diversity of information needs in general practice. Most of the top 20 consulted guidelines were for relatively common acute or chronic health problems (urinary tract infections, arterial hypertension, hypothyroidism); among them, 6 belonged to the 20 most frequent health problems managed by French physicians.³² This list also included rare conditions (hemochromatosis, swelling of the salivary glands, hypercalcemia and hyperparathyroidism) or publicised health problems (borreliosis of Lyme, cow’s milk allergy, dyslipidaemia).

Perspectives

In Finland in 1998, physicians consulted the EBM Guidelines – on CD-ROM at this time – on average three times per day.²² To implement EBMPracticeNet at a large scale in France, various barriers identified in our study should be overcome.

Navigation in the website should be facilitated and guidelines indexation enhanced in order to retrieve accurate information in a minimum of time. Guidelines in the website are currently indexed with the International Classification of Primary Care (ICPC-2), the International Classification of Diseases (ICD-10) and the Medical Subject Headings (MeSH).¹⁷ The Belgian Biclassified Bilingual Thesaurus (3BT) is also integrated into the website to help search guidelines more quickly and effectively.³³ This thesaurus could be improved based on the analysis of the website query logs, which include terms commonly used by French physicians. To fit the website to the French context, the Finnish translated guidelines should be adapted to improve their applicability and their appropriation by physicians. Among several processes developed worldwide for that purpose,³⁴ we may conform to the ADAPTE method³⁵ as used for the Belgian EBMPracticeNet.¹⁶ In addition, French guidelines should also be integrated into the website, which implies their design adaptation and XML-formatting.

French physicians are increasingly trained in evidence-based medicine. Specific training for postgraduate students in medical database searching may improve their skills and the frequency of use of the website.³⁶ In Finland and Belgium, access to the EBM Guidelines database is free for physician users. The economic model for France should be well thought-out, as the imposition of access fees to the website could impede its use by physicians.³⁷

Strengths and limitations

Among the three best evaluated point-of-care information systems (UpToDate, BMJ Best Practice and Dynamed), only UpToDate has been evaluated in context.^31,38 However, the setting of use differed from general practice in both studies and none of them included both a search-specific and a global assessment.

We cannot exclude a selection bias, as the physicians who evaluated the website via the search-specific and the global assessment questionnaires were those who used it most, and were presumably the most satisfied with it. However, it can be assumed that their better awareness of the website made their assessment more relevant. The generalisability of our findings can also be questioned, as most participating GPs were GP trainers. Although they are more interested in evidence-based medicine and have more information needs, they face the same difficulties in seeking and using evidence-based information^1,39 and do not consult guidelines more frequently than non-trainer GPs.⁴⁰ In addition, GP trainers can be considered as globally representative of all GPs.⁴¹

Unlike a one-year Finnish evaluation, which involved experienced users of the guidelines,²² this study involved French physicians who were new users. In this context where a too long study period could have led to physicians’ demobilisation, a 7-month length seemed a good compromise between the acceptability to the participants and the volume of data collected. The stability of results of the global assessment between the mid-point and the end of the study supports their reliability and sustainability, at least for physicians who responded to the final assessment.

Apart from the SUS score, the questions in the global assessment questionnaire and in the search-specific questionnaire were self-developed. This strategy resulted from the absence of a validated tool for evaluating the content of, and users’ satisfaction with, point-of-care information systems in their context of use.⁴² The questionnaires used were not assessed for validity or reliability.

Conclusion

French physicians who tested the Belgian EBMPracticeNet website had a limited use of it, apart from persistent physicians. A majority of searches led to an accurate answer, and website ergonomics was perceived as good. Almost all respondents wished to continue using the website. The main reported barriers were the time and effort required to seek information, and the uneven relevance of information retrieved. To implement EBMPracticeNet on a large scale in France, improvements should mainly focus on indexing guidelines and adapting them to the national context, and training physicians to search medical databases.

Since the present evaluation, a French website called ebmfrance.net, based on the EBM Guidelines collection translated into French, has been launched thanks to funding from the French Public health insurance system. Then two editorial processes have been undertaken, to integrate guidelines of the French National agency for healthcare quality (Haute Autorité de Santé) and adapt the EBM Guidelines to the French context.

Footnotes

Appendix

Appendix 2.

Global assessment questionnaire.

Questions	Answers options
Introduction
The following questionnaire evaluates your overall satisfaction with the ergonomics and content of the site It consists of 21 questions. Questions 3 to 12 may surprise you. They belong to the ‘System Usability Scale’, an international questionnaire which allows evaluating the ergonomics of a website. We kept the wording of the questions of its French validated version.
Global evaluation
1. Overall, what is your level of satisfaction with the site?	Very satisfied/Satisfied/Moderately satisfied/Unsatisfied/Not satisfied
2. Do you find this site useful for your practice?	Strongly agree/Somewhat agree/Neutral/Somewhat disagree/Strongly disagree
System usability scale
3. I think that I would like to use this system frequently.	Strongly agree/Somewhat agree/Neutral/Somewhat disagree/Strongly disagree
4. I found the system unnecessarily complex.	Strongly agree/Somewhat agree/Neutral/Somewhat disagree/Strongly disagree
5. I thought the system was easy to use.	Strongly agree/Somewhat agree/Neutral/Somewhat disagree/Strongly disagree
6. I think that I would need the support of a technical person to be able to use this system.	Strongly agree/Somewhat agree/Neutral/Somewhat disagree/Strongly disagree
7. I found that the various functions in this system were well integrated.	Strongly agree/Somewhat agree/Neutral/Somewhat disagree/Strongly disagree
Questions	Answers options
8. I thought there was too much inconsistency in this system.	Strongly agree/Somewhat agree/Neutral/Somewhat disagree/Strongly disagree
9. I would imagine that most people would learn to use this system very quickly.	Strongly agree/Somewhat agree/Neutral/Somewhat disagree/Strongly disagree
10. I found the system very cumbersome to use.	Strongly agree/Somewhat agree/Neutral/Somewhat disagree/Strongly disagree
11. I felt very confident using the system.	Strongly agree/Somewhat agree/Neutral/Somewhat disagree/Strongly disagree
12. I needed to learn a lot of things before I could get going with this system.	Strongly agree/Somewhat agree/Neutral/Somewhat disagree/Strongly disagree
Content evaluation
13. What do you think of the presentation of the information: clarity, legibility, conciseness	Very good/Good/Average/Bad/Very bad
14. Generally, do you find the answers to your questions?	Always/Often/Sometimes/Rarely/Never
15. Does the information found seem reliable to you?	Always/Often/Sometimes/Rarely/Never
16. Are the information adapted to the French context?	Always/Often/Sometimes/Rarely/Never
17. Are the information adapted to your practice?	Always/Often/Sometimes/Rarely/Never
18. Do you take this into account in your practice?	Always/Often/Sometimes/Rarely/Never
Others
19. Eventual comment(s)	Free text
20. Proposals for improvement	Free text
21. How do you use this site?	As a tool for decision support in consultation/As an information support out of the consultation (e.g. to prepare for a future consultation, or to reflect on a past consultation)/As a continuing education or teaching tool/Other
22. Would you like to continue using the site?	Yes/No

Acknowledgements

We would like to thank the physicians who participated to this study. We are grateful to Prs Cyrille Colin and François Gueyffier for their useful comments on the manuscript. We are indebted to the Collège de la Médecine Générale (CMG) and to the Collège Lyonnais des Généralistes Enseignants (CLGE) for funding the English editing of the manuscript.

Author contributions

All authors contributed to the conception and design of the study and interpretation of data. CR performed data analysis. CR and LL drafted the first version of the article, which was reviewed by BF, PC and HF. All authors approved the final version of the article.

Declaration of conflicting interests

The authors declare that they have no conflicts of interest in the research.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The French experimentation and assessment of EBMPracticeNet was supported by the French public health insurance system (CNAMTS). CNAMTS had no involvement in study design, data analysis or interpretation, or article writing.

ORCID iD

Laurent Letrilliart

Data sharing statement

The database is available on request from the corresponding author.

References

Davies

. The information-seeking behaviour of doctors: a review of the evidence. Health Info Libr J 2007; 24: 78–94.

Haynes

. Of studies, syntheses, synopses, summaries, and systems: the ‘5S’ evolution of information services for evidence-based healthcare decisions. Evid Based Med 2006; 11: 162–164.

Alonso-Coello

Irfan

Solà

, et al. The quality of clinical practice guidelines over the last two decades: a systematic review of guideline appraisal studies. Qual Saf Health Care 2010; 19: e58.

García

Sanabria

Alvarez

, et al. The validity of recommendations from clinical guidelines: a survival analysis. CMAJ 2014; 186: 1211–1219.

Francke

Smit

de Veer

AJE

, et al. Factors influencing the implementation of clinical guidelines for health care professionals: a systematic meta-review. BMC Med Inform Decis Mak 2008; 8: 38.

Versloot

Grudniewicz

Chatterjee

, et al. Format guidelines to make them vivid, intuitive, and visual: use simple formatting rules to optimize usability and accessibility of clinical practice guidelines. Int J Evid Based Healthc 2015; 13: 52–57.

Straus

Haynes

. Managing evidence-based knowledge: the need for reliable, relevant and readable resources. CMAJ 2009; 180: 942–945.

Caniard

, The practice guidelines: a tool for dialogue, responsibility and innovation disseminating, http://www.ladocumentationfrancaise.fr/rapports-publics/024000207/index.shtml (2002, accessed May 2021).

Clerc

Ventelou

Guerville

, et al. General practitioners and clinical practice guidelines: a reexamination. Med Care Res Rev 2011; 68: 504–518.

10.

Kwan

Ferguson

, et al. Computerised clinical decision support systems and absolute improvements in care: meta-analysis of controlled clinical trials. BMJ 2020; 370: m3216.

11.

Kwag

González-Lorenzo

Banzi

, et al. Providing doctors with high-quality information: an updated evaluation of web-based point-of-care information summaries. J Med Internet Res 2016; 18: e15.

12.

Gagliardi

Brouwers

Palda

, et al. How can we improve guideline use? A conceptual framework of implementability. Implement Sci 2011; 6: 26.

13.

Varonen

Jousimaa

Helin-Salmivaara

, et al. Electronic primary care guidelines with links to cochrane reviews—EBM guidelines. Fam Pract 2005; 22: 465–469.

14.

NICE, Accreditation decisions, https://www.nice.org.uk/about/what-we-do/accreditation/accreditation-decisions (accessed May 2021).

15.

Van de Velde

Macken

Vanneste

, et al. Technology for large-scale translation of clinical practice guidelines: a pilot study of the performance of a hybrid human and computer-assisted approach. JMIR Med Inform 2015; 3: e33.

16.

Delvaux

Van de Velde

Aertgeerts

, et al. Adapting a large database of point of care summarized guidelines: a process description. J Eval Clin Pract 2017; 23: 21–28.

17.

Van de Velde

Vander Stichele

Fauquert

, et al. EBMPracticeNet: a bilingual national electronic point-of-care project for retrieval of evidence-based clinical guideline information and decision support. JMIR Res Protoc 2013; 2: e23.

18.

Challenges of evaluation in biomedical informatics (chapter 1). In: Friedman

Wyatt

(eds) Evaluation methods in biomedical informatics. 2nd ed. New York, NY: Springer, 2006, pp.1–20.

19.

Chaudhry

Wang

, et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006; 144: 742–752.

20.

Eslami Andargoli

Scheepers

Rajendran

, et al. Health information systems evaluation frameworks: a systematic review. Int J Med Inform 2017; 97: 195–209.

21.

Banzi

Liberati

Moschetti

, et al. A review of online evidence-based practice point-of-care information summary providers. J Med Internet Res 2010; 12: e26.

22.

Jousimaa

Kunnamo

Mäkelä

. Physicians’ patterns of using a computerized collection of guidelines for primary care. Int J Technol Assess Health Care 1998; 14: 484–493.

23.

Brooke

. SUS: a retrospective. J Usability Stud 2013; 8: 29–40.

24.

Bangor

Kortum

Miller

. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud 2009; 4: 114–123.

25.

Coumou

HCH

Meijman

. How do primary care physicians seek answers to clinical questions? A literature review. J Med Libr Assoc 2006; 94: 55–60.

26.

Del Fiol

Workman

Gorman

. Clinical questions raised by clinicians at the point of care: a systematic review. JAMA Intern Med 2014; 174: 710–718.

27.

Sauro

, Measuring usability with the system usability scale (SUS), https://measuringu.com/usability-loyalty/ (accessed May 2021).

28.

Brassil

Gunn

Shenoy

, et al. Unanswered clinical questions: a survey of specialists and primary care providers. J Med Libr Assoc 2017; 105: 4–11.

29.

Bernard

Arnould

Saint-Lary

, et al. Internet use for information seeking in clinical practice: a cross-sectional survey among French general practitioners. Int J Med Inform 2012; 81: 493–499.

30.

Schuers

. Behavior and attitudes of residents and general practitioners in searching for health information: from intention to practice. Int J Med Inform 2016; 89: 9–14.

31.

Phua

See

Khalizah

, et al. Utility of the electronic information resource UpToDate for clinical decision-making at bedside rounds. Singapore Med J 2012; 53: 116–120.

32.

Letrilliart

Supper

Schuers

, et al. The ECOGEN study: elements of the consultation in general practice. exercer 2014; 114: 148–157.

33.

Letrilliart

Bacis

Mennerat

, et al. Interface terminologies: a case study on the International Classification of Primary Care. Int J Soc Manage Econ Bus Eng 2009; 3: 124–127.

34.

Darzi

Abou-Jaoude

Agarwal

, et al. A methodological survey identified eight proposed frameworks for the adaptation of health related guidelines. J Clin Epidemiol 2017; 86: 3–10.

35.

Fervers

. Guideline adaptation: an approach to enhance efficiency in guideline development and improve utilisation. BMJ Qual Saf 2011; 20: 228–236.

36.

Schifferdecker

Reed

Homa

. Training intervention to improve information management in primary care. Fam Med 2008; 40: 423–432.

37.

Gagnon

Pluye

Desmartis

, et al. A systematic review of interventions promoting clinical information retrieval technology (CIRT) adoption by healthcare professionals. Int J Med Inform 2010; 79: 669–680.

38.

Addison

Whitcombe

William Glover

. How doctors make use of online, point-of-care clinical decision support systems: a case study of UpToDate©. Health Info Libr J 2013; 30: 13–22.

39.

Te Pas

van Dijk

Bartelink

MEL

, et al. Factors influencing the EBM behaviour of GP trainers: a mixed method study. Med Teach 2013; 35: e990–e997.

40.

Bouton

Leroy

Huez

, et al. Représentativité des médecins généralistes maîtres de stage universitaires. Sante Publique 2015; 27: 59–67.

41.

Letrilliart

Rigault-Fossier Fossier

, et al. Comparison of French training and non-training general practices: a cross-sectional study. BMC Med Educ 2016; 16: 126.

42.

Lenaerts

Bekkering

Goossens

, et al. Tools to assess the trustworthiness of evidence-based point-of-care information for health care professionals: systematic review. J Med Internet Res 2020; 22: e15415.