Engaging target users to appraise and refine clinical practice guidelines in pediatric concussion: An integrated knowledge translation approach

Background

Pediatric concussion is considered a serious public health concern in Canada. ¹ Between 2003 and 2013, there was a 4.4-fold increase in pediatric concussion visits in Ontario, with nearly 35,000 visits in 2013. ² The identification, assessment and management of pediatric concussion have evolved rapidly over the past decade, yet emerging evidence is not always integrated into practice. ³ Thus, it is critical that the clinical community is provided with evidence informed clinical guidelines to ensure that their concussion practices reflect a high level of care and consistency.

Clinical practice guidelines (CPGs) are systematically developed statements that assist clinicians in making evidence informed decisions regarding patient care. CPGs cover an array of patient-related domains such as screening, diagnosis, and management, in addition to informing health policy. ⁴ The overall goal of a CPG is to reduce unnecessary variations in service delivery by informing the management of a certain health condition. ⁵ Unnecessary variations in service delivery may be due to a variety of demographic factors that influence if and how a CPG is used. For example, factors such as a healthcare professional’s discipline, experience, age, and even sex, have the potential to influence the management of concussion, but have yet to be investigated in the field of pediatric concussion. ⁶

Within pediatric concussion, the Ontario Neurotrauma Foundation (ONF) released the Guidelines for Diagnosing and Managing Pediatric Concussion in 2014 (herein referred to as the 2014 CPG) with the aim to clarify and standardize concussion recommendations, anchored in empirical evidence. At the outset, the ONF committed to update the guidelines (proposed for 2019). ⁷ The 2014 CPG is one of four CPGs specific to pediatric concussion available, with one study reporting that 71.4% of appraisers indicated that they would recommend the 2014 CPG with revisions. ⁸ To date, no research has examined the various barriers to the 2014 CPG.

Updating the 2014 CPG presented a unique opportunity to assess the guidelines to inform current development activities. Beyond reviewing new knowledge generated by empirical evidence, it was key to employ a structured and systematic approach to evaluate the 2014 CPG so its evolution aligned rigorously with CPG development methodology, and to facilitate optimal dissemination and uptake of updated versions. Therefore, the use of an integrated knowledge translation (iKT) approach, defined as a co-production of knowledge through a relationship between researchers and decision makers, ⁹ was integral to improving clinical uptake of the CPG given that uptake is influenced by a myriad of factors beyond the evidence alone.

Though CPGs have the potential to improve practice and support clinical decision-making, these benefits have not consistently been achieved due to barriers to use. In addition to demographic factors, limited impact of a guideline can be attributed to vagueness, ¹⁰ lack of awareness, complexity of the guidelines, accessibility to resources and tools, and the lack of end-user involvement in development. ¹¹ For instance, it has been reported that 16–66% of pediatricians were aware of the existence of various pediatric CPGs, but only 19–28% reported a change in their practice behaviour. ¹² This supports that guidelines that are clear to understand and do not require specific resources are more likely to be implemented. ¹³ Therefore, it is evident that an iKT approach is warranted to glean the barriers limiting clinical uptake from target users presumed to use CPGs in their daily clinical practice.

Evaluating previous CPGs can be an effective and inexpensive way to increase the impact a CPG will have on clinical practices. ¹⁴ The validated Appraisal of Guidelines for Research and Evaluation II (AGREE II) tool by Brouwers et al., ¹⁵ the most commonly used guideline appraisal tool, ¹⁶ was developed to assess CPG quality and to provide a methodological framework for the development and/or revision of CPGs. ¹⁵ The original AGREE tool was developed in 2003 through a multi-staged process and was tested on 100 guidelines from 11 countries. ¹⁷ In 2010, the AGREE II tool replaced the original AGREE project with improvements to its measurement properties, usefulness to stakeholders, and ease of implementation. ¹⁵ The AGREE II tool evaluates CPGs through six domains: (1) scope and purpose; (2) stakeholder involvement; (3) rigour of development; (4) clarity of presentation; (5) applicability; and (6) editorial independence through a seven-point Likert scale. ¹⁸ According to Appenteng et al., ⁸ the highest quality CPGs are targeted guidelines for a specific population, developed by a group with clinical, research and methodological expertise, and have undergone iterations of previous editions. The process of evaluating and developing CPGs should be viewed as progressively adaptive where quality is informed by the composition and expertise of the guideline developers. ¹⁹

The current study is anchored in an iKT approach and uses clinical experts as appraisers of the 2014 CPG given their scope of responsibility to invoke practice change. The purpose of this study was to rigorously evaluate the 2014 CPG using the AGREE II tool and utilize the feedback collected from end users to inform quality improvements across content, format, development processes and dissemination to support the development of an updated CPG. The study objectives include: (1) describe the quality of the 2014 CPG through AGREE II domain scores and frequency analysis of knowledge translation (KT) questions; (2) evaluate whether AGREE II ratings of the 2014 CPG are impacted by demographic factors; and (3) interpret written responses/justifications to AGREE II scores to describe potential areas for improvement beyond frequency analysis.

Evaluating AGREE II ratings by demographic factors may inform whether the CPG meets the needs of its target users, its content is well disseminated to end users, and whether there are barriers to use for subgroups of target users. Further, engaging end users in a CPG evaluation is a valuable method to revising a CPG as all end users are aware of the updating process as appraisers, purpose of the CPG as guideline developers, and carry out the recommendations in their own practice. The outcomes of this study are critical to inform the development of an impactful and evidence-informed knowledge product that has been co-created by concussion and knowledge translation experts.

Methods

Measures

Appraisal of Guidelines for Research and Evaluation II (AGREE II) tool ¹⁸

The AGREE II tool is a 23-item questionnaire, spanning six domains: (1) scope and purpose; (2) stakeholder involvement; (3) rigour of development; (4) clarity of presentation; (5) applicability; and (6) editorial independence. A description of the assessment focus for each domain is provided in Table 1. AGREE II items are rated on a seven-point Likert scale (1 = strongly disagree to 7 = strongly agree). The AGREE II tool recommends at least two, but preferably four external reviewers. ⁴ The study aim was not to increase the amount of external reviewers and deviate from AGREE II evaluation methodology, rather the goal was to engage the perspective of a diverse group of end users, as reviewers are typically content and methodology experts.

OPEN IN VIEWER

Table 1.

Assessment description for AGREE II domains. ¹⁸

Domain		No. of items	Description
1	Scope and purpose	3	Assesses the overall aim of the guideline, specific health questions and the target population
2	Stakeholder involvement	3	Assesses the extent to which the guideline was developed by the appropriate stakeholders and represents the views of its intended user
3	Rigour of development	8	Examines the process of evidence retrieval and synthesis, as well as the methods to formulate and update the recommendations
4	Clarity of presentation	3	Assesses the language, structure and format of the guidelines
5	Applicability	4	Examines the barriers and facilitators to implementation, strategies to improve uptake, and resource implications of applying the guideline
6	Editorial independence	2	Assesses the formulation of recommendations not being unduly biased with competing interests

The AGREE II tool is the most commonly used guideline appraisal tool, ¹⁶ and it is well suited for evaluating the 2014 CPG as it is able to target any step along the healthcare continuum (i.e., identification, assessment, management, intervention) and is intended for healthcare providers and guideline developers. The AGREE II user manual does not specify thresholds that distinguish a high-quality versus low-quality guideline at this time; however, higher scores indicate higher quality CPGs. ¹⁸

AGREE II score comments

Appraisers were asked to provide comments and/or justifications in response to their rating for each AGREE II item. This was an opportunity to share strengths and constructive criticism/areas of improvement to consider for the 2019 CPG.

Demographic information

Demographic information such as age, sex, discipline and years of experience were collected.

Knowledge translation appraisal

This brief survey, purposefully developed by the research team including clinicians and knowledge translation specialists included 11 knowledge translation questions, which included participant self-reported previous use and familiarity of the 2014 CPG; influence of the 2014 CPG on concussion identification, assessment and management; facilitators and barriers to the use of the 2014 CPG; and recommended tools to enhance future clinical uptake. These KT questions utilized Likert scale responses.

Results

Appraisal of the 2014 CPG Using the AGREE II Tool

The results of the AGREE II appraisal stratified by demographic variables and KT survey items are provided in Table 2. Table 3 contains a comprehensive overview of the AGREE II ratings by domain and item, a summary of the appraiser justifications, and CPG updates provided by an ONF guideline developer. Note that the N did not remain consistent throughout the survey, as there was some attrition with survey participation (i.e., number of responses ranged from 21 to 26 per AGREE II item).

OPEN IN VIEWER

Table 2.

Mean scaled domain scores (as percentages) of the AGREE II tool, stratified by demographic factors.

Mean Scaled Domain Score, % (95% CI)
Demographic variables	N	Scope and purpose	Stakeholder involvement	Rigour of development	Clarity of presentation	Applicability	Editorial independence
Sex
Male	13	81.3 (57.8–100.0)	72.8 (41.9–100.0)	82.8 (65.6–100.0)	86.1 (54.7–100.0)	80.7 (48.0–100.0)	99.0 (60.4–100.0)
Female	18	80.4 (47.6–100.0)	73.8 (40.2–100.0)	75.3 (60.4–90.3)	72.2 (50.4–94.1)	62.2 (43.7–80.7)	82.7 (76.9–88.5)
Age, years (M = 50.0 ± 9.6 years)
<40	3	87.0 (61.1–100.0)	83.3 (52.5–100.0)	83.3 (66.9–99.7)	79.2 (47.7–100.0)	75.0 (38.2–100.0)	94.4 (81.1–100.0)
40–49	11	71.7 (35.8–100.0)	65.3 (27.0–100.0)	77.1 (59.3–94.9)	81.5 (61.5–100.0)	75.0 (56.5–93.5)	86.1 (54.0–100.0)
≥50	17	86.3 (69.0–100.0)	76.1 (48.4–100.0)	77.6 (61.2–94.0)	75.0 (42.1–100.0)	64.9 (33.0–96.8)	88.9 (52.3–100.0)
Primary discipline
Physician	13	83.9 (51.3–100.0)	79.4 (48.7–100.0)	87.8 (76.1–99.5)	88.2 (67.3–100.0)	83.3 (67.3–99.4)	99.0 (93.1–100.0)
Psychologist/Neuropsychologist	4	77.8 (47.0–100.0)	63.6 (33.2–94.1)	76.4 (62.0–90.9)	63.9 (29.1–98.6)	59.4 (26.0–92.8)	85.4 (56.0–100.0)
Researcher	6	75.6 (47.2–100.0)	72.2 (41.4–100.0)	59.1 (37.2–81.0)	73.6 (36.8–100.0)	56.3 (13.7–98.8)	70.8 (14.7–100.0)
Nurse Practitioner	1	66.7 (N/A)	33.3 (N/A)	54.2 (N/A)	66.7 (N/A)	37.5 (N/A)	58.3 (N/A)
AHPa	7	84.3 (70.1–98.4)	75.6 (40.8–100.0)	84.3 (72.6–96.1)	76.4 (58.6–94.2)	71.9 (54.2–89.5)	97.9 (88.5–100.0)
Years of experience in discipline, years (M = 21.9 ± 10.3 years)
<10	3	90.7 (64.8–100.0)	90.7 (64.8–100.0)	81.2 (58.8–100.0)	94.4 (83.6–100.0)	86.1 (64.0–100.0)	100.0 (N/A)
10–19	10	71.0 (34.8–100.0)	61.9 (24.8–99.0)	79.7 (66.0–93.5)	72.2 (50.0–94.4)	66.7 (40.4–92.9)	83.3 (53.5–100.0)
≥20	18	84.9 (66.2–100.0)	75.4 (47.9–100.0)	77.2 (60.7–93.7)	75.4 (43.4–96.9)	66.4 (35.8–96.9)	88.1 (52.6–100.0)
Years of experience in pediatric concussion, years (M = 14.1 ± 7.7 years)
<10	8	80.6 (55.0–100.0)	73.0 (37.9–100.0)	70.4 (48.8–91.9)	77.8 (46.2–100.0)	60.7 (21.2–100.0)	79.8 (34.7–100.0)
10–19	13	79.8 (46.0–100.0)	72.8 (46.0–100.0)	79.0 (64.1–93.8)	83.3 (62.6–100.0)	76.2 (58.6–93.8)	86.9 (57.2–100.0)
≥20	10	82.5 (65.9–99.2)	74.6 (54.1–95.1)	85.1 (74.0–96.3)	71.4 (36.6–100.0)	70.8 (45.6–96.0)	100.0 (N/A)
Setting of use
Acute care	13	87.2 (73.1–100.0)	72.2 (48.5–100.0)	82.4 (68.3–96.6)	82.7 (61.4–100.0)	76.4 (55.5–97.3)	93.5 (73.3–100.0)
Rehabilitation	8	69.1 (30.6–100.0)	53.7 (18.5–88.9)	76.8 (60.7–93.0)	56.7 (27.8–85.5)	61.7 (31.0–92.3)	95.0 (83.5–100.0)
Policy	3	87.0 (61.1–100.0)	87.0 (59.7–100.0)	79.7 (60.9–98.5)	81.5 (53.4–100.0)	81.9 (59.3–100.0)	88.9 (59.1–100.0)
Education	7	76.2 (50.7–100.0)	76.9 (41.4–100.0)	82.5 (66.7–98.3)	78.9 (57.6–100.0)	76.7 (58.6–94.7)	85.0 (50.4–100.0)
Research	13	88.9 (67.3–100.0)	79.6 (50.9–100.0)	75.8 (56.1–95.6)	84.0 (56.0–100.0)	70.4 (37.6–100.0)	82.4 (38.1–100.0)
Contribution to 2014 CPG
Yes	13	75.5 (43.2–100.0)	73.2 (41.3–100.0)	74.6 (57.2–92.1)	70.1 (36.3–96.4)	63.5 (30.7–96.4)	84.4 (41.8–100.0)
No	18	85.3 (63.5–100.0)	73.5 (42.3–100.0)	80.7 (64.8–96.7)	82.1 (57.0–100.0)	72.8 (45.5–100.0)	91.7 (67.8–100.0)
Overall	31	80.8 (53.5–100.0)	73.4 (41.9–100.0)	78.3 (61.9–94.7)	77.5 (48.6–100.0)	69.3 (40.2–98.4)	88.9 (56.7–100.0)

^aAllied health professional.

OPEN IN VIEWER

Table 3.

AGREE II score results (%), score comment summaries and improvements in updated CPG. If no n is indicated in the comments, one appraiser provided the feedback.

Scaled item score (Median)	Summary of qualitative comments	Future improvements for updated CPG
Domain 1. Scope and Purpose, Mean scaled domain score = 80.8%
1. The overall objective(s) of the guideline is (are) specifically described (N = 26)
83.3 Med= 6	• Objective is well written/clear/concise (n = 16) • Hyperlinks make CPG easy to navigate• Concern that purpose is medically based with no education implications• Objective should be closer to beginning of CPG (n = 2) • Consider changing headings (ex., “Target Users and Population” to “Who are the guidelines for?”)• More assistance with persistent symptoms would be appreciated	• Content is now organized into a website, which enables ease of accessing specific information (e.g., objectives). A search function will be available to tailor content inquiry. • Hyperlinks will be embedded throughout the website to allow for easy access and flow to documents.• Headers have been changed (e.g., “Target Users and Population”) so that they are more intuitive and specific with respect to content presented.
2. The health question(s) covered by the guideline is (are) specifically described (N = 26)
78.2 Med= 6	• CPG clear/well described/concise given vagueness of concussion (n = 14) • Purpose and context for each recommendation need more detail (n = 3)• More assistance for family physicians• When guideline is targeting different stakeholders, it is difficult to address each person’s exact needs• Adequate if for healthcare only• No case studies • Hyperlinks good for specific questions, but cumbersome for cluster symptoms; algorithms better for short appts (n = 2)• Tip sheets useful, but repetitive• Presentation of who, where, why are a bit confusing	• The domains have been organized according to three sections within the CPG. An introductory paragraph has been written for each section to provide context regarding the domains covered in each section.• The updated guideline targets only healthcare providers (HCPs), the primary target audience.• Derivative products are being made for parents, and the sports and education communities to capture content from the CPG that is important and relevant to them.• The presentation of who, where, why has been removed from this guideline. Rather, recommendations have been written more concisely.
3. The population to whom the guideline is meant to apply is specifically described (N = 26)
80.8 Med= 6.5	• Population is clear/exclusions and age ranges described/easy to find (n = 15) • Specific age groups not well delineated from developmental perspective (i.e., school info–grade school vs. high school)• More assistance for family doctors required• Need to add speech-language pathologists & educational specialists as part of the concussion team• Might be better to separate primary users in one guideline and secondary/tertiary to another• Could be better formatted to be more clear• Should be moved up before “Target population” section	• A speech language pathologist and an expert in special education were part of the expert panel. • The updated guideline targets HCPs only as the primary target audience for this guideline.• Derivative products are being made for parents, and the sports and education communities to capture content from the CPG that is important and relevant to them.• The CPG is being translated into a website format, which will help to enhance the clarity of the presented content. A KT specialist and graphic designer have been engaged to enhance the clarity and presentation of the content.
Domain 2. Stakeholder Involvement, Mean scaled domain score = 73.4%
4. The guideline development group includes individuals from all relevant professional groups (N = 24)
75.0 Med= 6	• Clear and sufficiently detailed/diverse (n = 12) • Good location in guideline• Would be better if diversified expert panelists outside Ontario (n = 3), all specialists from urban areas• Should include: methodology expert, youth, family leader, family medicine, primary care provider, social work, clinical psychologist, developmental pediatrics, educators, sports coaches, CCC• Knowledge translation to general community could be improved	• There was national and international representation on the expert panel. • The following were represented on the expert panel: youth advisor, family medicine, primary care provider, social work, clinical neuropsychology, and special educators. Members of the expert panel have expertise in developing CPGs, and so provide methodology expertise.• KT support has been bolstered by having a KT Specialist on the expert panel. Supports created to foster KT into the community include a document to support sharing and using the guideline in clinical practice.
5. The views and preferences of the target population have been sought (N = 24)
63.2 Med= 5	• Clear (n = 7) and easy to find• Clear that the CPG is primarily for doctors to use and merely a resource for parents to further recognize symptoms/signs• Should include school personnel in updated CPG• Front-line family physicians not considered, unclear if teachers were considered• Family, youth, lived-experience not considered creating a lack of clear description of stakeholders (n = 5)• Knowledge translation to general community could be improved	• Special education was represented in the new CPG.• KT support has been bolstered by having a KT Specialist on the expert panel. • Derivative products are being made for parents, and the sports and education communities to capture content from the CPG that is important and relevant to them. Family leaders, coaches and teachers actively participated in the creation of these products.
6. The target users of the guideline are clearly defined (N = 24)
81.9 Med= 6	• Clearly defined (n = 14)• Need more guidance for family physicians• Should include pediatricians and athletic therapists• Somewhat difficult to navigate at times• Document not friendly for secondary/tertiary users such as parents/caregivers (n = 3)	• Athletic therapists and pediatricians were included in the expert panel.• Navigation of content will be improved by having the content translated into website format.• The guideline is for HCPs only as they are the primary audience. • Derivative products are being made for parents, and the sports and education communities to capture content from the guideline that is important and relevant to them.
Domain 3. Rigour of Development, Mean scaled domain score = 78.3%
7. Systematic methods were used to search for evidence (N = 22)
84.1 Med= 6	• Rigorous and well-described (n = 16)• Search terms not included (n = 2), no clear methodological description• Search not reproducible• Simply not many systematic reviews on the topic	• Scope of the search and inclusion/exclusion criteria is clearly outlined.
8. The criteria for selecting evidence are clearly described (N = 22)
80.3 Med= 6	Clearly described (n = 13)• Level of evidence pyramid used unclear• Description could have been more clear (n = 4) by providing justification, inclusion/exclusion criteria• Difficult to find in the document	• The CPG will be housed on braininjuryguidelines.org. This site is the main hub where all ONF guidelines specific to concussion and m/TBI are being housed.• A communication strategy will be put into place to promote the dissemination of the CPG.
9. The strengths and limitations of the body of evidence are clearly described (N = 22)
77.3 Med= 6	• Well written without bias, covers basic strengths and limitations (n = 10)• No definition of level of evidence scheme• Strengths and weaknesses not clear (n = 3), strengths and weaknesses stated but not specific• Additional description for secondary/tertiary users may be warranted• More information needed in some domains (social/behavioural, neuroimaging) and for some ages (youngest/oldest in population)	• The information provided reflects the current state of the evidence. This CPG will now operate as a living guideline, in which the recommendations and domains will be updated as the evidence emerges.• There are no longer secondary/tertiary users of the guideline. HCPs are the primary and only guideline users.
10. The methods for formulating the recommendations are clearly described (N = 21)
84.9 Med= 7	• Well described (n = 17)• Clear and concise (n = 8)• Not described at all• Absence of solid evidence• Possibly too much detail/could be put in an appendix (n = 2)	• A clear and stepwise methodology section has been added to the CPG.• Algorithms were developed in an effort to decrease the quantity of information and better guide clinical decision-making.
11. The health benefits, side effects and risks have been considered in formulating the recommendations (N = 21)
66.7 Med= 5	• More examples of benefits and risks/more explicitly defined (n = 5) • Unclear/needs to be updated (n = 3)• Option of A vs. B, standard vs. more conservative approach could include more context
12. There is an explicit link between the recommendations and the supporting evidence (N = 21)
74.6 Med= 6	• Yes, explicit link (n = 12)• No link (n = 2)• Must be updated/content was good at the time (n = 3)• Strong attempt	• The information provided reflects the current state of the evidence. The CPG will now operate as a living guideline, the recommendations and domains will be updated as the evidence emerges.
13. The guideline has been externally reviewed by experts prior to its publication (N = 21)
78.6 Med= 6	• Appropriate reviewers (n = 13)• Could not find/unclear if completed (n = 3)• Should have consulted general population (n = 2)• Could have used a few more external reviewers	• The guideline has been externally reviewed by two experts; this is stated explicitly in the guideline.
14. A procedure for updating the guideline is provided (N = 21)
79.4 Med= 6	• Description provided (n = 16)• Noted not on time (n = 3)• Not done	• This CPG is now going to be a living guideline; the recommendations and domains will be updated as the evidence emerges.
Domain 4. Clarity of Presentation, Mean scaled domain score = 77.5%
15. The recommendations are clear and unambiguous (N = 21)
76.2 Med= 6	• Straightforward and clear (n = 12)• Vague (n = 5)• Recommendations regarding when to refer to multi-disciplinary clinics would be helpful• Statements clear, but disagree with who is listed as primary “go to persons” and believe the who needs to change in some sections	• Prompts for multidisciplinary referral are now based on the HCPs scope of practice.
16. The different options for management of the condition or health issues are clearly presented (N = 21)
77.8 Med= 6	• Clearly presented/easy to find (n = 17), broadness is good so physician can use numerous approaches• Not found• Algorithms provided are useful, however they seem embedded in the document• Put the specific recommendations near the front of the CPG since many clinicians might only review the topline recommendations and miss the details further on in the document• Well-presented but could use more clarity• Too much reliance on user knowing when/where to seek additional information• Needs to be updated	• Recommendations will now be presented in web based format in addition to a search/find function to look for specific content.• A PDF version of the recommendations will also be made available for download.
17. Key recommendations are easily identifiable (N = 21)
78.6 Med= 6	• Clearly presented (n = 16)• In a few instances, school personnel are not adequately included• Some content no longer relevant• Suggest an executive/lay summary instead of algorithm• Too much repetition–could be more concise• Could improve language, font, graphics (n = 2)• All recommendations have same weighting, suggest that CPG highlights most important points	• A graphic designer and KT specialist have been engaged to improve the language, font, and graphics of the CPG and the website.• A derivative product for the education community is being developed, where school personnel will be engaged to inform its content.• Content has been written in such a way to minimize repetition and enhance clarity.
Domain 5. Applicability, Mean scaled domain score = 69.3%
18. The guideline describes facilitators and barriers to its application (N = 21)
70.6 Med= 6	• Well described (n = 13)• Length of guideline may limit usage• Information on how to overcome barriers is lacking• Too broad (n = 2)• Not easy to find	• CPG has been transformed into a website, which will enhance accessibility of content. • The CPG will be housed on braininjuryguidelines.org. This site is the main hub where all ONF guidelines specific to concussion and m/TBI are housed.• A communication strategy will be put into place to promote the dissemination of the CPG.
19. The guideline provides advice and/or tools on how the recommendations can be put into practice (N = 21)
82.5 Med= 7	• Practical recommendations (n = 16)• Need to provide more resources for family physicians in smaller communities that lack resources• Yes, but some info in the tools provides slightly different information which is confusing. Would be better to stick with a few good sources than include everything• Lots of tools, but who and how to use and interpret the results are not clear• Could be organized better to improve access	• Links to resources are provided throughout the recommendations for community HCPs to access.• Guideline has been transformed into a website, which will enhance accessibility of content, in addition to a search/find function to look for specific content.
20. The potential resource implications of applying the recommendations have been considered (N = 21)
53.2 Med= 5	• Agree (n = 3) • Not relevant for many of the guidelines• Not specifically addressed, not emphasized (n = 12)• Would be helpful to consider time constraints and level of training of end-users	• CPG includes language highlighting the context for which recommendations are best applied (e.g., use of multidisciplinary concussion teams when managing persistent post-concussion symptoms)
21. The guideline presents monitoring and/or auditing criteria (N = 21)
70.6 Med= 6	Included/clearly indicated (n = 14) • Somewhat brief• No tools for measuring the application of guidelines themselves (n = 3) • Desired outcomes for long term community participation may need additional consideration for students with persistent symptoms	• ONF will be collaborating with CPG co-leads to establish implementation initiatives.
Domain 6. Editorial Independence, Mean scaled domain score = 88.9%
22. The views of the funding body have not influenced the content of guideline (N = 21)
88.9 Med= 7	• No evidence of undue influence (n = 19) • Statement about the goals of ONF aligning with the guidelines—is this contradictory?	• CPG developers have employed the same standard to ensure there is no undue influence of guideline content. For example, funders were welcome to attend the consensus conference but were not included in the appraisal of research or in the revision of guidelines.
23. Competing interests of the guideline development group members have been recorded and addressed (N = 21)
88.9 Med= 7	• Clearly stated (n = 17) • Did not see• Unclear if this is the case	• Conflicts of interest were sought from all expert panel members. • A specific section that addresses conflicts of interest was added to the website.

Domain 1: Scope and purpose

Participants provided a mean overall rating of 80.8% (95% CI, 53.5–100.0%) in Domain 1, reflective of the 2014 CPG’s ability to demonstrate adequate scope and purpose. Physicians and allied health professionals provided the highest ratings (83.4% and 84.3%; 95% CI, 51.3–100.0% and 70.1–98.4%, respectively), while other professions (psychologists/neuropsychologists, researchers, one nurse practitioner) provided ratings approximately 6–17% lower. Participants with less than 10 years of experience in their discipline provided a higher rating (90.7%; 95% CI, 64.8–100.0%) than those with more than 10 years of experience who provided mean domain ratings between 71.0–84.9%. Additionally, participants using the CPG in rehabilitation (69.1%; 95% CI, 30.6–100.0%) and education (76.2%; 95% CI, 50.7–100.0%) settings rated this domain lower compared to those in acute care, research and policy settings (range of mean scores, 87.0–88.9%). Many appraisers (n = 16) stated that the scope and purpose were well described although they noted that more assistance with persistent symptoms was necessary (i.e., concussion management) and that the health questions of all stakeholders were not met as there were too many stakeholders considered.

Domain 2: Stakeholder involvement

The mean Stakeholder Involvement score across all participants was 73.4% (95% CI, 41.9–100.0%). Again, physicians and allied health professionals provided the highest overall scores in this domain (79.4% and 75.6%; 95% CI, 48.7–100.0% and 40.8–100.0%, respectively), while other professions (psychologists/neuropsychologists and one nurse practitioner) provided mean scores between a range of 33.3–63.6%. Participants with less than 10 years of experience in their respective discipline provided the highest rating (90.7%; 95% CI, 64.8–100.0%), while those with greater than 10 years of experience provided a rating that was approximately 15–28% lower. Use of the 2014 CPG in a rehabilitation setting provided the lowest rating of 53.7% (95% CI, 18.5–88.9%). Appraisers commented that the guideline development group could be more diverse, both disciplinarily and geographically, and that there was a clear lack of consideration from families and youth with lived experience (n = 5). Common feedback in this domain was that the CPG was not friendly for secondary/tertiary users.

Domain 3: Rigour of development

The mean domain score for Rigour of Development across all participants was 78.3% (95% CI, 61.9–94.7%). Physicians and allied health professionals provided the highest overall score in this domain (87.8% and 84.3%; 95% CI, 76.1–99.5% and 72.6–96.1%, respectively), while other professions (researchers and one nurse practitioner) provided mean domain scores ranging between 54.2–59.1%. Participants with less than 10 years of experience in concussion rated this domain at 70.4% (95% CI, 48.8–91.9%) while those with greater than 10 years of experience in concussion provided a mean scoring range between 79.0–85.1%. In terms of rigour of development, many appraisers commented that the methods and evidence were well-described (n = 16). Appraisers (n = 13) stated that the external reviewers were appropriate although some thought that the general population should have been consulted (n = 2).

Domain 4: Clarity of presentation

The mean Clarity of Presentation score across all participants was 77.5% (95% CI, 48.6–100.0%). Males provided a higher rating (86.1%; 95% CI, 54.7–100.0%) than females (72.2%; 95% CI, 50.4–94.1%). Physicians provided the highest rating (88.2%; 95% CI, 67.3–100.0%) while psychologists/neuropsychologists and one nurse practitioner provided the lowest ratings (63.4% and 66.7%; 95% CI, 29.1–98.6% and N/A, respectively). Participants with less than 10 years of experience in their primary discipline provided the highest rating (94.4%; 95% CI, 83.6–100.0%), while those with greater than 10 years of experience provided ratings approximately 22–25% lower. Further, those with greater than 20 years of pediatric concussion experience provided the lowest rating of 71.4% (95% CI, 36.6–100.0%) compared to those with less experience. Finally, participants from a rehabilitation setting provided the lowest rating of 56.7% (95% CI, 27.8–85.5%), compared to other settings which provided higher mean scores ranging between 78.9–84.0%. Appraisers (n = 17) stated that the presentation of the recommendations was straightforward and clear, but five appraisers believed the recommendations to be too vague. As well, two appraisers noted that the language, font and graphics could be improved for better clarity.

Domain 5: Applicability

The mean score across all participants for Domain 5 was 69.3% (95% CI, 40.2–98.4%). Males provided a higher rating (80.7%; 95% CI, 48.0–100.0%) compared to females (62.2%; 95% CI, 43.7–80.7%). Physicians and allied health professionals provided higher ratings compared to ratings by psychologists/neuropsychologists (59.4%; 95% CI, 26.0–92.8%), researchers (56.3%; 95% CI, 13.7–98.8%), and one nurse practitioner (37.5%; 95% CI, N/A). Participants with less than 10 years of experience in their primary discipline provided the highest applicability rating (86.1%; 95% CI, 64.0–100.0%), yet those with less than 10 years of pediatric concussion experience provided the lowest applicability rating (60.7%; 95% CI, 21.2–100.0%). Finally, participants from a rehabilitation setting provided the lowest rating of 61.7% (95% CI, 31.0–92.3%), compared to other settings, which provided higher mean ratings ranging between 70.4–81.9%. Appraisers reported that while the 2014 CPG describes the facilitators and barriers to use (n = 13), additional barriers such as the CPG’s length exists. Appraisers identified that while many tools were provided to support implementing the recommendations into practice, some content between tools conflicted. Additionally, who should use the tools and how to interpret the results of the tools was unclear.

Domain 6: Editorial independence

The mean score across all participants for Domain 6 was 88.9% (95% CI, 56.7–100.0%). Females provided a lower mean score (82.7%; 95% CI, 76.9–88.5%) compared to males (99.0%; 95% CI, 60.4–100.0%). Researchers and one nurse practitioner rated editorial independence (70.8% and 58.3%; 95% CI, 14.7–100.0% and N/A, respectively) lower than that of the remaining professions. Interestingly, those with less than 10 years of pediatric concussion experience provided the lowest rating of 79.8% (95% CI, 34.7–100.0%), while those with greater than 10 years of concussion experience provided higher mean ratings ranging between 86.9–100.0%. Most appraisers (n = 19) noted that there was no undue influence by the funding body and that any competing interests were clearly stated (n = 17).

Discussion

The primary objective of the study was to evaluate the ONF’s Guidelines for Diagnosing and Managing Pediatric Concussion using the AGREE II tool in order to inform its update. Overall, the findings revealed that the 2014 CPG was acceptable across domains, with variations within domain scores suggested to be accounted for by various demographic factors examined in this study. The findings not only support the use of the AGREE II tool to ensure standards for a high-quality CPG were being met through critical appraisal by end users, but also to inform the development of the updated CPG to optimize implementation of of the revised 2019 CPG. While evaluating CPGs using the AGREE II tool is a widespread practice, the engagement of end users within an iKT lens has not been carried out in the field of pediatric concussion. This study makes a notable contribution to understanding how end user demographic factors influence CPG use and acceptability. The iKT framework engaged end users in the evaluation of the 2014 CPG, which enabled maximum participation in both the conception and future implementation of the 2019 CPG. The implication of this work is to prompt clinical communities to expand how they appraise and revise a CPG. As evidenced in this study, a high external reviewer score does not necessarily reflect optimal clinical uptake or acceptability to end users. Thus, it behoves the pediatric concussion community to take a more systematic approach to investigate factors limiting CPG usage.

The AGREE II appraisal of the 2014 CPG was in line with other pediatric concussion CPGs across the world. A study by Appenteng et al. ⁸ reported mean domain AGREE II scores for Scope and Purpose, Stakeholder Involvement, Rigour of Development, Clarity of Presentation, Applicability, and Editorial Independence of 86.1%, 55.7%, 46.7%, 81.0%, 36.2%, and 43.2%, respectively for CPGs that addressed only pediatric concussion or mild traumatic brain injury.^7,27–29 In the current study, the quality ratings from appraisers were in line with or higher than these reported ratings in the literature.

The KT appraisal survey identified some key areas to consider for the updated version of the CPG. Firstly, over 90% of appraisers reported that they did not regularly use the 2014 CPG, which is of concern but may be improved. For example, clinician adherence to CPGs in brain injury rehabilitation in a brain injury rehabilitation unit in Melbourne, Australia with no active KT interventions was approximately 39%. With a >12-month audit and feedback intervention, adherence increased to 84%. ³ Thus, removing identified barriers to use (i.e., knowledge of the guideline, length of the guideline, and relevance) and implementing active KT interventions may lead to more frequent use by healthcare professionals. For the updated ONF guidelines, the use of KT specialists with dissemination and implementation expertise will help with widespread distribution of the updated CPG. The length barrier will be removed as the updated CPG will be on a website with sub-sections, making the information more digestible and less repetitive. Relevance of recommendations in practice settings will be improved based on more explicitly defining the target population of healthcare professionals to have more specific and relevant recommendations. Further, from the KT appraisal survey, an important finding was that healthcare professionals (i.e., physicians and allied health professionals), the new target users of the updated CPG, reported that the 2014 CPG was highly influential in identifying, assessing and managing concussions. This suggests that the guideline developers are well positioned to create an impactful, evidence-informed CPG for their target audience.

The combined analysis of the quantitative AGREE II domain ratings and qualitative appraiser feedback informed necessary updates to the 2014 CPG as outlined previously through a systematic approach. The use of end users focussed and streamlined the process. The Scope and Purpose domain of the 2014 CPG appeared to effectively convey objectives in a clear and concise manner. In terms of Stakeholder Involvement, there was some concern regarding specifically outlining the target population, which has now been explicitly addressed by expanding the guideline development team, addressing healthcare professionals as the primary user, and creating derivative products specific to secondary/tertiary users (i.e., parents, sport and education communities) previously included in the 2014 CPG. Rigour of Development has been improved by having the CPG operate as a living document on the ONF website, in which recommendations and domains will be updated as the evidence emerges. Having a living CPG on the ONF website addresses concerns about difficulty finding information with the addition of a search function, having too much or too little detail as information is now on easily digestible sub-pages, and the information being up-to-date as the CPG is a living document and will be updated as the evidence emerges. Clarity of Presentation has been enhanced by engaging a graphic designer and KT specialist to improve the language and design of the website and derivative products. Low Applicability ratings have been addressed; the new website will enhance accessibility of the guideline content and include language highlighting the context for which recommendations are best applied. Regarding Editorial Independence, guideline developers provided an unbiased environment to develop the guidelines, which set a high standard to ensure no undue influence.

In line with the literature, the Applicability domain had the lowest rating,^8,30–32 where low ratings are often attributed to a lack of noted facilitators or barriers to CPG use and vague implementation guidelines. ³⁰ CPGs in other healthcare settings such as evidence-based traditional Korean medicine, ³¹ as well as nutrition ³² reported mean scaled Applicability domain ratings of the CPGs within their fields between 20–63% and 59–99%, respectively. One must note that it is unknown if the variability in scores can be solely attributed to the actual quality of the CPGs within these different healthcare settings since the current study revealed differences in AGREE II scores between rater demographic factors. However, the mean Applicability domain scores for the 2014 CPG fall above or within other CPGs in the healthcare field.

Unique to the field of pediatric concussion was the use of appraisers from various disciplines that were both guideline developers and end users, as well as the high number of appraisers engaged. The engagement of these stakeholders brings value of learning from the use and development of prior iterations. Results from this unique sample were stratified according to experts’ demographic factors and experience. Similar trends were found across domains. For example, physicians and allied health professionals consistently provided higher ratings across domains compared to other professions. This finding may be due to the recommendations in the 2014 CPG being aligned more explicitly with these professions compared to other professions (i.e., neuropsychologists, nurse practitioner, researchers), in which recommendations may have been perceived as ambiguous. Interestingly, less years of experience in an appraiser’s respective discipline revealed stronger ratings across the Scope and Purpose, Stakeholder Involvement, Clarity of Presentation, and Applicability domains compared to more years of experience. These domains (aside from Stakeholder Involvement) involve the use of the 2014 CPG rather than its development. It is suspected that more experience is associated with access to a greater breadth of knowledge for which to compare the current state of evidence and temporal patterns of clinical care resulting in less relevance of the scope/purpose and applicability of the contents of the 2014 CPG to more experienced appraisers. The remaining domains had high scores across all experience groups. Additionally, those practicing in a rehabilitation setting provided much lower domain ratings compared to those in other settings such as primary care, education, policy and research. Here, appraisers’ comments alluded to an inadequate representation of other experts, which have now been included in the development of the updated CPG. This finding may also merely reflect the difference in clinical care offered by primary care versus rehabilitation settings. Primary care settings may see pediatric concussion patients less frequently and in the acute phase of concussion, readily employing the decision-making matrices to assess the injury. Conversely, rehabilitation settings likely observe the complexities of concussion recovery (i.e., physical, social, psychological/emotional and cognitive sequalae), which may enhance the difficulty of deciding what ‘best practice’ intervention to apply resulting in more difficulty of use of the 2014 CPG. The vagueness of the target users in the 2014 CPG may have accentuated the lack of alignment with the objectives, stakeholder involvement, rigour of development, clarity of instructions and applicability for appraisers using the 2014 CPG in rehabilitation settings.

As stated by Brouwers et al., ¹⁵ the potential benefits of CPGs are only as good as the quality of the guideline itself, which is why it is critical to not only appraise CPGs but to involve key stakeholders in an integrated approach. Engaging in an iKT approach to CPG appraisal, as done in this study, brings greater accountability and credibility to practice recommendations, especially given that primary users of the guideline play a role in its development. ⁹

Limitations

While the study provides a rigorous approach to revising a CPG, it is not without limitations. While the use of appraisers who were also end users was a unique strength of this study, some appraisers had contributed to the development of the 2014 CPG, which may enter bias into their scoring. Wilcoxon Rank-Sum tests were conducted for each domain between groups to investigate this bias, however no difference between the groups was measured. This study presented AGREE II domain ratings; however, the AGREE II user manual does not specify thresholds that distinguish a high-quality versus low-quality guideline, making it difficult to objectively confirm the quality of the evaluated CPG. ¹⁸ The main purpose of the AGREE II tool is to assess the quality of structure and content, ³⁰ but it does not explicitly evaluate the clinical utility/appropriateness of the recommendations themselves. Thus, high quality CPG content and structure does not guarantee better health outcomes for the target clinical population, ³⁰ although it acts as a useful tool to inform usability while adding context that unless used, better health outcomes cannot be achieved.

Conclusion

High-quality CPGs reflect the perspectives of end users and other relevant stakeholder groups, are based in empirical evidence, are critically reviewed, and reflect specific recommendations. ¹⁵ This study provided a foundational iKT framework to appraise a CPG, while honing in on key factors that make a strong CPG in the context of pediatric concussion. Poor uptake of CPGs continues to be a critical barrier to utilizing evidence informed information to make clinical decisions. Through an iKT approach, the critical appraisal of CPGs and subsequent KT strategies provide a foundation for which to revise and implement an updated CPG to more appropriately reach and be used by the target audience of healthcare providers. Further, CPG recommendations need to be simple and clear to minimize the cognitive load and increase uptake. This study supported that ease of access, succinct information presentation and a user-friendly approach are key to enabling target users to implement the CPG. Demographic factors within end users are important to consider given their impact on domain score ratings and CPG use. Thus, engaging end users in the CPG appraisal resulted in these important outcomes: (1) identified areas of the guidelines that may have affected the lack of previous clinical uptake through a mixed-methods, integrated knowledge translation study design while abiding by a clinical practice guideline development framework; (2) shared and informed decision making regarding content and format of revised clinical practice guidelines; and (3) targeted content, clinical questions and dissemination strategies, which are key to clinical uptake.