The Voices of Stakeholders Involved in Precision Medicine: The Co-Design and Evaluation of Qualitative Indicators of Intervention Acceptability,Fidelity and Context in PRecISion Medicine for Children With Cancer in Australia

Abstract

We report a novel approach of amalgamating implementation outcomes of acceptability and fidelity alongside context as a new way of qualitatively evaluating implementation outcomes and context of a precision medicine intervention. A rapid qualitative online proforma was co-designed with stakeholders and sent to a purposive sample of healthcare professionals involved in an early-phase clinical trial intervention. Data were analysed using Framework Analysis. A total of 24 out of 68 proformas were returned. Although some participants raised concerns about drug medication access issues, the main intervention was well accepted and understood across professional groups. Comprehension was enhanced through exposure to specialist multidisciplinary meeting arrangements. In conclusion, a rapid data collection tool and framework are now available to assess readily measurable, qualitative indicators of acceptability, fidelity of receipt and contextual fit within the dynamic precision medicine context.

Keywords

behaviour change behaviour cancer complexity qualitative

Background

The PRecISion Medicine for Children with Cancer trial intervention (PRISM trial) was designed in Australia to support children and adolescents with a high-risk or relapsed malignancies (less than 30% chance of surviving 5 years after their diagnosis) (Rapport, Smith, et al., 2020). PRISM is a national clinical trial within a Precision Medicine Program called the Zero Childhood Cancer Program (ZERO) (Rapport, Smith, et al., 2020). ZERO and its embedded early-phase clinical trial PRISM provide personalised treatment recommendations for children and adolescents with high-risk or relapsed malignancies (Rapport, Smith, et al., 2020). PRISM uses multiomics which includes whole genome sequencing (WGS), RNA sequencing (RNASeq) and methylation profiling (Wong et al., 2020). Beyond the positives of giving hope to children and adolescents with the most aggressive cancers with little or no treatment options, high-technology medical research is not without its critics. For example, the promise of medical advances is often in tension with the actual performance of these advances (Brown, 2005). A lack of provider acceptance of precision medicine by healthcare professionals (e.g. clinicians) has been noted (Ginsburg & Phillips, 2018; Manolio et al., 2013; Manolio et al., 2019) with criticism over a lack of rigorous evidence of precision medicine’s success for most childhood cancers (Evans et al., 2020). This is important from an implementation science perspective because strong provider acceptability of the appropriateness of an intervention is essential for ensuring the success of implementation efforts (Palsola et al., 2020; Weiner et al., 2017). Furthermore, Curran (2020) indicates that implementation science has its own primary outcome that is distinct from clinical/preventative outcomes used in effectiveness research, with ‘the thing’ (e.g. intervention), how we support people to ‘do the thing’ (e.g. implementation strategy) and the ‘outcome’ (e.g. acceptability and fidelity).

Identifying provider barriers around acceptability can help to optimise an intervention (Palsola et al., 2020). However, implementing new interventions that are acceptable to providers can be challenging (Anderson et al., 2021). According to Sekhon et al. (2017), acceptability reflects the extent to which people delivering a healthcare intervention consider it appropriate, based on anticipated or experienced cognitive-affective responses to an intervention. The Theoretical Framework of Acceptability developed by Sekhon et al. (2017) defines acceptability according to seven constructs: (1) affective attitude (feelings about taking part in an intervention), (2) burden (effort required in an intervention), (3) perceived effectiveness (likelihood of an intervention achieving its purpose), (4) ethicality (fit with a value system), (5) intervention coherence (understanding the intervention and how it works), (6) opportunity costs (what must be given up to engage in the intervention) and (7) self-efficacy (confidence that the behaviours required to participate in the intervention can be performed). These seven constructs can be used to assess the factors influencing intervention acceptability from the healthcare professional’s perspective and can be assessed from three temporal levels (prospectively, concurrently or retrospectively) (Anderson et al., 2021).

An intervention’s fidelity assessment is also important to consider (Carroll et al., 2007; Palsola et al., 2020). Fidelity is defined as the ongoing assessment, monitoring and enhancement of an intervention’s reliability and internal validity (Bellg et al., 2004; Borrelli et al., 2005). However, according to reviews, most fidelity assessments have focused on provider delivery, that is, the extent to which the intervention was delivered as intended but neglect the importance of assessing provider receipt (Rixon et al., 2016; Walton et al., 2017). Only a small proportion of studies report on healthcare professionals’ fidelity of receipt (Rixon et al., 2016), thus highlighting a gap in the implementation science literature. Therefore, studies not accounting for fidelity of receipt ultimately neglect how healthcare professionals (providers) understand and comprehend the intervention being delivered (Gould et al., 2014). Factors influencing behaviour change are often complex and poorly understood (Michie & Johnston, 2012). Due to the dynamic nature of precision medicine, the potential is there for different groups of healthcare professionals to misunderstand and have difficulties with the actual intervention due to an evidence base that is constantly evolving and the ever-increasing demands placed on staff to expand their knowledge of genomic medicine (McGill et al., 2020; Willemsen et al., 2019). Models and frameworks of fidelity agree that fidelity is relevant at the provider level (Bellg et al., 2004). The American Behaviour Change Consortium’s (BCC) framework of fidelity supports the need to investigate fidelity of receipt, defined as healthcare professionals’ understanding and comprehension of the intervention (Gould et al., 2014).

PRISM is a novel early-phase clinical trial involving transdisciplinary groups across historically siloed sectors, coming together in new configurations, which are expected to adopt a new behaviour within what could be seen to be an already adequately functioning system (Rushforth & Greenhalgh, 2020). Implementation strategies to help integrate multidisciplinary teams and implement change in practice exist within the precision medicine literature (Rankin et al., 2018; Rolfo et al., 2018). Examples include multidisciplinary meeting arrangements, which act as a tool to enhance behavioural support, and ultimately improve clinical decisions in multidisciplinary precision medicine teams (Rankin et al., 2018; Rolfo et al., 2018) (See Supplementary material, implementation strategy). Likewise, the Consolidated Framework for Implementation Research (CFIR) can be used to identify contextual factors that influence the implementation process of this new practice, such as those described as inner (inner networks) and outer (external/outer networks) contextual determinants (Damschroder et al., 2009). For example, McDonald (2013) conceptualised grouping contextual domains of CFIR together, a practice subsequently followed by others (Holen-Rabbersvik et al., 2020; Wiig et al., 2019). By using this approach, the practice of grouping the contextual domains (inner and outer system levels) allows for greater flexibility in line with the new practice demands of the transdisciplinary groups interacting in this precision medicine study context (See Supplementary material, implementation strategy). Although this framework has proved to help identify contextual factors in implementation success (Damschroder et al., 2009), it does not consider the different subtypes of healthcare professional roles in making implementation happen (McGuier et al., 2021; Wandersman et al., 2012). Therefore, the dynamic forces of context assessed by CFIR need more concerted study in terms of professional roles and its influence on implementation outcomes (Damschroder et al., 2022). As implementation science matures, implementation science frameworks must develop too (Damschroder et al., 2022). The CFIR framework can therefore be augmented with demographic contextual aspects, such as the healthcare professional’s role (i.e. case-based demographics of professional role), based on guidance from the conceptual Interactive Systems Framework for Dissemination and Implementation (ISF) (Wandersman et al., 2012). This combination recognises that different professional roles face different challenges with implementation when they try to work with other healthcare professionals (e.g. bioinformatician and clinician) (Wandersman et al., 2012). This advances ideas in implementation science as this insight can enable us to understand acceptability and fidelity more fully through different contextual layers (the importance of professional role and the inner and outer contextual layers) (McGuier et al., 2021). It could also help in understanding contextual fit and navigating and working within a precision medicine research program to help inform PRISM intervention optimisation.

The idea of rapid qualitative research approaches (Vindrola-Padros, 2021) has not been taken up in qualitative research studies using implementation science frameworks applied to precision medicine models of care (Best et al., 2021, 2022; Levy et al., 2019; McGill et al., 2020; Zebrowski et al., 2019). For example, these studies applied static interviews and lengthy transcription and analysis procedures to make sense of data (Best et al., 2021, 2022; McGill et al., 2020; Zebrowski et al., 2019). The online proforma (Smith et al., 2021), called the Rapid Health Implementation Proforma (RHIP is described in detail later, see Method section), is a candidate for rapidly resolving timeliness issues that influence the utility of research and evaluation findings within healthcare (Nunns, 2009). Responding to the call for rapid qualitative data collection (Smith, 2022; Vindrola-Padros, 2021; Vindrola-Padros et al., 2020), the online proforma takes the form of an open-ended, text-box survey that can inform decision-making around intervention optimisation (Smith et al., 2021). Rapid data collection tools that use qualitative approaches (e.g. online proformas) are well suited to understanding acceptability and fidelity of receipt and the complexity of individually tailored, complex, multidimensional interventions. There is a gap in the literature regarding research formally applying implementation outcomes (e.g. acceptability and fidelity) in their reporting within genomics-related research (Morrow et al., 2021). A recent systematic review on genetic referral practices revealed that few studies exist within the literature that consider implementation outcomes such as acceptability of interventions (Morrow et al., 2021) and others have called for more guidance over methodological specificity to monitor fidelity to enhance the reliability and validity of complex interventions (Ginsburg et al., 2021). Given that there is a lack of guidance on methodological specificity for monitoring and evaluating implementation outcomes within genomics-related literature, our qualitative study attempted to bridge this methodological gap and aimed to evaluate intervention acceptability (affective and cognitive responses to the intervention), fidelity of receipt (understanding and comprehension of the intervention) and contextual determinants of PRISM (understanding responses over contextual fit) (Figure 1).

Figure 1.

Visually representing our study based on Curran's (2020) simplified implementation science tool.

Method

The method outlines in detail the co-design process consisting of a qualitative evaluation method developed to help combine implementation outcomes of acceptability, fidelity and context.

Study Design

This was a qualitative study design combining information about individuals who self-classified into different professional groups (see Table 1), with information qualitatively mapped against group views and experiences of acceptability, fidelity and contextual intervention factors assessed using Framework Analysis (Ritchie & Spencer, 1994). Data were collected during the PRISM early-phase clinical trial (pre-results). Ethical approval was granted by the Hunter New England Research Ethics Committee, New South Wales, Australia, approval number: 2019/ETH12025.

Table 1.

Self-classified Demographic Sample Characteristics (N = 24).

Characteristic	N	(%)
Professional role self-classification
Group one: Other clinical staff	4	(16.6)
Group two: Research scientist/team member	13	(54.2)
Group three: Bioinformatician	3	(12.5)
Group four: Clinician	4	(16.6)
Age (years)
20–29	5	(20.8)
30–39	11	(45.8)
40–49	5	(20.8)
50–59	1	(4.2)
60–69	2	(8.3)
Work
Hospital	4	(16.7)
Research institution	13	(54.2)
Both (hospital and research institution)	7	(29.2)
Years in profession
<1	1	(4.2)
1–5	5	(20.8)
6–10	9	(37.5)
11–15	2	(8.3)
16–20	2	(8.3)
20–25	2	(8.3)
25–30	1	(4.2)
30+	2	(8.3)

Sample and Recruitment

For the early-phase PRISM clinical trial, the inclusion criteria for the sample were staff aged over 18 to be drawn from central program members and national coordinating teams in Sydney. Patient cases were brought to the consideration of transdisciplinary healthcare professionals through weekly curation meetings (CMs) and biweekly national Multidisciplinary Tumour Board (MTB) meetings (see Supplementary material, implementation strategy). Purposive sampling was used to recruit staff members working on PRISM who attended CMs and MTBs. Alongside the program leader, a lead ZERO clinician (the lead program clinician leads a team of clinical and research staff on ZERO dedicated to improving clinical outcomes) from the central site and one program manager are investigators on the implementation science study facilitating access to staff working on the PRISM trial. Staff members were invited to complete the RHIP via an online proforma survey. Staff members were sampled broadly to include trial managers, oncologists, surgeons, consultants and researchers and to encourage rich, nuanced perceptions of program activities. Participants included different clinical and non-clinical program members self-classified into homogeneous groups (Table 1).

Rapid Health Implementation Online Proforma

RHIP was managed and distributed via Qualtrics (a university-supported platform hosted by the lead university). Informed consent was obtained at the start of the online proforma. Those who consented to participate were directed to the open-ended, text-box survey (RHIP). No incentives were offered for taking part. RHIP was live between January and April 2020. Participants were sent one reminder email through Qualtrics 2 weeks after initial contact.

RHIP is a rapid qualitative data collection tool comprising 16 questions (Bierbaum et al., 2020; Rapport, Francis-Auton, et al., 2020; Smith et al., 2021). RHIPs are designed to encourage expansive answers from proforma participants (Table 2). RHIP differs from structured interviews or closed question surveys (Rapport, Smith, et al., 2020) which are highly formalised and often rigid. Unfortunately, health research frequently addresses questions of limited relevance to clinicians leading to research ‘waste’ (Ioannidis, 2016). To avoid this, our study describes in detail the research co-design process. Question formulation was a co-designed experience involving implementation scientists and key ZERO stakeholders to ensure shared understandings of implementation progress. Our co-design mandate (Riggs et al., 2017, 2021) therefore focussed on developing RHIP and the co-construction of readily measurable qualitative indicators using the lived experience and tacit knowledge of key stakeholders who support the child (and family) along the precision treatment pathway of care. By adopting this approach, we ensured that the voices of key stakeholders were incorporated into RHIP, with stakeholders acting as co-researchers in the co-design process. Our co-design process (Riggs et al., 2017, 2021) involved equal collaboration between implementation scientists and key stakeholders (Vindrola-Padros, 2021). In addition, it is generally believed in co-design that minimal instructions provide participants more agency and can produce richer outputs in group work (Milasan, 2020; Tran Smith et al., 2015). Therefore, an in-person team meeting activity was arranged with the key stakeholders within the given mandate that involved full-day discussions and brainstorming around RHIP to elicit and incorporate stakeholders’ tacit and practice-based knowledge of the PRISM trial. The role of the field researcher (JS) in this process became that of a facilitator than sole authority. The outcome of the co-design process was the development of RHIP to help illuminate readily measurable qualitative indicators and incorporation of the research team’s desire to include context-specific information. The RHIP open-ended proforma questions are shown in Table 2.

Table 2.

RHIP Open-Ended, Text-Box Survey Comprising 16 Questions.

RHIPs open-ended question format
1. What is your level of confidence in understanding and applying molecular genetics?
2. (For clinical staff only) If a recommendation was made in the clinical research report issued, how confident are you in applying this result in a clinical setting to a patient?
3. What suggestions do you have to accelerate the translation of precision medicine research into clinical care and decrease the time between precision medicine guided patient management recommendation and treatment delivery?
4. How do you see precision medicine approaches and the molecular profiling information that it generates integrating into paediatric cancer care in the next 5 years?
5. Thinking more broadly than ZERO/PRISM trial, what are the current research and clinical limitations/barriers to accelerating precision medicine translation?
6. (For clinical staff only) What are the main barriers to drug access that you or your colleagues have experienced?
7. Have you received any specific training in molecular oncology?
8. What tools would help you to implement change and integrate precision medicine molecular testing into routine patient care?
9. What is your level of understanding of the information provided in the clinical research report issued to treating clinicians?
10. What would help to increase your understanding and confidence, or level of comfort with current precision medicine recommendations?
11. In your opinion, what are the greatest facilitators/enablers faced in applying new knowledge gained from the ZERO childhood cancer program to implement precision medicine driven clinical management of children with cancer in the future, nationally?
12. In your opinion, what are the greatest facilitators/enablers faced in applying new knowledge gained from the ZERO childhood cancer program to implement precision medicine driven clinical management of children with cancer in the future, locally (your site)?
13. Following on from ZERO/PRISM clinical trial, what are the barriers faced in integrating precision medicine into the healthcare system for children in the future?
14. In your opinion, what are the greatest barriers faced in applying new knowledge gained from the ZERO childhood cancer program to implement precision medicine driven clinical management of children with cancer in the future? Nationally
15. In your opinion, what are the greatest barriers faced in applying new knowledge gained from the ZERO childhood cancer program to implement precision medicine driven clinical management of children with cancer in the future? Locally (your site)
16. What aspects of the current precision model should become standard of care?

Analysis

Data were analysed using Framework Analysis (Ritchie & Spencer, 1994) and managed using NVivo 12 Plus (QSR, 2019). Framework Analysis is a systematic approach to structuring and summarising data and explores similarities and differences between group views and experiences (Ritchie & Spencer, 1994; Smith, Rapport, et al., 2020). An experienced male qualitative researcher (JS) led the analysis. Table 3 outlines the application of Framework Analysis according to Ritchie and Spencer’s (1994) five highly connected but distinct stages of thematic interpretation. The demographic variables provided descriptive features for RHIP groups (see professional role self-classification in Table 1). Groups consisted of individuals who were seen to hold commonalities, in this case, identified through their professional role affiliation. Final themes and codes were reviewed by JS, FR and JB.

Table 3.

Set of Stages and Processes Involved.

Analytical process	Description
1. Familiarisation	Immersion in RHIP open-ended responses, reading and re-reading the data to absorption. Import data into NVivo data management tool
2. Identifying a thematic framework	Thematic framework identification with data deductively examined and referenced. The identified frameworks of acceptability (Sekhon et al., 2017), fidelity of receipt (Bellg et al., 2004; Borrelli et al., 2005) and CFIR (Damschroder et al., 2009) deductively guided the thematic framework. This included the development of a codebook based on the three a priori themes (fidelity, acceptability and context) (Bellg et al., 2004; Borrelli et al., 2005; Damschroder et al., 2009; Rixon et al., 2016; Sekhon et al., 2017), which was refined during thematic indexing
3. Indexing	Iterative comparative process, selecting and coding text by applying the framework, and developing the deductive themes and categories. The analysis was supported by researchers from the wider study team (F.R. and J.B.), who encouraged discussion around decision-making as the framework took shape (Smith, Rapport, et al., 2020). Adaptions were made through this process (for example, the context component ‘learning climate’ (Damschroder et al., 2009) became ‘rapid health learning system’, and as coding continued, components took on greater relevance concerning the intervention)
4. Charting	Charting data for summary and arrangement into a matrix in NVivo
5. Mapping and interpretation	Interpretation of all materials, using the matrix linking case-based and theme-based analysis through triangulation to develop explanations

Results

Twenty-four of the 68 RHIPs distributed were returned (35% response rate). Demographic items taken forward for further qualitative examination concentrated on perceptions of group identity and role characteristics (see Table 1, professional role self-classification). Most of the participants (54.2%) identified as ‘research scientist/team member’ (n = 13), 16.6% as ‘clinician’ (n = 4), 16.6% as ‘other clinical staff’ (n = 4) and 12.5% as ‘bioinformatician’ (n = 3). Most were aged 30–39 years (n = 11, 45.8%), worked in a research institution (n = 13, 54.2%) and had between 6 to 10 years’ experience in the clinical setting (n = 9, 37.5%). The demographic characteristics of the sample are presented in Table 1. Themes and their related sub-themes are outlined (Table 4).

Table 4.

Themes.

Theme	Definition	Identified qualitative framework
Theme 1: Acceptability
Sub-themes
Self-efficacy	Provider’s self-efficacy is known as a participant’s confidence in participating in the intervention	Sekhon et al. (2017)
Affective attitude	Refers to how a provider feels about an intervention	Sekhon et al. (2017)
Perceived effectiveness	Provider’s perceived effectiveness covers the extent to which an intervention is perceived likely to achieve its purpose	Sekhon et al. (2017)
Theme 2: Fidelity of receipt
Sub-themes
Knowledge	Provider’s knowledge of the intervention and ability to use the skills or recommendations learned in the intervention	Bellg et al. (2004), Gould et al. (2014), Rixon et al. (2016)
Comprehension	Provider’s comprehension of the intervention delivered	Bellg et al. (2004), Gould et al. (2014), Rixon et al. (2016)
Theme 3: Contextual fit
Sub-themes
Rapid health learning system	Availability of sudden learning opportunities	Damschroder et al. (2009)
Cosmopolitanism	The degree to which an organisation is networked with other external organisations	Damschroder et al. (2009)
Inner networks	The nature and quality of webs of social networks and the nature and quality of formal and informal communications within an organisation	Damschroder et al. (2009)
External policy & incentives	External policy, regulations, mandates, recommendations or guidelines	Damschroder et al. (2009)

Theme 1: Acceptability

Self-Efficacy

Participants reported various confidence levels in applying PRISM and the subsequent recommendations derived from intervention reporting.

Educational background and training were believed to impact confidence. The bioinformatician group had many years’ experience, accreditation and niche skills in genomics in children and adults with rare cancers. As expected, this group was highly qualified and confident in applying the intervention in terms of molecular genetic knowledge:

“[I] developed a lot of the genomic analysis methods used to diagnose patients with a rare disease.” (Bioinformatician).

Similarly, the clinician group reported high confidence in the interpretation of results and the ability to gain further confidence through training in the application of molecular genetics and ongoing trial development by being part of PRISM:

“I also continue to develop it [referring to their confidence], notable through my involvement in the ZCC [ZERO] Program.” (Clinician).

This group expanded further on what influenced their confidence. According to the clinician group, confidence was linked to the strength of clinical evidence:

“When clinical evidence has been shown in the same tumour type or when a clinical trial is available for the patient in Australia, I am very confident in applying it in clinics. However, for weaker evidence, I will be more careful.”

Confidence in the recommendations and reports issued were discussed regarding clinical recommendations issued to treating clinicians. Reports were favoured with limitations noted:

“I like the reports, but sometimes hard to read, as it is dense.” (Clinician).

Some of the research scientist/team member group reported a mix of confidence levels attributed to education and training. The following highlights the contrast of confidence in applying molecular genetics and/or their pioneering molecular biology/genetics work linked to their background:

“I am very confident that I am well versed in understanding and applying molecular genetics.” (Research scientist/team member).

“As I have not received any specific molecular training in my position (only what I learned at university), I’m not confident in understanding nor applying molecular genetics … I understand simple aspects.” (Research scientist/team member).

One member highlighted that their confidence was linked to the PRISM weekly interactions with the curation team and the interaction advanced on their education and background:

“I have a degree in Molecular biology and genetics and am exposed to molecular genetic concepts through my interactions with the PRISM curation team.” (Research scientist/team member).

The other clinical staff presented different clinical or molecular knowledge and experience levels than the other three groups.

“I am very confident in the application of molecular genetics.” (Other clinical staff).

Affective Attitude

Insights towards the intervention accelerating the translation of precision medicine research into clinical care were discussed. Respondents were optimistic about the program and how it could be used as a vehicle to accelerate the translation of the intervention into clinical care.

Similarly, the research scientist/team member group described cross-collaboration as leading to a robust evidence base and ‘a full suite of clinical trials bolted on [to the trial/intervention]’. The bioinformatician group added that it would encourage a centralised national infrastructure network to support a high volume of samples and, according to clinicians, more trained staff.

Access to novel medications was seen as a significant barrier for several reasons, and it was reported that there was a need to improve access summed up concisely by the other clinical staff:

“Clinical teams are relentless in pursuing access to novel drugs ... there is no pharmacist on the core team … ZERO asks approval of drugs that in some instances have no paediatric safety and efficacy data, no dosing data or paediatric specific formulation. This is a leap of faith even for the bravest of clinical staff sitting on drug approval committees.” (Other clinical staff).

Perceived Effectiveness

All group members supported the intervention and believed it would achieve its purpose of becoming standard care. Consequently, it was said that:

“Paediatric precision medicine is likely to become the standard of care everywhere.” (Other clinical staff).

The research scientist/team member and bioinformatician groups suggested that precision medicine should become standard care. The clinician group indicated that genetic testing is used for some cancers already. Still, precision medicine is not broadly used, and genome-wide approaches have not yet been applied. Clinicians also indicated that precision medicine should be applied to all children, not just high-risk groups.

“molecular profiling was the standard of care for some diseases [already being delivered], such as acute leukaemia or neuroblastoma, but not as a genome-wide approach.” (Clinician).

Current research and clinical limitations were raised as barriers and focused on the lack of workforce in the face of rising patient numbers to build ambitious trials:

“workforce limitations … establishment of training programs to grow a paediatric genetic counselling workforce to cope with the increased influx of referrals.” (Other clinical staff).

Similarly, a lack of targeted drug agents was perceived as a result of many different factors or the interaction between these factors limiting the perceived effectiveness of precision medicine for the research scientist/team member:

“lack of appropriate targeted agents for a given target driver either through no drug existing, or a lack of paediatric safety data for a single drug or combination.”

For the clinician group, effectiveness was seen in having meaningful clinical results and for pharmaceutical companies to include paediatric components in clinical trials. Yet, the effectiveness was limited by certain barriers to do with international collaboration, differences between countries (United States and Europe) and poor infrastructure:

“It takes international collaboration … There are barriers and differences between countries (the USA and Europe) that can build more ambitious trials because of their numbers. And, from there, the infrastructure is too weak.”

Perceived effectiveness was also discussed in the broadest of terms. Participants were concerned with what the evidence base generated from the precision medicine trial would look like post-intervention suggesting that the effectiveness of precision medicine in the future is contingent on the cooperation of local organisational structures and policies which were outside the control of the trial. Therefore, drug medication access and consequent delays indicate a need for greater collaboration between local hospitals:

“Access to drugs outside of the clinical trial is a very big challenge, if compassionate access is not possible, the cost is not always covered by the hospital. Moreover, there are different approaches between hospitals.” (Clinician).

“There are delays in treatment due to hospital executive approval processes. This is outside the control of precision medicine trials, however, as the approval process follows an evidence-based medicine approach, it is difficult to fit the recommendations of precision medicine within this structure.” (Other clinical staff).

Theme 2: Fidelity of Receipt

Knowledge

Opinions over specific training in molecular oncology were similar across the groups, where participants defined their knowledge through the lens of their accredited training and resultant expertise (such as being board certified or trained in oncology at BSc, MSc or PhD level). Some of the clinician group focused on ‘curation workshops’ to further enhance their knowledge of the intervention. Understanding available precision medicine tools can improve the implementation of change and the integration of precision medicine molecular testing into routine patient care. Yet, identified tools that fulfilled this purpose differed between groups based on their different professional needs and requirements for their particular professional role:

“A drug access database framework.” (Other clinical staff).

“I believe we have all the tools, just need to make better use of the tools.” (Research scientist/team member).

“Better computational infrastructure and a skilled workforce.” (Bioinformatician).

“Training [needed] for clinicians to feel more confident using the findings.” (Clinician).

Comprehension

For most, the biweekly MTB meetings enhanced their understanding. Comprehending technical content delivered at MTBs and reports issued and the level of detail that went into finished reports were discussed. MTBs were sometimes perceived as very technical. In contrast, reports that supplement the recommendation within the MTB were viewed as concise overviews of meeting discussions and helped aid comprehension of what the specific recommendations for precision medicine included:

“I am not clinically trained. However, I attend most MTBs and feel like I am learning the basics. The MTBs can be very technical (for obvious reasons), however, the finished reports that are circulated are clear and concise as to what the PRISM recommendation is for that particular patient.” (Other clinical staff).

Nevertheless, clinical descriptions were often viewed as: ‘hard to follow’ even though reports were said to make ‘a lot of sense’ (Bioinformatician).

Comprehension was perceived to be high amongst the clinician group, especially in terms of their involvement:

“I am part of the PRISM curation team (CM) and present some of the clinical cases. Therefore, I know firsthand the content of these presentations [in the MTB] and have worked on them.” (Clinician).

One Clinician nominated a specific test within the trial (“Somatic sequencing”) as an area in which they lacked understanding at MTB and in reports issued and therefore required more support.

The meeting arrangements (CMs and MTBs) were directly addressed to enhance comprehension. They were highlighted by members of the other clinical staff group, who requested that particular workshops be made available for greater knowledge acquisition:

“[the need for workshops such as] in variant curation pipeline and technologies; to determine germline variant pathogenicity.”

Overall, it was commonly reported that one’s level of understanding grew considerably due to attendance at CMs and MTBs.

Theme 3: Contextual Fit

Rapid Health Learning System

National post-intervention enablers to integrating the intervention were reported. They represented the interplay between the intervention/trial and the contextual fit to which it would be applied at a national level. Generating evidence-based knowledge and practice was seen as an essential component in supporting future national initiatives:

“Building a comprehensive database about the mutations in children i.e., what are new mutations emerging.” (Other clinical staff).

MTBs were considered as central to a rapid health learning system and essential to collaborative engagement:

“MTB meetings are great and essential as it increases everyone's knowledge and opens up a forum for discussion and to hear about potential previous cases … helps keep the clinicians engaged who are off-site … Program managers/coordinators visit the national sites to keep them updated on processes and progress to keep their engagement.” (Research scientist/team member).

Delivering analysis for the recommendation and reports in a rapid way was beneficial to securing high levels of trust between different groups:

“The trust that the referring clinicians now have in the process and its recommendations has enabled rapid uptake nationally. Achieving seven-week turn-around time for analysis, MTB and report has led to a high level of trust.” (Bioinformatician).

In addition, the centralised research and implementation strategy was predicted to be a central aspect of the trial that would enable national success:

“Centralised research strategy and implementation strategy [as enablers of integrating the intervention].” (Clinician).

Cosmopolitanism (External Network/System)

Future forecasted national external system barriers noted by other clinical staff included the application, by junior doctors, of new precision medicine knowledge, with this sector of the workforce seen as needing to be trained in molecular oncology – presenting a national system barrier in the future. Further barriers related to both logistics and clinician engagement:

“Logistics, specifically regarding sample sending and reception at different hospitals/institutes/healthcare centres. Ensuring clinicians are engaged on a national scale.” (Research scientist/team member).

The bioinformatician group presented mixed feelings about cosmopolitanism and post-intervention barriers. They indicated that even with the best of efforts, drug medication access would remain a problem:

“Building a national program with true collaboration, will encourage sample and sharing in, and data sharing back out to partners, but still drug access will remain a challenge.” (Bioinformatician).

Inner Networks and Communication

Local post-intervention enablers to allow sustainability were forecast. The culture of having a motivated workforce and the dissemination of information centrally, channelled through the MTB was an enabler for future local implementation:

“PRISM is well known as it is coordinated out of [site named]. There is already great buy-in from clinical staff outside of the PRISM team (e.g., surgeons, pathologists). The study molecular oncologist is based at [site named] and can offer great support in terms of consenting/interpreting PRISM results etc.” (Other clinical staff).

Cross-collaboration and co-ordination, shown through a close relationship between researchers and clinicians (research scientist/team member), were described as a vehicle to facilitate the intervention:

“The easy access and close relationship between the researchers and the clinicians. The constant interaction and feeling of being a team really help break down barriers and understand the process from both sides which helps ensure the success of the program.” (Research scientist/team member).

Bioinformaticians stated that one of the enablers likely to benefit from applying the intervention locally in the future was:

“The trust that referring clinicians have in the process and its recommendations [referring to PRISM] has enabled rapid uptake of the intervention.” (Bioinformatician).

The clinician group stated that collaboration and the training of existing staff were necessary for future success locally:

“Collaboration and more staff and training of existing staff.” (Clinician).

External Policy and Incentives

Local post-intervention barriers to applying new knowledge gained from the intervention were forecast across groups. Applying new knowledge gained from the intervention was seen as something that may run into trouble when considering barriers related to drug medication access (clinician and other clinical staff), paediatric dose availability (Research scientists/team member) and not many actionable targets (bioinformatician):

“Drug access and drug committee restrictions and fear of the unknown by frontline clinical staff.” (Other clinical staff).

Discussion

This study used a rapid online proforma, RHIP, to evaluate acceptability, fidelity and contextual determinants of a clinical trial, PRISM. RHIP included a range of healthcare groups to understand how different groups coming together in new configurations think and feel whilst delivering the PRISM intervention. Acceptability and fidelity lost some important impact and meaning if contextual fit was not implicit in data understanding. As a result, contextual fit significantly impacts overall thematic meaning.

Evidence from the contextual fit theme highlights that it is highly integrative in and of itself, cross-cutting many of the other sub-thematic clusters. Thus, contextual fit pervades much of the data and attests to its meaning. For example, it highlighted group perceptions of integrating precision medicine, both locally and nationally, primarily focussing on precision medicine as a potential for a new standard of care and a possible pre-requisite to overcoming drug medication access and drug committee restrictions. Furthermore, the professional groups illuminate a need to include contextual fit in intervention dissemination and delivery, recognising that a nuanced understanding, according to group opinion of context, should be at the forefront of the implementation of PRISM. Context in our study relates to the inner (rapid health learning system, inner networks and communication), outer (cosmopolitanism, external policies and incentives) and demographic group contexts. Within this trial, identifying the group context and the inner and outer contextual determinants could help make implementation more effective. PRISM now has the opportunity to address finely-honed understandings of acceptability and fidelity gaps. This enables the identification of different contextual barriers that manifest for specific groups and considerations of different and shared views on contextual barriers that may limit intervention acceptability and fidelity in the future. Furthermore, by addressing the contextual fit of the intervention in PRISM, according to, for example, the clinician or bioinformatician group, we may be able to begin to leverage facilitators and mitigate against barriers that can enable groups to carry out their roles more effectively.

Participants reported self-efficacy, perceived effectiveness and affective attitude towards the intervention (acceptability). Thus, confirming the theoretical definitions of acceptability (Palsola et al., 2020; Sekhon et al., 2017). Quality was indicated by fidelity of receipt relating to knowledge (specific training in molecular oncology and understanding tools to enhance molecular testing) and comprehension (strategies to improve comprehension and understanding of current precision medicine recommendations) (Bellg et al., 2004; Gould et al., 2014; Rixon et al., 2016). Group views also differed across themes, adding nuance to our understanding of thematic variation. This also bodes well for intervention optimisation and tailored implementation strategies offering opportunities to overcome acceptability and fidelity gaps. This is in line with recent challenges to the traditional convention of assuming a ‘one-size-fits-all’ implementation strategy approach (Chambers, 2020). Our unique approach supports the development of built-in feedback loops (Braithwaite et al., 2014) for intervention optimisation and, when applied to PRISM, a new way to monitor acceptability and fidelity of receipt and context over time (pre-intervention, during-intervention and post-intervention) (Vindrola-Padros, 2021). This study highlights that qualitative research can be built into the intervention design at an early stage of intervention development (Bradley et al., 1999). Qualitative methods can contribute to both ongoing intervention optimisation (Chambers et al., 2013; Levati et al., 2016; Palsola et al., 2020) and evaluation (Braithwaite et al., 2014) assessed in parallel to a clinical trial (Gould et al., 2014). Understanding healthcare provider acceptability and fidelity of receipt at this stage of development, evaluation and implementation highlights the possible ways of intervention improvement, helping to inform both the current strategies (MTBs and CMs) as well as helping to develop broader implementation strategies. In this study, participant views across different groups converged regarding the potential for the intervention to achieve its aims and fulfil its purpose (a novel drug target, early referral to explore cancer predisposition and a change in diagnosis). However, consensus across the groups showed that the intervention was more specifically geared to the bigger picture, the future purpose, rather than immediate change at its current stage.

Although perceived effectiveness was linked to the intervention in terms of its ability to generate evidence that could be used for standards of care in the future, strong concerns were raised, particularly by the other clinical staff group that limited perceived effectiveness due to drug medication access. These types of concerns and how they are linked to specific groups within a wider consortium have the potential to influence an intervention achieving its purpose. If this is not overcome, this can negatively impact intervention scale-up. Drug medication access is not unique to or within PRISM’s control. However, it is intertwined within the contextual fit and relates to wider external policy constraints that indicate broader contextual issues regarding drug committees and regulatory and legislative restrictions. This identifies some of the wider challenges and opportunities for scale-up in understanding contextual components that operate at multiple levels that could be used for sense-making to help create much-needed change. Such an approach has been used to support decision-making for the COVID-19 pandemic response and recovery measures (Sharma et al., 2021). Some clinicians and other clinical staff reported concerns over understanding information discussed at MTBs, such as somatic sequencing, which restricts fully understanding personalised medicine opportunities. This resonates with the extant literature, which indicates that some clinicians may not feel adequately skilled to interpret and communicate the results of advanced genomic sequencing to patients and their families (Arora et al., 2016; Burton et al., 2012).

Through our understanding of the theme of acceptability in terms of affective attitude, exploration of CMs indicated that attendance can help educate clinical groups about technologies around OMICs data and can increase self-efficacy through MTB meeting attendance, interaction with PRISM members and involvement with ZERO clinical and research program senior leaders and that this can boost confidence in the intervention. An opportunity arises to further encourage participation within these implementation strategies that create rapid health system learning contexts (Braithwaite, 2018). To more fully embrace a transdisciplinary approach to learning could include other disciplinary groups, such as pharmacists, at an earlier part of the decision-making process and as an integral part of the skill mix (Battaglia & Glasgow, 2018). This would not only develop competencies but also approaches to collaboration to positively impact outer system-level barriers (external policies and influences on the intervention, such as drug medication access and procurement) (Braithwaite, 2018; Smith et al., 2022; Smith, Rapport, et al., 2020). Although the concept of a rapid health learning environment is considered, in many respects, to be in its embryonic stage (Coiera, 2020), its potential for use within the PRISM trial as both a training and educational driver seems plausible.

The intervention was viewed most positively as the benchmark for accelerating the translation of precision medicine research into clinical care across the nation. The suggestion that if an intervention is perceived to be acceptable at source, it will be implemented with high levels of fidelity which attunes with this study’s findings. Acceptability, fidelity and contextual fit can contribute to implementation success. However, a precision medicine intervention’s acceptability, fidelity and contextual fit have rarely, if ever, been assessed simultaneously in a single study. Our novel idea of amalgamating implementation outcomes of acceptability (Sekhon et al., 2017) and fidelity (Bellg et al., 2004; Borrelli et al., 2005; Rixon et al., 2016; Sekhon et al., 2017) alongside context (Damschroder et al., 2009) is a new way of qualitatively assessing implementation outcomes and context (Proctor et al., 2011). RHIP should now be added to the repertoire of readily measurable, qualitative indicators helping to mature implementation science frameworks (Damschroder et al., 2022; Nevedal et al., 2021) and enhance our ability to rapidly understand both implementation and trial outcomes (Anderson et al., 2021; Chambers et al., 2013; Curran, 2020; Palsola et al., 2020; Price et al., 2019; Proctor et al., 2011; Smith, Rapport, et al., 2020).

Study strengths and limitations: This study has assisted in understanding acceptability and fidelity in relation to contextual fit by using RHIP, which can contribute to intervention optimisation. This has multiple possibilities for developing wider qualitative work for acceptability and fidelity within implementation science (Morrow et al., 2021) and assessing methodological strategies to enhance the reliability and validity of complex interventions (Ginsburg et al., 2021). Full RHIP collaboration regarding the co-design of the open-ended text-box survey with stakeholder involvement proved to be a key strength of this study’s design. Having more participants for the qualitative study would help produce larger numbers of participants per group and would have been advantageous for comparing groups. Nevertheless, the sample was sufficient for proforma work, as we made appropriate use of a purposeful sampling framework to ensure multidisciplinary healthcare professional involvement, recognising that real-world data is dependent on participants’ willingness to provide information during a busy working day. Moreover, competing studies delivered at the same time as our study could have negatively influenced our sample numbers. This may also help to explain the lower rates of participation by clinicians who were involved in other research projects outside of our study, and in addition to this, also managing heavy patient loads. While a 35% RHIP response rate may appear low, it is widely accepted that published online survey response rates with healthcare professionals can often be lower than 30% (Bonevski et al., 2011; Cunningham et al., 2015). Nonetheless, future research using online proforma work that is not impacted by COVID-19 (as our study was) would no doubt further help to increase the level of engagement from participants.

Conclusion

Randomised controlled trials typically fail to address how and why an intervention works (Greenhalgh & Papoutsi, 2018). It is useful to provide in-depth insights into implementation outcomes and contextual factors potentially impacting implementation success. Our study contributes to ZERO’s early-phase clinical trial, PRISM, by addressing how and why those who participated and/or contributed felt PRISM would be effective. This was understood in relation to the qualitative understanding of groups’ cognitive-affective needs and their intervention expectations. Our study advances knowledge of different levels of healthcare professional groups’ involvement in an intervention. This can now be explored through wider contextual enablers and barriers to group- and system-level functioning. While implementation outcomes are distinct from trial outcomes (Curran, 2020; Proctor et al., 2011), it is important to recognise that an intervention cannot be effective if it is not implemented well (Bellg et al., 2004; Borrelli et al., 2005; Rixon et al., 2016). Nor will it be effective if it is not accepted by those involved. Therefore, fidelity of receipt, acceptability and contextual factors need careful monitoring (Rixon et al., 2016; Sekhon et al., 2017) and ongoing intervention optimisation (Chambers et al., 2013). Identifying critical implementation factors, viewed from the perspective of different groups, can help overcome system gaps and enhance intervention success and the valid interpretation of trial outcomes.

Supplemental Material

Supplemental Material - The Voices of Stakeholders Involved in Precision Medicine: The Co-Design and Evaluation of Qualitative Indicators of Intervention Acceptability, Fidelity, and Context in PRecISion Medicine for Children With Cancer in Australia

Supplemental Material for The Voices of Stakeholders Involved in Precision Medicine: The Co-Design and Evaluation of Qualitative Indicators of Intervention Acceptability, Fidelity and Context in PRecISion Medicine for Children With Cancer in Australia by James Smith, Jeffrey Braithwaite, Tracey A. O’Brien, Stephanie Smith, Vanessa J. Tyrrell, Emily V. A. Mould, Janet C. Long and Frances Rapport in Qualitative Health Research.

Footnotes

Authors’ Contributions

J.S. is the guarantor for this study. J.S. and S.S. conceived and designed the study, which was conducted by J.S. with assistance from S.S., F.R., J.B., T.O’B., V.T., E.M., and J.L. who helped interpret the results. J.S. drafted the manuscript with editing advice and support from J.B. and F.R. All authors edited the manuscript for important intellectual content and approved the final manuscript.

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: All authors have completed the ICMJE uniform disclosure form at

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by Luminesce Alliance – Innovation for Children’s Health for its contribution and support. Luminesce Alliance – Innovation for Children’s Health is a not for profit cooperative joint venture between the Sydney Children’s Hospitals Network, the Children’s Medical Research Institute and the Children’s Cancer Institute. It has been established with the support of the NSW Government to coordinate and integrate paediatric research. Luminesce Alliance is also affiliated with the University of Sydney and the University of New South Wales Sydney. JB reports grants from NHMRC; APP1176620, APP1135048, APP9100002.

Research Ethics and Patient Consent

Ethical approval and site governance was granted by the Hunter New England Research Ethics Committee, NSW, Australia. Approval number: 2019/ETH12025. Data were collected during the PRISM early-phase clinical trial (pre-results).

Guarantor

J.S.

Data Availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

ORCID iDs

James Smith

Jeffrey Braithwaite

Frances Rapport

Supplemental Material

Supplemental material for this article is available online.

References

Anderson

N. C.

Kesten

J. M.

Ayres

Hickman

Amlôt

Michie

Lorencatto

(2021). Acceptability of, and barriers and facilitators to, a pilot physical health service for people who inject drugs: A qualitative study with service users and providers. International Journal of Drug Policy, 99(Suppl_5), 103437. https://doi.org/10.1016/j.drugpo.2021.103437

Arora

N. S.

Davis

J. K.

Kirby

McGuire

A. L.

Green

R. C.

Blumenthal-Barby

J. S.

Ubel

P. A.

MedSeq

(2016). Communication challenges for nongeneticist physicians relaying clinical genomic results. Personalized Medicine, 14(5), 423–431. https://doi.org/10.2217/pme-2017-0008

Battaglia

Glasgow

R. E.

(2018). Pragmatic dissemination and implementation research models, methods and measures and their relevance for nursing research. Nursing Outlook, 66(5), 430–445. http://simsrad.net.ocs.mq.edu.au/login?url=http://ovidsp.ovid.com/ovidweb.cgi?T=JS&CSC=Y&NEWS=N&PAGE=fulltext&D=prem&AN=30093135

Bellg

Borrelli

Resnick

Hecht

Minicucci

Ory

Ogedegbe

Orwig

Ernst

Czajkowski

(2004). Enhancing treatment fidelity in health behavior change studies: Best practices and recommendations from the NIH behavior change consortium. Health Psychology: Official Journal of the Division of Health Psychology, American Psychological Association, 23(5), 443–451. https://doi.org/10.1037/0278-6133.23.5.443

Best

Brown

Lunke

Patel

Pinner

Barnett

C. P.

Wilson

Sandaradura

S. A.

McClaren

Brett

G. R.

Braithwaite

Stark

(2021). Learning from scaling up ultra-rapid genomic testing for critically ill children to a national level. NPJ Genomic Medicine, 6(1), 5. https://doi.org/10.1038/s41525-020-00168-3

Best

Long

J. C.

Braithwaite

Taylor

(2022). Standardizing variation: Scaling up clinical genomics in Australia. Genetics in Medicine, S1098–S3600(22), 00019–00023. https://doi.org/10.1016/j.gim.2022.01.004

Bierbaum

McMahon

C. M.

Hughes

Boisvert

Lau

A. Y. S.

Braithwaite

Rapport

(2020). Barriers and facilitators to cochlear implant uptake in Australia and the United Kingdom. Ear and Hearing, 41(2), 374–385. https://doi.org/10.1097/AUD.0000000000000762

Bonevski

Magin

Horton

Foster

Girgis

(2011). Response rates in GP surveys: Trialling two recruitment strategies. Australian Family Physician, 40(6), 427–430.

Borrelli

Sepinwall

Ernst

Bellg

Czajkowski

Breger

Defrancesco

Levesque

Sharp

Ogedegbe

Resnick

Orwig

(2005). A new tool to assess treatment fidelity and evaluation of treatment fidelity across 10 years of health behavior research. Journal of Consulting and Clinical Psychology, 73(5), 852–860. https://doi.org/10.1037/0022-006X.73.5.852

10.

Bradley

Wiles

Kinmonth

A. L.

Mant

Gantley

(1999). Development and evaluation of complex interventions in health services research: Case study of the southampton heart integrated care project (SHIP). The SHIP collaborative group. BMJ (Clinical Research ed.), 318(7185), 711–715. https://doi.org/10.1136/bmj.318.7185.711

11.

Braithwaite

(2018). Changing how we think about healthcare improvement. Bmj: British Medical Journal, 361. https://doi.org/10.1136/bmj.k2014.

12.

Braithwaite

Marks

Taylor

(2014). Harnessing implementation science to improve care quality and patient safety: A systematic review of targeted literature. International Journal for Quality in Health Care, 26(3), 321–329. https://doi.org/10.1093/intqhc/mzu047

13.

Brown

(2005). Shifting tenses: Reconnecting regimes of truth and hope. Configurations, 13(3), 331–355. https://doi.org/10.1353/con.2007.0019

14.

Burton

Cole

Lucassen

A. M.

(2012). Genomic medicine: Challenges and opportunities for physicians. Clinical Medicine, 12(5), 416–419. https://doi.org/10.7861/clinmedicine.12-5-416

15.

Carroll

Patterson

Wood

Booth

Rick

Balain

(2007). A conceptual framework for implementation fidelity. Implementation Science, 2(1), 40. https://doi.org/10.1186/1748-5908-2-40

16.

Chambers

D. A.

(2020). Considering the intersection between implementation science and COVID-19. Implementation Research and Practice, 1, 0020764020925994. https://doi.org/10.1177/0020764020925994.

17.

Chambers

D. A.

Glasgow

R. E.

Stange

K. C.

(2013). The dynamic sustainability framework: Addressing the paradox of sustainment amid ongoing change. Implementation Science, 8(1), 117. https://doi.org/10.1186/1748-5908-8-117

18.

Coiera

(2020). The cognitive health system. The Lancet, 395(10222), 463–466. https://doi.org/10.1016/S0140-6736(19)32987-3

19.

Cunningham

C. T.

Quan

Hemmelgarn

Noseworthy

Beck

C. A.

Dixon

Samuel

Ghali

W. A.

Sykes

L. L.

Jetté

(2015). Exploring physician specialist response rates to web-based surveys. BMC Medical Research Methodology, 15(1), 32. https://doi.org/10.1186/s12874-015-0016-z

20.

Curran

G. M.

(2020). Implementation science made too simple: A teaching tool. Implementation Science Communications, 1(1), 27. https://doi.org/10.1186/s43058-020-00001-z

21.

Damschroder

L. J.

Aron

D. C.

Keith

R. E.

Kirsh

S. R.

Alexander

J. A.

Lowery

J. C.

(2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4(1), 50. https://doi.org/10.1186/1748-5908-4-50

22.

Damschroder

L. J.

Reardon

C. M.

Opra Widerquist

M. A.

Lowery

(2022). Conceptualizing outcomes for use with the consolidated framework for implementation research (CFIR): The CFIR outcomes addendum. Implementation Science, 17(1), 1–10. https://doi.org/10.1186/s13012-021-01181-5

23.

Evans

W. E.

Pui

C. H.

Yang

J. J.

(2020). The promise and the reality of genomics to guide precision medicine in pediatric oncology: The decade ahead. Clinical Pharmacology and Therapeutics, 107(1), 176–180. https://doi.org/10.1002/cpt.1660

24.

Ginsburg

G. S.

Phillips

K. A.

(2018). Precision medicine: From science to value. Health Affairs, 37(5), 694–701. https://doi.org/10.1377/hlthaff.2017.1624

25.

Ginsburg

L. R.

Hoben

Easterbrook

Anderson

R. A.

Estabrooks

C. A.

Norton

P. G.

(2021). Fidelity is not easy! Challenges and guidelines for assessing fidelity in complex interventions. [Trials Electronic Resource], 22(1), 372. https://doi.org/10.1186/s13063-021-05322-5

26.

Gould

N. J.

Lorencatto

Stanworth

S. J.

Michie

Prior

M. E.

Glidewell

Grimshaw

J. M.

Francis

J. J.

(2014). Application of theory to enhance audit and feedback interventions to increase the uptake of evidence-based transfusion practice: An intervention development protocol. Implement Sci, 9(1), 92. https://doi.org/10.1186/s13012-014-0092-1

27.

Greenhalgh

Papoutsi

(2018). Studying complexity in health services research: Desperately seeking an overdue paradigm shift. BMC Medicine, 16(1), 95. https://doi.org/10.1186/s12916-018-1089-4

28.

Holen-Rabbersvik

Ullebust

Ree

Schibevaag

Hurup-Thomsen

Strømme

Aase

Ellis

Wiig

(2020). How to deal with context? Evaluation of the SAFE-LEAD context tool for quality and safety in nursing home and homecare services. Tidsskrift for Omsorgsforskning, 6(2), 87–102. https://doi.org/10.18261/issn.2387-5984-2020-02-07

29.

Ioannidis

J. P.

(2016). Why most clinical research is not useful. Plos Medicine, 13(6), e1002049. https://doi.org/10.1371/journal.pmed.1002049

30.

Levati

Campbell

Frost

Dougall

Wells

Donaldson

Hagen

(2016). Optimisation of complex health interventions prior to a randomised controlled trial: A scoping review of strategies used. Pilot and Feasibility Studies, 2(1), 17. https://doi.org/10.1186/s40814-016-0058-y

31.

Levy

K. D.

Blake

Fletcher-Hoppe

Franciosi

Goto

Hicks

J. K.

Holmes

A. M.

Kanuri

S. H.

Madden

E. B.

Musty

M. D.

Orlando

Pratt

V. M.

Ramos

Ginsburg

G. S.

(2019). Opportunities to implement a sustainable genomic medicine program: Lessons learned from the IGNITE network. Genetics in Medicine, 21(3), 743–747. https://doi.org/10.1038/s41436-018-0080-y

32.

Manolio

T. A.

Chisholm

R. L.

Ozenberger

Roden

D. M.

Williams

M. S.

Wilson

Bick

Bottinger

E. P.

Brilliant

M. H.

Eng

Frazer

K. A.

Korf

Ledbetter

D. H.

Lupski

J. R.

Marsh

Mrazek

Murray

M. F.

O’Donnell

P. H.

Rader

D. J.

… Ginsburg

G. S.

(2013). Implementing genomic medicine in the clinic: The future is here. Genetics in Medicine, 15(4), 258–267. https://doi.org/10.1038/gim.2012.157

33.

Manolio

T. A.

Rowley

Williams

M. S.

Roden

Ginsburg

G. S.

Bult

Chisholm

R. L.

Deverka

P. A.

McLeod

H. L.

Mensah

G. A.

Relling

M. V.

Rodriguez

L. L.

Tamburro

Green

E. D.

(2019). Opportunities, resources, and techniques for implementing genomics in clinical care. The Lancet, 394(10197), 511–520. https://doi.org/10.1016/s0140-6736(19)31140-7

34.

McDonald

K. M.

(2013). Considering context in quality improvement interventions and implementation: Concepts, frameworks, and application. Academic Pediatrics, 13(6), S45–S53. https://doi.org/10.1016/j.acap.2013.04.013

35.

McGill

B. C.

Wakefield

C. E.

Hetherington

Munro

L. J.

Warby

Lau

Tyrrell

Ziegler

D. S.

O’Brien

T. A.

Marshall

G. M.

Malkin

Hansford

J. R.

Tucker

K. M.

Vetsch

(2020). “Balancing expectations with actual realities”: Conversations with clinicians and scientists in the first year of a high-risk childhood cancer precision medicine trial. Journal of Personalized Medicine, 10(1), 9. https://doi.org/10.3390/jpm10010009

36.

McGuier

E. A.

Kolko

D. J.

Klem

M. L.

Feldman

Kinkler

Diabes

M. A.

Weingart

L. R.

Wolk

C. B.

(2021). Team functioning and implementation of innovations in healthcare and human service settings: A systematic review protocol. Systematic Reviews, 10(1), 189. https://doi.org/10.1186/s13643-021-01747-w

37.

Michie

Johnston

(2012). Theories and techniques of behaviour change: Developing a cumulative science of behaviour change. Health Psychology Review, 6(1), 1–6. https://doi.org/10.1080/17437199.2012.654964

38.

Milasan

(2020). Recovery snapshots: A photographic exploration of Romanian mental health service users’ experience of recovery. Lancaster University.

39.

Morrow

Chan

Tucker

K. M.

Taylor

(2021). The design, implementation, and effectiveness of intervention strategies aimed at improving genetic referral practices: A systematic review of the literature. Genetics in Medicine, 23(12), 2239–2249. https://doi.org/10.1038/s41436-021-01272-0

40.

Nevedal

A. L.

Reardon

C. M.

Opra Widerquist

M. A.

Jackson

G. L.

Cutrona

S. L.

White

B. S.

Damschroder

L. J.

(2021). Rapid versus traditional qualitative analysis using the consolidated framework for implementation research (CFIR). Implementation Science, 16(1), 67. https://doi.org/10.1186/s13012-021-01111-5

41.

Nunns

(2009). Responding to the demand for quicker evaluation findings. Social Policy Journal of New Zealand, 34(34), 89–99. ■■■

42.

Palsola

Renko

Kostamo

Lorencatto

Hankonen

(2020). Thematic analysis of acceptability and fidelity of engagement for behaviour change interventions: The Let’s Move it intervention interview study. British Journal of Health Psychology, 25(3), 772–789. https://doi.org/10.1111/bjhp.12433

43.

Price

Smith

Dakin

Kang

Eibich

Cook

Gray

Harris

Middleton

Gibbons

Benedetto

Smith

Dawson

Fitzpatrick

Sayers

Miller

Marques

Gooberman-Hill

Blom

… Beard

(2019). The arthroplasty candidacy help engine tool to select candidates for hip and knee replacement surgery. Health Technology Assessment, 623(32), 1–216. https://doi.org/10.3310/hta23320

44.

Proctor

Silmere

Raghavan

Hovmand

Aarons

Bunger

Griffey

Hensley

(2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health, 38(2), 65–76. https://doi.org/10.1007/s10488-010-0319-7

45.

QSR (2019). NVivo qualitative data analysis version 12 Plus. QSR International Pty Ltd.

46.

Rankin

N. M.

Lai

Miller

Beale

Spigelman

Prest

Turley

Simes

(2018). Cancer multidisciplinary team meetings in practice: Results from a multi‐institutional quantitative survey and implications for policy change. Asia‐Pacific Journal of Clinical Oncology, 14(1), 74–83. https://doi.org/10.1111/ajco.12765

47.

Rapport

Francis-Auton

Cartmill

Ryder

Braithwaite

Clay-Williams

(2020). A mobile methods pilot study of surgical spaces: ‘Fit for purpose? Organisational productivity and workforce wellbeing in workspaces in hospital’ (FLOURISH). BMC Health Services Research, 20(1), 78. https://doi.org/10.1186/s12913-020-4938-8

48.

Rapport

Smith

O’Brien

T. A.

Tyrrell

V. J.

Mould

E. V.

Long

J. C.

Gul

Braithwaite

(2020). Development of an implementation and evaluation strategy for the Australian ‘Zero childhood cancer’ (Zero) program: A study protocol. BMJ Open, 10(6), e034522. https://doi.org/10.1136/bmjopen-2019-034522

49.

Riggs

Muyeen

Brown

Dawson

Petschel

Tardiff

Norman

Vanpraag

Szwarc

Yelland

(2017). Cultural safety and belonging for refugee background women attending group pregnancy care: An Australian qualitative study. Birth, 44(2), 145–152. https://doi.org/10.1111/birt.12272

50.

Riggs

Yelland

Mensah

F. K.

Gold

Szwarc

Kaplan

Small

Middleton

Krastev

McDonald

East

Homer

Nesvadba

Biggs

Braithwaite

Brown

S. J.

(2021). Group pregnancy care for refugee background women: A codesigned, multimethod evaluation protocol applying a community engagement framework and an interrupted time series design. BMJ Open, 11(7), e048271. https://doi.org/10.1136/bmjopen-2020-048271

51.

Ritchie

Spencer

(1994). Qualitative data analysis for applied policy research. In Bryman

Burgess

R. G.

(Eds.), Analysing qualitative data (pp. 173–194). Routledge.

52.

Rixon

Baron

McGale

Lorencatto

Francis

Davies

(2016). Methods used to address fidelity of receipt in health intervention research: A citation analysis and systematic review. BMC Health Services Research, 16(1), 663–663. https://doi.org/10.1186/s12913-016-1904-6

53.

Rolfo

Manca

Salgado

Van Dam

Dendooven

Ferri Gandia

Rutten

Lybaert

Vermeij

Gevaert

Weyn

Lefebure

Metsu

Van Laere

Peeters

Pauwels

Machado Coelho

(2018). Multidisciplinary molecular tumour board: A tool to improve clinical practice and selection accrual for clinical trials in patients with cancer. ESMO Open, 3(5), e000398. https://doi.org/10.1136/esmoopen-2018-000398

54.

Rushforth

Greenhalgh

(2020). Personalised medicine, disruptive innovation and ‘trailblazer’guidelines: Case study and theorization of an unsuccessful change effort. Accepted Milbank Quarterly, 98(2), 581–617. https://doi.org/10.1111/1468-0009.12455

55.

Sekhon

Cartwright

Francis

J. J.

(2017). Acceptability of healthcare interventions: An overview of reviews and development of a theoretical framework. BMC Health Services Research, 17(1), 88. https://doi.org/10.1186/s12913-017-2031-8

56.

Sharma

Walton

Manning

(2021). Social determinants of health influencing the New Zealand COVID-19 response and recovery: A scoping review and causal loop diagram. Systems, 9(3), 52. https://doi.org/10.3390/systems9030052

57.

Smith

(2022). Sentiment analysis for use within rapid implementation research. How far and fast can we go? Methodology and methods of implementation science. In Rapport

Clay-Williams

Braithwaite

(Eds.), Implementation science (1st ed.). Taylor & Francis Group. https://doi.org/10.4324/9781003109945-34

58.

Smith

Braithwaite

O’Brien

Smith

Tyrrell

Mould

Long

Rapport

(2022). Re-imagining the data collection and analysis research process by proposing a rapid qualitative data collection and analytic roadmap applied to the dynamic context of precision medicine. International Journal of Qualitative Methods, 21, 160940692211030. https://doi.org/10.1177/16094069221103097.

59.

Smith

Lee

M. D.

Ellis

L. A.

Ijaz

Yin

(2021). Developing a novel psychographic-behavioral qualitative mapping method for exergames. International Journal of Serious Games, 8(2), 87–107. https://doi.org/10.17083/ijsg.v8i2.422

60.

Smith

Rapport

O’Brien

T. A.

Smith

Tyrrell

V. J.

Mould

E. V. A.

Long

J. C.

Gul

Cullis

Braithwaite

(2020). The rise of rapid implementation: A worked example of solving an existing problem with a new method by combining concept analysis with a systematic integrative review. BMC Health Services Research, 20(1), 449. https://doi.org/10.1186/s12913-020-05289-0

61.

Smith

Alvand

Locock

Ryan

Smith

Bayliss

Wilson

Price

(2020). Partial or total knee replacement? Identifying patients’ information needs on knee replacement surgery: A qualitative study to inform a decision aid. Quality of Life Research, 29(4), 999–1011. https://doi.org/10.1007/s11136-019-02381-9

62.

Tran Smith

Padgett

D. K.

Choy-Brown

Henwood

B. F.

(2015). Rebuilding lives and identities: The role of place in recovery among persons with complex needs. Health Place, 33, 109–117. https://doi.org/10.1016/j.healthplace.2015.03.002.

63.

Vindrola-Padros

(2021). Can we re-imagine research so it is timely, relevant and responsive? Comment on “experience of health leadership in partnering with university-based researchers in Canada: A call to ‘re-imagine’ research”. International Journal of Health Policy and Management, 10(3), 172–175. https://doi.org/10.34172/ijhpm.2020.43

64.

Vindrola-Padros

Chisnall

Cooper

Dowrick

Djellouli

Symmons

S. M.

Martin

Singleton

Vanderslott

Vera

Johnson

G. A.

(2020). Carrying out rapid qualitative research during a pandemic: Emerging lessons from COVID-19. Qualitative Health Research, 30(14), 2192–2204. https://doi.org/10.1177/1049732320951526

65.

Walton

Spector

Tombor

Michie

(2017). Measures of fidelity of delivery of, and engagement with, complex, face‐to‐face health behaviour change interventions: A systematic review of measure quality. British Journal of Health Psychology, 22(4), 872–903. https://doi.org/10.1111/bjhp.12260

66.

Wandersman

Chien

V. H.

Katz

(2012). Toward an evidence-based system for innovation support for implementing innovations with quality: Tools, training, technical assistance, and quality assurance/quality improvement. American Journal of Community Psychology, 50(3–4), 445–459. https://doi.org/10.1007/s10464-012-9509-7

67.

Weiner

B. J.

Lewis

C. C.

Stanick

Powell

B. J.

Dorsey

C. N.

Clary

A. S.

Boynton

M. H.

Halko

(2017). Psychometric assessment of three newly developed implementation outcome measures. Implementation Science, 12(1), 108. https://doi.org/10.1186/s13012-017-0635-3

68.

Wiig

Aase

Johannessen

Holen-Rabbersvik

Thomsen

L. H.

van de Bovenkamp

Bal

Ree

(2019). How to deal with context? A context-mapping tool for quality and safety in nursing homes and homecare (SAFE-LEAD context). BMC Research Notes, 12(1), 259. https://doi.org/10.1186/s13104-019-4291-3

69.

Willemsen

A. E.

Krausz

Ligtenberg

M. J.

Grünberg

Groen

H. J.

Voest

E. E.

Cuppen

E. P.

van Laarhoven

H. W.

van Herpen

C. M.

(2019). Molecular tumour boards and molecular diagnostics for patients with cancer in The Netherlands: Experiences, challenges, and aspirations. British Journal of Cancer, 121(1), 34–36. https://doi.org/10.1038/s41416-019-0489-3

70.

Wong

Mayoh

Lau

L. M. S.

Khuong-Quang

D.-A.

Pinese

Kumar

Barahona

Wilkie

E. E.

Sullivan

Bowen-James

Syed

Martincorena

Abascal

Sherstyuk

Bolanos

N. A.

Baber

Priestley

Dolman

M. E. M.

Fleuren

E. D. G.

… Cowley

M. J.

(2020). Whole genome, transcriptome and methylome profiling enhances actionable target discovery in high-risk pediatric cancer. Nature Medicine, 26(11), 1742–1753. https://doi.org/10.1038/s41591-020-1072-4

71.

Zebrowski

A. M.

Ellis

D. E.

Barg

F. K.

Sperber

N. R.

Bernhardt

B. A.

Denny

J. C.

Dexter

P. R.

Ginsburg

G. S.

Horowitz

C. R.

Johnson

J. A.

Levy

M. A.

Orlando

L. A.

Pollin

T. I.

Skaar

T. C.

Kimmel

S. E.

(2019). Qualitative study of system-level factors related to genomic implementation. Genetics in Medicine, 21(7), 1534–1540. https://doi.org/10.1038/s41436-018-0378-9

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.42 MB