Abstract
Our implementation science study focuses on implementing a new way of practice and offers methodological specificity about how to rapidly investigate an individually tailored precision medicine intervention. A qualitative study advancing a new methodology for speedily identifying barriers and enablers to implementation in the context of childhood cancer. Data were collected through rapid ethnography, coded using the Consolidated Framework for Implementation Research, and analysed by Sentiment Analysis. Thirty-eight data collection events occurred during 14 multidisciplinary tumour board meetings, 14 curation meetings, and 10 informal conversations. Sentiment Analysis distilled Consolidated Framework for Implementation Research codes to reveal key barriers and enablers to implementation. A traffic light labelling system has been used to present levels of positivity and negativity (green for strong enablers and red for strong barriers), highlighting levels of concern regarding implementation. Within the intervention design characteristics, “Adaptability” was the strongest enabler and “Design quality and safety” the strongest barrier. Among the contextual factors: “Networks and communication” were the strongest enabler, and “Available resources” were the strongest barrier. Overall, there was a higher percentage of negative sentiment towards intervention design characteristics and contextual factors than positive sentiment, while more concerns were raised about intervention design factors than contextual factors. This study offers a rapid qualitative data collection and analytic methodological roadmap for establishing barriers and enablers to a paediatric precision medicine intervention.
Introduction
The ZERO Childhood Cancer Personalised Medicine Model
The ZERO Program is an innovative Program that involves a transdisciplinary approach to decision-making in developing individually tailored cancer treatments for children through its embedded early phase clinical trial: PRecISion Medicine for Childhood Cancer (PRISM). The ZERO Program and PRISM trial aim to carry out tumour molecular profiling and drug screening by completing and returning this evidence together with an interpretation of data to the treating physician in a clinically relevant timeframe. Curation meetings (CMs) and a national multidisciplinary tumour board (MTB) are newly formed strategies designed for the national clinical trial (Rankin et al., 2018; Rolfo et al., 2018). Attendees include representation across a range of specific clinical and scientific areas working together to adopt a new way of delivering childhood cancer care. For example, the diagnostic platform includes comprehensive testing of both tumour and patient genome. The information derived from the tests are assessed in CMs that include genomic, drug curation, and biological scientists, bioinformaticians, molecular oncology clinicians, and clinical geneticists who critically appraise the validity of the sample, variants of known and unknown significance, and determine clinical significance and relevance before being shared through a national MTB. The MTB consists of paediatric oncologists, molecular oncology specialists, genomic and drug curation scientists, and treating clinicians. They aim to assess and reach a consensus on the information tabled from the curation teams. Clinically relevant information is fed back to the treating physician, that is, if there is an actionable recommendation (novel drug target/referral to cancer predisposition/change in diagnosis).
An Individually Tailored Intervention
The complete process of delivering precision medicine in childhood cancer care, from testing and analysis, to discussion and information sharing followed by a precision medicine recommendation, is the ZERO Program’s intervention (Baker et al., 2010). This significant departure from ‘standard’ oncological approaches (National Academies of Sciences & Medicine, 2016) bases treatment recommendations on potential drug targets and genomic data – thus introducing a new framework of evidence-based practice. The content of each child’s case includes discussions at CMs. Following CMs, the clinically relevant information is shared through the national MTB. This has implications for changes to real-time delivery of care and effects (Rowbotham et al., 2019). Precision medicine is placing new demands on childhood cancer care. For example, healthcare professionals are having to adopt new ways of practice that include changing their behaviour, such as role changes, professional changes in decision-making and new ways of communication as well as acquiring and assimilating new workflows and requiring ever-increasing knowledge of genomic medicine (Horowitz et al., 2019; McGill et al., 2020; Willemsen et al., 2019; Wong et al., 2020). Likewise, our complex intervention requires a change in the behaviour of healthcare professionals to meet the demands of changing practice making healthcare professionals the focus of this intervention (Keith et al., 2017; Morrow et al., 2019; Rixon et al., 2016; Taylor et al., 2014). Change can be challenging without a clear understanding of which determinants act as enablers to positive implementation and which are barriers (Smith et al., 2021a). Understanding enablers of and barriers to change is vital to tailoring implementation strategies and creating greater surety over long-term effectiveness (Atkins et al., 2017).
The Consolidated Framework for Implementation Research
The Consolidated Framework for Implementation Research (CFIR) (Damschroder et al., 2009) is a determinant framework that can help identify and explain factors that influence implementation and recognise how multiple levels of behaviour change, operate and influence implementation (Squires et al., 2015). CFIR includes factors related to the “intervention design” itself, the “outer context” (e.g., the level at which the implementing organisation is networked with other organisations/external influences on the intervention), the “inner setting” (e.g., the internal networks and communications that can influence implementation), “individuals involved” (e.g., their knowledge and beliefs), and the “process” under consideration (e.g., planning and reflection work). However, some authors argue that the framework may require tailoring of factor definitions, in order to fit the needs of any given study context (Damschroder et al., 2022; Keith et al., 2017; Means et al., 2020; Safaeinili et al., 2020).
McDonald (2013) framed CFIR in terms of its intervention design characteristics but also conceptualised grouping the contextual domains together, a practice subsequently followed by others (Holen-Rabbersvik et al., 2020; Wiig et al., 2019). In line with this conceptualisation, we grouped the contextual domains (inner, outer, individual characteristics and processes) for greater flexibility in line with the new practice demands of the transdisciplinary groups interacting in this precision medicine study with CFIR’s intervention design characteristics depicting the intervention design. This included factors such as, “Evidence, strength and quality” (do people believe in the strength of the evidence available) and “Complexity” (views on whether the intervention is too complex to implement). We extended McDonald’s (2013) conceptualisation of CFIR to indicate combined and individual influencers of implementation to ensure a comprehensive approach towards our assessment of intervention implementation within this complex context.
A Methodological Roadmap to Speeding Up Qualitative Data Collection and Analysis
The speed of delivering information is no less important than its accuracy (McNall et al., 2004). The CFIR approach has been criticised when applied recently to a precision medicine context for being time-intensive and potentially delaying the identification of findings (Best et al., 2021). In line with this criticism, the CFIR approach indicates some recent attempts by CFIR’s creators and other authors to speed up the qualitative data collection and deductive analytic process for contexts with rapid change (Gale et al., 2019; Nevedal et al., 2021; Zucco et al., 2021).
However, despite the potential to speed up the qualitative data collection and analytic process, and fit for within high-stake contexts (e.g., precision medicine contexts, COVID-19, etc.), this idea has not been taken up in recent qualitative research studies using CFIR approaches applied to precision medicine models of care (Best et al., 2021; Levy et al., 2019; McGill et al., 2020; Zebrowski et al., 2019). For example, these studies applied static interviews and lengthy transcription and analysis procedures to make sense of data (Best et al., 2021; McGill et al., 2020; Zebrowski et al., 2019). Recently such qualitative approaches have been criticised for being narrow in scope or superficial in eliciting contextual data (Haines et al., 2021).
In our implementation science study, we propose the need to adopt qualitative methods that could instead deliver speedy actionable results in a complex precision medicine context yet still maintain rigour (Nevedal et al., 2021; Smith et al., 2020b). Therefore, we sought rapid methods that could contend with the intervention context and chose to apply a novel, rapid ethnographic methodology to generate findings in a timely manner and ensure the study researcher moves with study participants as they go about their work lives (Rapport et al., 2020). In this way, rapid ethnography allows examination of the people, the organisations, and the different system-level under consideration and change over time in a contemporary fashion (Rapport et al., 2020). However, speedy data collection is of limited use if the data analysis still proceeds at the painstakingly slow rate that is typical of most qualitative research (Millen et al., 2000). Sentiment Analysis is a candidate for resolving that deficit in a rapid way. In line with this paradigm shift, our implementation science study proposes the need for rapid qualitative analysis of the deductive CFIR approach (Gale et al., 2019; Nevedal et al., 2021; Zucco et al., 2021) through the use of Sentiment Analysis. Sentiment Analysis (Pang & Lee, 2008; Pinto et al., 2018) identifies and categorises opinions to determine whether participants’ attitudes toward a particular topic (intervention or product, etc.) are positive or negative. Sentiment Analysis, therefore, has the potential to help in the rapid analysis of the CFIR approach and is in keeping with a new awareness of the value of turning from implementation to rapid implementation in all respects (Smith et al., 2020b). Sentiment Analysis offers the CFIR approach a new way to rapidly detect the range of sentiments displayed and sentiment polarity from large qualitative data sets while providing timely results in rapidly changing environments (Coppersmith et al., 2015; Golder & Macy, 2011; Gore et al., 2015; Zhou et al., 2015). Sentiment Analysis has successfully been applied to several health interventions to date, such as vaccination (Dunn et al., 2015; Naseem et al., 2021; Zhou et al., 2015), mental health (Oyebode et al., 2020) and the delivery of outcomes such as those relating to seasonal affective disorder and obesity (Coppersmith et al., 2015; Golder & Macy, 2011; Gore et al., 2015). The CFIR approach will be helped by Sentiment Analysis by maturing the framework (Smith, 2022) because of its speedy processing of large qualitative data sets (such as rapid ethnographic fieldnotes) in a short amount of time (Fondevila-Gascón et al., 2016; Morgan et al., 2021; Willson et al., 2021; Zhou et al., 2015).
This paper presents a novel methodological roadmap on how we can speed up qualitative data collection and analysis to provide an understanding of qualitative data in a highly complex, multi-faceted and ever-changing tailored intervention. It also examines barriers and enablers to implementation and has an overarching goal of providing insights into key barriers to implementation success (Damschroder et al., 2009).
Study Aim
Study aims were derived from multi-stakeholder meetings and discussions between ZERO Program partners, university-based researchers, and implementation scientists in Australia.
This study aimed to: (1) Provide a methodology to identify barriers and enablers to the delivery of a childhood cancer personalised medicine Program (the ZERO Program). (2) Assess key enablers and barriers in real-time care delivery according to stakeholders’ strength of feeling and order of importance.
Method
Study Setting
Data were collected from the multicentre study’s eight sites (eight hospitals in the ZERO Program and PRISM) through a central site in Sydney, hosting CMs and MTBs in person and available via video link for those not in the state. In addition, as part of a rapid ethnography, purposeful informal conversations took place at the central site to capture the views and experiences of key players regarding the complex intervention.
Study Population
The ZERO Program Operations and Program Managers identified key players to take part in data collection who were directly responsible for design, delivery and oversight. The researchers regularly sought out the lead person making the decisions about progress and implementation. Key players comprised of various clinical and non-clinical staff members representing different disciplines. Work across sites was collaborative.
While data were collected from one central site, CMs and MTBs brought JS, a male researcher who is experienced in rapid ethnography, into contact with: clinicians, curation specialists, administrators, managers, technicians and laboratory scientists, who were themselves in discussion with treating paediatric oncologists of children with high risk paediatric malignancies (expected survival <30%).
The Application of Rapid Ethnography
Data Collection
Ethical approval was granted by the Hunter New England Research Ethics Committee, New South Wales, Australia. Approval number: 2019/ETH12025. Purposive sampling was used, with an opt-out consent process for the rapid ethnographic research approved and followed. For observations, this process is appropriate in identifying and managing participants and non-participants as any number or combination of healthcare professionals or other Program members (participants) may also come into contact with the study researcher during a given observation period. It is also hard to anticipate participants in advance of the research taking place. Program managers provided advance notice and information about the study to various teams involved, anticipated to be involved or present at planned observations. JS also wore a badge detailing his name and a “research in progress” annotation so that incidentally observed participants were made aware of the research taking place and could opt-out of study involvement (become a non-participant). The study had no staff opt-outs. As opposed to some of the historical anthropology studies suggesting spending around 2 years in the field, our implementation science study consisted of a 9-month intensive rapid ethnographic data collection period (Vindrola-Padros & Vindrola-Padros, 2017) whilst simultaneously focusing on ongoing intervention optimisation (Chambers et al., 2013; Rapport et al., 2020). This intensive timeframe enabled JS as the lone field researcher to immerse himself in the ZERO Program context, attend CMs and MTBs, and undertake observations and informal conversations with individuals in the ZERO Program. Data were collected from clinicians and other stakeholders in line with study aims, examining: (a) how the ZERO Program functioned, (b) Program activities and meetings (to determine the faithfulness to or departures from the intended intervention), (c) collaborations and interactions (to clarify characteristics of individuals and the influence of setting characteristics) and (d) decision-making and information-sharing. While the focus was on barriers and enablers of intervention design characteristics, it was also important to consider contextual factors to delivery and implementation, allowing a nuanced picture to emerge of individual and group activity, environments, events, and personal circumstances. Fieldnotes were completed during each event taking place for reliable record-keeping (Smith et al., 2020a). Fieldnotes included commentary on the dynamics of different meetings and non-verbal communication. As data were collected, they were de-identified, and quotations were anonymised. All participants were given pseudonyms ready for the application of a CFIR coding framework according to a CFIR codebook. There was no attempt to capture data saturation, in line with research in ethnography (Jones & Smith, 2017; Streubert & Carpenter, 1996) and reflexive thematic analysis (Braun & Clarke, 2021). We recognise that our ethnographic contribution is interpretative, and meaning is generated through the interpretation of data.
Analysis
Methodological roadmap.
CFIR: Intervention characteristics, inner setting, outer setting, individual characteristics, process implementation.
Sentiment Analysis: moderately positive, very positive, moderately negative, very negative.
Implementation Scientists (JS, FR, JB, JL) helped clarify barriers or bottlenecks to implementation. Groupwork led to a final CFIR framework indicating key aspects of Program delivery. We represent our qualitative findings through a visual presentation to appeal to and facilitate understanding amongst clinicians and other stakeholders (Smith et al., 2020c, 2021b).
Results
Data capture events.
All individuals have een given pseudonyms to protect their identity.
Henrietta and Harriet met JS together.
Grading and Visual Presentation of Data
Grading data: A visual presentation of the intervention design and contextual characteristics across data collection events.
KEY: Traffic light system: Barriers are represented in red = The total negative percentage is higher than the total positive percentage; Enablers are shown in green = The total positive percentage is higher than the total negative percentage; Barrier & Enabler are represented in Orange: The total number of barriers and enablers are equal.
N= Total number of references coded across sentiment; n= Number of references coded across sentiment for each data collection activity; 0= Participant had the opportunity to discuss this, but did not discuss it; NA= not applicable as this component was not present during data capture.
There was a higher percentage of total negative sentiments for all factors (N) according to each data capture event (n) for both intervention design characteristics (negative sentiment: N = 165, n = 114, 69.1%) and contextual characteristics (negative sentiment: N = 71, n = 38, 53.5%) compared to positive sentiment for intervention design characteristics (N = 165, n = 51, 30.9%) and contextual characteristics (N = 71, n = 33, 46.5%). This indicates perceptions of more barriers than enablers, particularly regarding key implementation features available in Table 3. For individual intervention design characteristics factors, “Adaptability” was the strongest enabler (N = 21, n = 14, 66.7%) and “Design quality and safety” the strongest barrier (N = 14, n = 13, 92.9%) while for contextual characteristics, “Networks and communication” (N = 8, n = 8, 100.0%) was the strongest enabler, and “Available resources” the strongest barrier (N = 4, n = 4, 100.0%) (Table 3).
Intervention Design Characteristics: Key Themes
Adaptability: An Enabler Overall for Intervention Application and Applicability
“Adaptability” is the degree to which the intervention fits the needs of the people, sites and settings and appears to be a strong enabler to overall intervention applicability. To be adaptable relies on the intervention’s ability to accommodate people’s behaviours and relationships, but also healthcare structures and systems through which the intervention is being introduced. For example, positive sentiment from MTBs towards the adaptation of treatment protocols and changes to clinical practice that might enable children, who were not originally eligible to be part of the precision medicine program, to now be eligible. Within the CMs, adaptability as an enabler for intervention application, related to the adaptation of databases and technologies within healthcare systems, the addition of new technologies (for example, a “tape station” that would enable methylation to be an internal as opposed to an outsourced process), and the updating of existing technology. No data were gleaned for ‘adaptability’ from informal conversations.
Design Quality and Safety: A Barrier to Implementation
“Design quality and safety” relates to how the intervention is perceived to ensure such aspects as quality and safety as an absolute priority within the intervention. However, this was a component that was perceived to be a barrier to implementation. This was due to the identification of a small number of instances where the arrival of poor samples or quality-control issues relating to tumour samples were perceived as aspects not yet solved. For example, CMs focused on bad samples, small tumour samples, degraded material, and the lack of quality control in terms of potential processing errors and batch effects. MTBs focused on the quality and size of the tumour (coming from bone or containing lots of contaminated cells). No data were forthcoming on this aspect from individual informal conversations.
Cost: An Overall Barrier to Implementation
“Cost” relates to how much the intervention costs in terms of testing and data retrieval, which was seen as an overall barrier to implementation, with MTBs and CMs concentrating on costs to carry out extra tests. Yet, for CM attendees, there were some positive views expressed about finding new ways to reduce costs, such as upfront testing or conducting a pilot. During informal conversations, concerns were also voiced about costs, expressly the cost of storing and retrieving data from high-performance data storage systems.
Evidence, Strength and Quality: A Barrier to Implementation
“Evidence, strength and quality” relates to the strength of the evidence supported by either good strong high-quality evidence or weak evidence. When evidence, strength and quality from evidential sources was lacking (e.g., the strength of evidence in relation to a lack of current published literature), this was perceived as a barrier to informing diagnostic and treatment decisions, as key players questioned whether sources of information were reliable and continued to question whether more evidence was available. During MTBs, negative sentiments were expressed regarding a lack of information sources for paediatric treatment dosages, drug combinations, and tumour use during drug screening, as well as concerns about evidence-use in these areas. Within CMs, similar issues were raised regarding poor-quality information or lack of information sources on these topics. In CMs, positive sentiments were also voiced around cases that presented good-quality data shared on a server. One informal conversation with Lesley focused on concerns over poor-quality evidence regarding toxicity and dosage and the need for recommendations for new toxicity and dosage practices to build a stronger evidence-base.
Speed of Delivery: A Barrier to Implementation
“Speed of delivery” of the intervention was perceived as both a barrier and an enabler to implementation, while the overall strength of feeling placed this as a barrier to implementation (delays in delivery). MTB’s focused on problems with delays for methylation, while CMs questioned why methylation lagged behind the presentation of cases at MTB, such that results were not timely and had to be reviewed a second time, by which time people had forgotten about the case. This led to questions about how to present information more effectively. CMs indicated an equal mix of negative and positive sentiment regarding optimising curation workflow. In informal conversations, speed was perceived as an enabler to optimising turnaround through automation processes but equally a barrier; reducing time meant having less time to discuss the complexity of cases.
Complexity: A Barrier to Intervention Implementation
“Complexity” relates to the perceived difficulty of implementing the intervention, the more complex an intervention, the more difficult it will be to implement. In MTBs difficulties concerned: (1) what to report in complex cases; (2) what recommendations to make; and (3) what the treating oncologist should be communicating to patients in a complex case. In CMs, negative sentiments focused on the complexity of managing issues raised in MTBs, specifically around the complex nature of precision medicine and the interpretation of novel data. From informal conversations, perceived difficulties surrounded the complexity of individualising treatments and managing patients with complex disease presentations, especially children who had exhausted all other avenues of treatment; as Joanna stated, complex cases where children’s only hope for survival is enrolment on the ZERO Program or PRISM trial.
Trialability: A Barrier and an Enabler to Implementation
“Trialability,” the testing of the intervention on a small scale, helps to show the intervention can be achieved before it is implemented on a much larger scale. Trialability was both perceived as a barrier and an enabler to implementation. Trialability (represented by a pre-ZERO efficacy study: TARGET) was not discussed during either MTBs or CMs, but in informal conversations, it was raised on occasion. Some study participants, like Henrietta and Harriet, compared the trialability of TARGET to the provision of comparative data for the full PRISM national trial and that the efficacy pilot study aimed at achieving feasibility had received better quality tumours compared to the national PRISM trial. In this respect, Henrietta and Harriet felt that TARGET received better support in the form of an engaged and willing staff. It is important to note, that TARGET was a much smaller feasibility study and easier to manage when compared to the national PRISM study. No data were returned to families or clinicians regarding whether tests for precision medicine could be achieved. John echoed this position and further explained that PRISM morphed into something better than TARGET. It was generating evidence locally that would have relevance for the whole country, aligning to international standards.
Contextual Characteristics: Key Themes
Inner Setting
Networks and Communication: Enablers for Implementation
“Internal networks” were perceived as clear enablers and related to how individuals communicated with other members of their internal groups, with the success of this depending on the people in small collaborative teams, who then made-up Program networks. It was teamwork and how “networked” team members were at a local level that was the deciding factor for people to positively influence implementation. MTBs were seen as the place where staff could focus on cases and highlighted interdependencies between members of internal networks and specialists with unique skills on specific trials. CMs focused on the network of individuals (inner network) that empowered people to raise clinical questions. During informal conversations, individuals such as Henrietta and Harriet, Joanna and Diego spoke of key people in their inner network and staff members they relied upon. Overall, networks were seen to foster good communication and ongoing collaboration between staff members, helping people focus on clinically relevant questions.
Rapid Learning Systems: An Enabler to Implementation
“Rapid Learning Systems” (Riley et al., 2013) focus on learning and evidence development and were enablers to implementation, encouraging healthcare workers to be in an appropriate position to take advantage of sudden learning opportunities. MTBs and CMs included expert advisory clinicians (or mentors) who were seen as crucial for underpinning the positive effect of Rapid Learning Systems, alongside curation and bioinformatic specialists working in paediatric precision medicine. Evidence from MTBs indicated that Rapid Learning Systems helped junior staff and supported other clinicians and treating oncologists to discuss patients, while attendance at both MTBs and CMs offered opportunities for new Rapid Learning Systems to come to fruition in terms of ongoing learning and evidence development. Within CMs, senior clinicians’ and curation specialists’ knowledge helped conceptualise Rapid Learning activities, particularly in line with complex novel cases. There was no coded data on this subject from informal conversations.
Available Resources: A Barrier for Individuals
“Available resources,” in the form of infrastructure and time, were said to be barriers to implementation, while resources were said to have been diverted to the overall Program and ongoing operations. Overall, negative sentiment was expressed by individuals during informal conversations, with mention of annoying bottlenecks and challenges to resource delivery. Jenny, for example, remarked that there were problems with data storage and inadequate infrastructure to handle the speed and amount of data that came into the ZERO Program. Henrietta, Harriet and Jenny mentioned inefficiencies within resource use, and Henrietta and Harriet drew attention to problems with small amounts of tumours available for research purposes that impacted precision medicine recommendations and reports (reports include interpretation of test results, and potentially actionable recommendations, for example, novel drug targets and change in diagnosis) delivered to the treating physician.
Outer Setting
External Policies and Incentives: A Barrier to Implementation
“External policies and incentives” focus on regulations (government-led) and guidelines (surrounding cutting-edge work of the ZERO Program) with particular emphasis on the work of PRISM. Within CMs, the focus was on the restrictions resulting from regulation and the impact on access to trial data, Food and Drug Administration approval (FDA - Food and Drug Administration), and the ability to bypass “X” organisation to talk directly to “Y.” At MTBs, strongly voiced negative sentiment was expressed regarding drug access and trial data. External policies were also considered barriers to trial access and Therapeutic Goods Administration approval (TGA – Australia’s Food and Drug Administration). Nevertheless, during informal conversations, external policies shone as enablers, as John suggested bolting on external trials to the Program work to do a proper evaluation of trial effectiveness and access. John also mentioned that while compassionate drug access was effective, there was no way to evaluate it (uncontrolled anecdotal outcome and not useable from a clinical perspective), and thus evidence around the value of compassionate access became more anecdotal than concrete.
Cosmopolitanism: A Barrier to Implementation
“Cosmopolitanism” is the degree to which the organisation underpinning the work of the ZERO Program and PRISM was externally connected and networked with other organisations. Cosmopolitanism was perceived as a barrier in CMs and marginally a barrier in MTBs. MTBs focused more on the positive sentiment of the sponsor, the trial, and the external networks (such as visiting professors and genetic counsellors). CMs expressed more negative sentiments towards connectivity with other organisations, raising issues regarding getting feedback from others on the uncertainty of a test result and the problems stemming from a lack of firm relationships with drug companies, while positive sentiments focused on being able to seek national expert advice on novel test results. During an informal conversation about what was being delivered in the ZERO Program and PRISM, cosmopolitanism was seen as an enabler. Joanna commented that the ZERO Program delivered much more than other countries had been able to achieve as well as being able to deal with more complex cancer cases than in other countries, for example.
Implementation Process
Planning: A Barrier to Implementation
“Planning,” the actual process of preparing for implementation was clearly hindered (poor communication within planning) by work being undertaken during dial-in meetings at MTBs and CMs, as this impacted upon implementation process. As a result, planning was listed as a barrier to implementation. During MTBs and CMs, attendees noted that important details were often hard to hear. In addition, a number of conversations were interrupted by unplanned situations or issues with video cameras or personal dial-ins. This led to some meetings being abruptly abandoned before arrangements were finalised, as such cases were rearranged where they were to be presented at a later date.
Individual Characteristics
Knowledge and Beliefs: A Barrier to Implementation
“Knowledge and beliefs” relate to attitudes and beliefs held about current knowledge that represented a barrier to implementation. Lesley spoke informally to JS about her lack of knowledge in terms of a sense that technology provides a way to capture a lot of genomic data, yet there was much still unknown within the field of genomics on how to interpret some of this data, which resulted in holding firm to long-held beliefs about the right approach to take. This is known as “clinical equipoise,” being guided by one’s own standpoint. For example, key players mentioned that there was a general lack of knowledge in the wider medical profession (particularly amongst clinicians) about precision medicine and some aspects of Program development and delivery, which in turn influenced attitudes towards the intervention and led to a lack of familiarity with the full facts. Informal conversations with John and Connor, indicated that educational forums (including Continued Professional Development) were not being attended because of a general lack of interest in upskilling, leading to a lack of important complex, highly specialised understanding and ability. John remarked that there were educational barriers, such as in paediatric oncology, with people often uneducated about precision medicine. Henrietta and Harriet also discussed a lack of knowledge amongst colleagues and remarked that there was a clear need for more information upfront and educational events, which would lead to greater understanding and clearer collaboration, particularly for paediatric oncologists.
Summary of findings
In summary, the descriptions above highlight that the enablers listed were (in order of importance): (1) “Networks and communication” (N = 8, n = 8, 100%), (2) “Rapid Learning Systems” (N = 12, n = 11, 91.7%) and (3) “Adaptability” (N = 21, n = 14, 66.7%).
Intervention design characteristics that were barriers (in order of importance) were: (1) “Design quality and safety” (N = 14, n = 13, 92.9%), (2) “Cost” (N = 5, n = 4, 80.0%), (3) “Evidence, strength and quality” (N = 59, n = 44, 74.6%), (4) “Speed of delivery” (overall a barrier though included an enabler component) (N = 14, n = 10, 71.4%), (5) “Complexity” (N = 50, n = 35, 70.0%) and (6) “Trialability” (overall a barrier though it included an enabler component) (N = 2, n = 1, 50.0%).
Contextual factors that were barriers (in order of importance) were: (1) “Available resources” (N = 4, n = 4, 100%), (2) “External policy and incentives” (N = 11, n = 10, 90.9%), (3) “Planning” (N = 6, n = 5, 83.3%), (4) “Knowledge and beliefs” (N = 16, n = 10, 62.5%), and (5) “Cosmopolitanism” (N = 14, n = 8, 57.1%).
Reviewing intervention design characteristics and contextual factors to determine the order in which dominant barriers need addressing, the three most pressing issues were: (1) “Available resources” (N = 4, n = 4, 100%), (2) “Design quality and safety” (N = 14, n = 13, 92.9%), (3) “External policy and incentives” (N = 11, n = 10, 90.9%).
Overall, the individual intervention design characteristics and contextual characteristics, when considered in their constituent domains, in order of importance of domains (according to barriers) were: (1) “Implementation process” (N = 6, n = 5, 83.3%), (2) “Outer setting” (N = 25, n = 18, 72.0%), (3) “Intervention characteristics” (N = 165, n = 114, 69.1%), and (4) “Individual characteristics” (N = 16, n = 10, 62.5%) (Table 3).
Discussion
This study provides a methodological roadmap to help rapidly identify determinants of change. To our knowledge, this is the first study to: (1) specify aspects of determinants of change in paediatric precision medicine to operationalise a method to assess intervention design characteristics and contextual characteristics in real-time, (2) convert rapid ethnographic datasets according to strength of feeling against each factor, and (3) comprehensively “opinion-mine” to understand barriers and enablers to implementation in order of need. Thus, we locate this research in terms of both “the art and science of implementation” within the context of a complex tailored intervention in childhood cancer.
Implementation science can rapidly speed up the translation of basic and clinical genomic research findings by evaluating how individuals, groups and systems behave and then applying that knowledge to determine how well an intervention is being implemented by those involved across different system-levels in different contexts. For example, as new drugs are developed and as new gene targets are identified in precision medicine (Horton & Lucassen, 2019), this brings into question, how does implementing evidence-based practice fit within the ethos of implementation science, if that very evidence is still evolving (Chambers et al., 2016; Manolio et al., 2013; National Academies of Sciences & Medicine, 2016)? There is a new opportunity to learn from both genomics research and newly developed evidence on COVID-19 (Chambers, 2020; Randhawa et al., 2020; Wilder-Smith et al., 2020) since these two settings show evidence-based practice can and is implemented at the same time as we are still learning and at the same time as new evidence is still being generated (Curran et al., 2012; Lane-Fall et al., 2019; Smith et al., 2020b). For example, the rapid pace at which precision medicine changes and the subsequent demands this places on healthcare professionals is challenging researchers to use more innovative methodologies to deliver actionable results in a more rapid way (Smith et al., 2020b).
We applied a rapid ethnographic methodology with observation methods such as physically travelling alongside participants by attending multidisciplinary meeting arrangements and having informal conversations to generate rich contextual information on barriers and enablers of a precision medicine intervention. This was in line with recent criticism directed at more standard methods (interviews as the pre-eminent method of choice) in terms of their superficial elicitation of contextual data from interviews combined with their time and resource-intensive approach (Haines et al., 2021). Such slow and static methods like interviews appear counterintuitive to the speed with which precision medicine changes within the childhood cancer context. Therefore, our method has a comparative advantage over other qualitative methods currently used that fail to recognise the importance of context and not generating findings quickly and effectively. In addition, researchers should move towards determinant frameworks (barriers and/enablers) in precision medicine (Bangash & Kullo, 2020; Roberts et al., 2017) and speed up qualitative research quickly and effectively (Gale et al., 2019; Smith et al., 2020b, 2021b) to meet demands within high stake contexts (Smith et al., 2020b). To our knowledge, our study is the first of its kind to use a determinant framework (CFIR) (Damschroder et al., 2009) together with Sentiment Analysis (Pang & Lee, 2008; Pinto et al., 2018) to speedily determine barriers and enablers of a precision medicine intervention. We see an opportunity to apply our rapid data collection and analytic roadmap to other precision medicine settings (e.g., adults with rare cancers or all children with cancer or genetic referral and testing practices) to rapidly identify barriers within these settings. Our methodology, therefore, has application beyond high risk childhood cancers, and can be used in other precision medicine contexts as a way to avoid the widely held practice of scaling a precision medicine intervention across entire states and populations without prior identification of the types of internal barriers or wider system bottlenecks that could hinder successful implementation (Bangash & Kullo, 2020; Roberts et al., 2017). The descriptor “dynamic” that we use throughout our study refers to the rapid evolution of the technology used in precision medicine, the constant refinement of processes, and the constantly changing knowledge base as new genes are studied and reported. Our methodological roadmap may therefore have intuitive appeal within (Lewis et al., 2022) and across broader dynamic and rapidly evolving environments such as its application within the COVID-19 vaccination program as a diagnostic tool. Our methodological roadmap would work well in assessing the national vaccination program amid the emergence of rapid viral variants causing ongoing changes to the evidence base as new strains of COVID-19 evolve, such as the global spread of the Omicron subvariant BA.1 and stealth Omicron BA.2 or Fluorna (Chadeau-Hyam et al., 2022; Dejnirattisai et al., 2022; Grabowski et al., 2022; He et al., 2021; Lyngse et al., 2022).
Our methodological roadmap has proved useful to implementation science to bridge this temporal evidence-to-practice gap and bring context to the forefront of implementation science. It provides iterative improvements in efficiencies while at the same time speedily identifying and rapidly monitoring barriers of greatest concern. This could lead to not only a refinement of evidence-based practice but enable ongoing implementation at the same time that new genomic evidence is rapidly being generated. Similarly, whilst more negative than positive sentiments were expressed overall in our study across intervention design characteristics and contextual characteristics, this is perhaps not unexpected when specific to this context, there is a need for ongoing evidence development at the same time as implementation (Chambers et al., 2016). This is especially the case for intervention design characteristics that pertain to “evidence strength and quality” with a clear lack of published data in this context (in particular regarding drug access and drug combinations in paediatric precision medicine). It should also be reiterated that this is a novel Program, being tested in a clinical trial, involving transdisciplinary groups, coming together in new configurations, which are expected to change both practice and care management plans within what could be seen to be an adequately-functioning system (Rushforth & Greenhalgh, 2020). As Sirkin et al. (2005) discovered (in business), there are always conflicting opinions over the changes most needed and the essential milestones necessary for measuring success. Our findings should be considered in light of this fact, while our study was delivered in the context of an early-phase clinical trial, with clear boundaries and time dependencies. However, enablers were in evidence, most noticeably in relation to: “Networks and communication” (a contextual factor), “Rapid Learning Systems” (a contextual factor) and “Adaptability” (an intervention design characteristic’).
To explore this further, it is possible to link contextual factors such as “Networks and Communication” and “Rapid Learning Systems” with the intervention design characteristic; “Adaptability” (Damschroder et al., 2009). Echoing the literature on the importance of interconnections (Damschroder et al., 2022; Safaeinili et al., 2020) together these three enablers indicate how people share educational events or experience supportive learning environments and how influential this can be in helping them to adapt to change to successfully implement the intervention; engendering new behaviours and new organisational work patterns. The ZERO Program includes groups of key players working together through larger networks across sites, sharing evidence and knowledge in both CM’s and MTB’s.
Context does indeed matter (Damschroder et al., 2009; Greenhalgh et al., 2004; Hawe et al., 2004; Kemp, 2016; McDonald, 2013; Safaeinili et al., 2020; Wong et al., 2020). Contextual factors clearly operate at multiple levels to impact on implementation strategy and effectiveness. Providing a methodology to identify and assess this brings us one step closer to uncovering the intervention “black box.” Our findings show that the inner layer is an important enabler, as other studies have identified, which report on the success of tailoring CFIR’s inner layers to reflect complexity within a healthcare system (Safaeinili et al., 2020).
Our implementation science study focused on the ongoing optimisation of an early-phase precision medicine intervention by rapidly collecting and analysing information amid ongoing change over time. We propose that our method can be used to optimise and improve the types of design and delivery of interventions through rapidly identifying barriers that can be proactively addressed/brought into focus as they arise, rather than not being flagged until full-scale evaluation occurs. Opportunistic iterative change in response to rapid feedback supports intervention optimisation of the intervention in real time. The opportunity, therefore, lies in understanding the sentiment of those at the coalface. A central part of our method is the ongoing improvement to flatten out barriers before scaling a pilot program by orders of magnitude because, by the time the intervention is fixed in effectiveness research, it is too late to change or optimise. In taking this research forward, the method could be integrated to examine the fidelity (as clearly defined, a faithful implementation) and contextual aspects (as implementation tailored to the circumstances, situation and needs of the intervention recipients). Understanding this is vital to tailoring implementation strategies and creating greater surety over their long-term effectiveness and scale-up. Using rapid ethnography, a rich, in-depth and “naturalistic” dataset can be assembled from which to identify vital CFIR factors. Furthermore, there is an opportunity to study how data from both rapid ethnography and clinical trials improve both implementation and treatment outcomes in real-world practice settings (Chambers et al., 2016). Initial evidence for a newly developed roadmap to speeding up qualitative data collection and analysis in precision medicine has now been established. Our intention in the future is for our methodology to be embedded within rapid-cycle feedback loops (Braithwaite et al., 2014; Rapport et al., 2020), where actionable findings are shared with stakeholders during implementation in order to share interim discoveries and allow for Program corrections over time. Our goal in the future is ongoing optimisation and co-design of tailored implementation strategies. To achieve this, we need to be able to use our visual presentation of qualitative findings as a visual tool to appeal to clinicians and other stakeholders and create a culture of learning in ways that other qualitative approaches may not be able to. With our newly established methodological roadmap, the future is bright to be able to speedily identify specific barriers that appear at different time points and address them in real-time, and through our visual presentation of the qualitative data we now have a way of sense-making around immediate Program adaptions.
Limitations
We made the best use of purposeful data, our number of observations is suffice from informal conversations with transdisciplinary healthcare professionals. Still, we recognise that this type of real-world data is dependent on those people most willing to speak to researchers during a busy workday and may not have included all the key program influencers. This study has depended on relatively small samples; expanding the population demographic would offer opportunities for a wider translation of findings. Finally, our study must be recognised as a snapshot in time where attitudes may change over the longitudinal course of a clinical trial.
Conclusion
This study provides a methodological roadmap for speedily evaluating complex interventions in precision medicine and advances implementation theory and research by providing a novel approach to conduct a barriers and enablers analysis. It is timely, as there are no guidelines on how to establish the most effective way of doing this speedily in complex tailored interventions. We have attempted to provide a way to qualify and grade the level of positivity or negativity towards implementation, using intervention design characteristics and contextual assessment. This will not only support a growing awareness of the value of context but also the successful prediction of intervention outcomes whilst highlighting the value of flexibility in tailoring interventions, as situations and circumstances can and do change. Careful ongoing evaluation (Braithwaite et al., 2014) and intervention optimisation (Chambers et al., 2013) and rapid qualitative assessment can mitigate against the negative consequences of barriers to implementation and ensure clearer predictions of implementation effectiveness.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the New South Wales Health.
