Exploring healthcare professionals’ perceptions of artificial intelligence: Validating a questionnaire using the e-Delphi method

Workforce perception of AI

Workforce perception is a powerful indicator of organisational readiness and requires consideration in this new age of technological advancement. ¹² Technology adoption theories developed during the third industrial revolution in the 1980s, for example the popular technology acceptance model ¹³ or diffusion of innovations theory, ¹⁴ explored perception as a predictor of use and acceptance which was useful for designers to improve technology characteristics and function. These theories found that the adoption of computers in healthcare was influenced by the prior experience, knowledge and skill set of the user. ¹⁵ If the technology aligned with professional values, was trusted, easy to use and improved job performance the healthcare professional would willingly accept it into their practice.^16–18 The more difficult and complicated the technology, the less the user engaged, demonstrating the power of design, useability and usefulness.^19–21

The socio-technical theory adds another dimension to this field of research, by acknowledging the essential interdependencies that exist between individuals, organisations, and technology. ²² Developed in the 1940s by psychologists at the Travistock Institute of Human Relations, this model explores the “de-humanising effect” of scientific breakthroughs, reveals powerful human factors that impact the quality, safety and value of new interventions, and demands a holistic review of technology implementation.^22–25 Socio-technological studies have found that stakeholders within healthcare systems often hold different perceptions depending on their expectations and objectives for the technology.^26–28 A healthcare professionals’ focus on improving patient outcome and clinical decision-making, may differ from the organisation’s vision of maximising financial performance or workload productivity.^28,29 The difficulty lies in finding an appropriate way to understand all stakeholder perceptions when implementing new technology like AI into healthcare. ³⁰

Research about perceptions of AI is emerging as organisations seek to understand workforce readiness, however at this stage these studies largely focus on medical professionals. ³¹ Jha et al. ³¹ developed a survey instrument to measure American physicians’ perceptions about the impact of health information systems on primary care delivery, and found that physicians were sceptical about its ability to perform better than humans. A qualitative study measuring psychiatrists perceptions of AI role replacement, ³² found that their views were divergent about the value and impact of AI, but that it would never replace the relational aspects of psychiatric care. Laï et al. ³³ used a qualitative approach to study physicians’ perceptions of AI, and found that they share concerns about the management of data, the development of knowledge, the upheaval of the doctor-patient relationship, and the disruption of the diagnosis and decision-making landscape. The Robot Use Self-efficacy in Healthcare (RUSH) study in Finland, ³⁴ focused on their healthcare workforce more broadly and developed a theoretical questionnaire to measure perceived self-efficacy in task-specific robot use. They found healthcare professionals were confident in their use of the technology and on average were very interested in its application. A survey of students’ was conducted ³⁵ to understand whether their perceptions of AI influenced their career intentions for radiology. Students perceived that AI would play an important role in their careers, and were less likely to pursue radiology due to the perception that AI would one day replace them. ³⁵ The introduction of AI into healthcare delivery requires an understanding of how the workforce perceives AI, so that formal processes and training can be implemented to support its management, use and application in healthcare.^36-40 A questionnaire that measures healthcare professionals’ perception of AI more broadly does not yet exist. Thus, the aim of this study was to draw upon the collective knowledge of experts in the fields of health and technology, to develop a questionnaire that measured healthcare professionals’ perceptions of AI.

The e-Delphi method

An e-Delphi method was chosen for this study to develop and validate a questionnaire that measures healthcare professionals’ perception of AI. Originally created in the 1950s by the Rand corporation, ⁴¹ the aim of the Delphi method is to gain consensus on group opinion. ⁴² A panel of carefully selected participants that demonstrate an interest or involvement in the field related to the research,^42,43 are invited to participate in several rounds of feedback or discussion to provide an impartial reflection of current knowledge or perception.^41,44

The main advantage of this method of study is that group diversity is replaced by a single representative opinion. ⁴⁵ The Delphi method has been used in healthcare to establish research priorities, ⁴⁶ develop competencies and frameworks,^47,48 and guide key components of intervention. ⁴⁹ Traditionally, the Delphi method uses a paper-based questionnaire to collect data from participants, however digital methods, called e-Delphi methods, are now being used.^50,51 The use of electronic and internet-based questionnaires allows for a faster response time, points of anonymity and a reduction of resource costs. This study used the e-Delphi method to develop and validate a questionnaire that explores healthcare professionals’ perceptions of AI.

Design

The study followed the traditional structure of the Delphi method, consisting of a series of structured rounds to facilitate discussion among experts and to reach consensus about the questionnaire. ⁴¹ Instead of using post-mail to correspond, the e-Delphi method used an online survey platform, virtual meeting rooms and e-mail to facilitate discussion, collect data and provide structure to the research. ⁵²

Delphi method consensus remains disputed in the literature.^19,22,31 Consensus for agreement in this study was based upon Diamond et al.’s ⁵³ systematic review, which proposed 75% as the median threshold to define consensus. At the outset of the study it was decided that group agreement greater that 75% on each question would be an acceptable level of consensus for the study.

Participants

The panel for this study were carefully selected participants who demonstrated an interest and/or involvement in AI from the fields of health or information technology (IT).^42,43,54 Recruitment was accomplished via email which invited the panel member to participate in the modified e-Delphi study and included study and consent information. A follow-up phone call provided the opportunity for further questions and clarification of the project. The participants were known to the researcher but remained anonymous to other panel members initially to encourage the expression of unbiased opinion, particularly in the first round.

Criteria for the choice of expert panel members were either: a) healthcare professionals who were registered with the Australian Health Practitioners Regulation Agency (AHPRA), who had clinical experience of more than 10 years in either an acute or primary health sector, and came from a broad range of disciplines, and with an interest in health technology; or professionals from the IT sector with greater than 10 years’ experience in technology development or project management related to healthcare; b) able to access an email account; c) be willing to participate. The diverse panel could provide an impartial reflection of current knowledge or perception in both the health and technology spheres. ⁴¹

Data collection

Data were collected from three rounds of the modified e-Delphi study between January and March of 2019 in the form of an online survey, an online group meeting and email communication. The study was conducted according to the national statement on Ethical Conduct in Human Research (2007) ⁵⁵ and approved by Southern Cross University Human Ethics Committee (HREC Register Number ECN-18-086).

Round 1: Identifying issues, structure and content

Round 1 was delivered in the form of an electronic survey and was sent to each participant via the Qualtrics platform. The Qualtrics survey consisted of 18 open-ended questions that were designed to identify the key issues associated with developing the questionnaire and establish its suggested structure and content (see Table 1). Preliminary questions for the questionnaire were informed by the validated Finnish-language questionnaire, Robot Use Self-Efficacy in Healthcare work (RUSH) ³³ for example: “I have been adequately trained to use AI technology that is specific to my role.” The panel could provide comments and suggest additional questions and topics for the questionnaire. Panel members had two weeks to respond, with individually completed surveys returned anonymously to the researcher.

OPEN IN VIEWER

Table 1.

Structure of Round 1 e-Delphi survey.

Topic	No. of questions
Key issues associated with questionnaire that explores healthcare professionals’ perceptions of AI	1
Review and comment of suggested demographic items	5
Review and comment of suggested perception items	10
Comments and suggestions of further items for discussion in Round 2	2

Panel characteristics

The panel included experts from a range of professions including four healthcare professionals from optometry, medicine, nursing and allied health; and four IT professionals from health, science, engineering and finance backgrounds. The majority of panel members were knowledgeable about AI but not all had substantial hands-on experience with AI applications. They had a diverse range of industry experience, four working as clinicians, three as technologists, two business owners, two academics, one program manager and three executive level professionals ³ or a combination of these. Panel members were located in three Australian states (Victoria, Queensland, New South Wales). All eight panel members participated in all three rounds. The response rate to the online survey, group discussion and email discussion was 100% for each round and response exhaustion was reached within these three rounds.

Round 1: Identifying issues, structure and content

Panel responses identified issues about general understanding of AI and established demographic variables that would be of interest to future studies (see Table 2). The panel suggested that AI was not well understood by healthcare professionals, that AI education was not yet established in healthcare and that a universal, user-friendly definition of AI was needed to precede the questionnaire. Because cohorts being surveyed, would be diverse, they suggested that this definition should also need to be supported by clinical examples of AI. The panel achieved consensus on demographic questions related to age, but they did not achieve consensus on the range of disciplines, gender and job description options, and suggested they needed to be inclusive of the diversity found within healthcare settings (see Table 3). These items were taken to Round 2 for further discussion. Nine draft items measuring perception of AI achieved more than 75% consensus from panel members, and covered a range of topics including perceptions of use of AI, financial and ethical impact, training, as well as broad perceptions of AI’s impact on healthcare (see Table 3). The panel suggested that AIs impact on role be given consideration in the questionnaire, which were taken to Round 2 for discussion in the small group meetings.

OPEN IN VIEWER

Table 2.

Issues raised by panel in Round 1.

Understanding of AI	• “If participants have little or no understanding of AI how valid are their responses to the perception of AI in their workplaces?”• “Perhaps you need to measure their understanding of AI at the beginning of the survey and then provide some more specific definitions and examples to gain more accurate information around use of AI in their workplaces.” • “The information about AI should be participant friendly - particularly for those participants with a limited understanding of AI.” • “Without being aware of the full range of available AI, I wonder if more specific examples would assist participants to identify the use of AI in their work. This also makes me think that at the beginning of the survey some inclusion of examples of things that are not AI but might be considered to be AI by participants might be useful.”
Demographics	Use: • “what will the differences between users and non-users be? Does technology use impact perception and will it be an important variable to include?”Discipline• “Not all disciplines within Australia have been represented as options” • “I am interested in the options provided - assuming that these are the professions that you are targeting I am wondering why pharmacy and paramedicine are not represented?Gender• “The issue of gender is becoming an increasingly contested and sensitive area. I recommend careful consideration for inclusion of this question as well as the options provided.” Job Description• “How would Healthcare assistants respond the previous question? Perhaps provide some more Allied Health specific options such as Discipline Team Leader, Allied Health Manager, Clinical Educator. How would an Informatician answer the previous question? I am interested in the way you have scaffolded the responses to this question - I imagine most respondents would be clinicians - would it be possible to have this as the first option followed by all other clinical options and then management options?”
Further Items	• “Have you considered questions about AI taking over part of role? This could be really interesting”

OPEN IN VIEWER

Table 3.

Consensus items generated from Round 1 and Round 2.

Round 1
Demographic item: Age
Item 1. The use of AI on my specialty could improve the delivery of direct patient care
Item 2. The use of AI in my specialty could improve clinical decision making
Item 3. The use of AI could improve population health outcomes
Item 4. The introduction of AI will reduce financial costs associated with my role
Item 5. I have been adequately trained to use AI that is specific to my role.
Item 6. AI may take over part of my role as a healthcare professional
Item 7. There is an ethical framework in place for the use of AI technology in my workplace
Item 8. Should AI technology make an error; full responsibility lies with the healthcare professional
Item 9. The introduction of AI will change my role as a healthcare professional in the future
Round 2.
Demographic item: gender options, AI use, Discipline options, job description options
Item 1. AI will change my role as a healthcare professional in the future
Item 2. Overall healthcare professionals are prepared for the introduction of AI technology

Round 2: Online group meeting

The small group meetings for Round 2 lasted one hour each and were held in three sessions with 2-3 participants in each. In Round 2 the panel achieved consensus on the remaining demographic questions and included use of AI, for example: “Based on your knowledge of the technology that you are currently using within your role, have you been using AI technology?” (see Table 3). Consensus was achieved (100%) for gender, AI use, health discipline and job description options, sufficiently comprehensive to include all healthcare professionals practising in Australia. Two further items were developed by the panel to measure perception of AI and consensus was achieved on all 11 items (see Table 3). These included questions regarding perceptions of impact on role, for example: “I believe that AI will change my role as a healthcare professional in the future” and questions regarding perceptions of professional preparedness, for example: “I believe that I have been adequately trained to use AI that is specific to my role”.

Round 3: Email communication

In a final e-Delphi round, a draft of the final questionnaire was distributed via email to the panel members for comment. No further amendments were put forward and 100% consensus was achieved.

Limitations

The selection of a small number of panel experts could be considered a limitation of this study, however it is equally as important in the Delphi method to ensure that there is not an over-representation of panel members. The lack of anonymity in the Round 2 small group meetings may have limited contribution or compelled panel members to conform to discussion. The face-to-face structure of the round may have compromised validity when panel members were faced with strong opinions and subsequently changed their view. To manage this, panel members were explicitly asked to consider opposing points of view, thereby eliciting further discussion and relieving socio-psychological pressure. The final Round 3 minimised bias by allowing individuals to provide further anonymous comment. The transparent nature of the study design, informed by Diamond et al.’s⁵³ quality indicators, is thought to have led to the full attendance of particpants, facilitating high quality feedback and contributing to the rigor of the study.