Applying the Delphi method to generate interventions to reduce unnecessary interruptions in clinical nursing

Abstract

The objective of this study was to describe a panel process of selecting interventions to reduce unnecessary interruptions in clinical nursing. Interruptions in nursing may jeopardise patient safety, disorganise planning, and lead to ineffective working procedures. The Delphi panel process may be useful to create consensus on interventions to reduce interruptions. Using the Delphi method, a structured group-based decision-making tool, a panel of 14 nurses agreed on interventions to reduce interruptions. The results showed that a consensus on reducing interruptions was reached in four focus areas: expectations, physical framework, work procedures, and medication room procedures. The panel process was chosen as a means to involve nursing staff to create consensus and ownership about interventions. Our conclusions are that the Delphi panel process was a feasible method of selecting multi-factorial interventions to reduce interruptions. The panel process contributed to dialogue, staff involvement, and competency development. Further prospective studies are needed to explore whether these multi-factorial interventions can reduce unnecessary interruptions and improve patient safety.

Keywords

Delphi method interruptions patient safety panel process

Introduction

It is said that nurses can be interrupted every sixth to ninth minute. An interruption means leaving a task, and creates a change in context or thoughts. Interruptions may lead to delay in working routines.^1,2 The consequences are wasted time, disordered planning, ineffective working procedures and disrupted reflective processes.^3,4 Additionally, there is a growing problem with errors caused by interruptions.^3–9 Thus, it may lead to reduced patient safety if nurses are not concerned about interruptions. There is a need for new knowledge to develop interventions to reduce unnecessary interruptions in clinical nursing.^2,9–11

Background

Within the last decade, research on interruptions and especially the association between interruptions and medication has expanded.^{3,4,8,9,12–14} Interruptions challenge nurses due to a dilemma between being accessible and staying focused.² Interruptions lead to a constant rearrangement of priorities and may induce a feeling among staff of being ineffective and under time pressure; this can cause reduced job satisfaction and produce stress-related symptoms.^5,15

Interruptions affect decision-making processes because of changed awareness and a shift of focus.¹⁶ They may lead to reduced precision and increased time spent on problem-solving.¹⁰ The consequences for nurses are feelings of incompetence, being under time pressure, and disrupted care relations, leading to a need for strategies to keep up with the pace of work.¹²

Depending on the context, nurses distinguish between necessary and unnecessary interruptions.^17,18 Necessary interruptions can be external or self-inflicted, and contribute to getting an overview of work procedures, sharing information, competence development or teamwork. Unnecessary interruptions negatively affect organisation of work, attention, relations with colleagues and patient safety, and may also impact negatively on patient–nurse interaction.^2,9 An interruption is different for the interrupted and for the sender respectively, and interruptions distress behaviour differently depending on the person, frequency, period, complexity and circumstances.^8,19

The primary sources of interruptions are health professionals, followed by the surrounding environment such as background noise and small talk.²⁰ An observational study showed that in 41% of interruptions, people never returned to or finished the original task, keeping in mind that it takes up to 20 minutes to restart the task.²¹ The short-term memory only contains five to nine units at a time; if new information appears, other information is tossed aside.²²

Studies describe different ways of reducing interruptions but they are often based on single interventions and mostly related to the administering of medication.^9,23 Few studies suggest multi-factorial interventions as a more effective way of reducing interruptions during medication procedures.^14,15,24 Relihan et al. found that by using no-talk signs, quiet zones and ‘please do not disturb’ vests, interruptions could be reduced during administering of medication.²⁴ Furthermore, certain educational and environmental interventions have been shown to reduce interruptions.^14,25 Studies have investigated the consequences of interruptions in clinical nursing, but there is inconsistency concerning the process of choosing interventions to reduce these interruptions.

The aim of this study was to describe a panel process for choosing interventions to reduce unnecessary interruptions in clinical nursing.

Methods

A pilot study was carried out in 2011 to identify the nature of interruptions in our department, and to investigate how nurses cope with interruptions in clinical practice.²⁶ The participating nurses found interruptions to be aggravating and stressful, but interruptions also contributed to gaining an overview, sharing knowledge, and to being part of a team. Furthermore, nurses often interrupted their own colleagues with doubts/questions. This pilot study was used to prepare the first-round panel meeting to choose interventions to reduce interruptions.^27,28

We selected the Delphi method (Delphi) to develop meaningful interventions to reduce unnecessary interruptions for nursing leaders and staff. In this paper we focus primarily on describing the panel process of this method. We will only briefly present the selected focus areas and derived interventions to prevent interruptions, since we have not yet investigated the impact of implementing these interventions.

Delphi is a structured group-based decision-making tool that affects ways of thinking through dialogue and prioritising.²⁹ It is a validated and widely accepted method to achieve convergence of opinions concerning everyday-world knowledge from different perspectives.^26,29 It has been used in various studies on programme planning, needs assessment, policy determination, resource utilisation, and development of nursing practice.^26,29,30 It can facilitate the development of alternatives, be used to explore or expose underlying assumptions as well as to correlate judgments on a topic spanning a wide range of disciplines.^26,29 We assumed that the Delphi was suitable for our purposes.

The process encompassed self-administered scoring forms and panellist meetings between August and October 2011 at a surgical department with a nursing staff of about 170 nurses. The process consisted of rounds of feedback from the panellists using both anonymous surveys and round table meetings.

The number of subjects in a Delphi panel is recommended to be between 10 and 50 to constitute a representative and homogeneous pooling of judgments and information processing capabilities.²⁹ The panellists included the head nurse, five charge nurses, six registered nurses – all interviewed in the pilot study – an education and development nurse, and a senior nursing researcher. We chose these 14 panellists strategically to represent different job positions and competencies, up-to-date knowledge on the topic, and experiences in clinical nursing.²⁹ To avoid impact of pre-understandings, the panellists were unaware of the results of the pilot study before starting the panel process.

Data collection and analysis

The process was based on anonymity, repetition, validation and feedback to ensure broadness, convergence, creativity, effectiveness and consensus.²⁹ The repetitions related to the meetings were expected to make the panellists more oriented towards problem-solving, to encourage them to offer their opinions more insightfully and to minimise noise. The process consisted of interactions between the panellists where prioritisation, dialogue and summaries of intervention were emphasised. The content of the panel meetings was listening and responding to written statements, and exchange of opinions, positions and attitudes. Each panel meeting round consisted of a preparation, a meeting and a compilation phase.²⁶ The rounds were held at two-week intervals during the daytime to involve as many panellists as possible.

Data collection and analyses were two simultaneous processes. Data were obtained with open-ended questions among the panellists as suggested in the literature. This was to solicit the opinions of the panellists and achieve the desired level of consensus as well as any changes in judgments.²⁹ Four different forms were used for different purposes: preparation, scoring, conclusion and decision. The forms were sent to panellists by email. There were two ways of reaching consensus: a) agreement marked with a notch (on the scoring form) and b) prioritisation based on a five-point Likert Scale (on the preparation form).³¹ Statements rated three or higher were included in the further process. To avoid subtle pressure among the panellists, the facilitators (LB and SL) lead the panel process at meetings and discussed process progression concurrently. The meetings were all tape-recorded, but not fully transcribed. The tape-recordings were used between meetings to ensure that the right statements were included in the preparation forms. Furthermore, we extracted paragraphs with statements to support benefits of the panel process. Table 1 provides an overview of the process.

Table 1.

Distribution of roles of the panellists in the four Delphi panel round meetings.

		Facilitator		Panellists
Preparations	Meetings
Before 1st meeting		Preparation form 1	Scoring form 1	Filling in preparation form 1
	1st meeting			Two minute presentation	Filling in scoring form 1
Before 2nd meeting		Counting scoring form 1	Scoring form 2	Filling in preparation form 2
Before 2nd meeting		Preparation form 2	Scoring form 2	Filling in preparation form 2
	2nd meeting			Discussion scoring form 2	Filling in scoring form 2
Before 3rd meeting		Counting scoring form 2	Scoring form 3	Filling in preparation form 3
Before 3rd meeting		Preparation form 3	Scoring form 3	Filling in preparation form 3
	3rd meeting			Discussion scoring form 3	Filling in scoring form 3
Before 4th meeting		Counting scoring form 3	Conclusion forms
	4th meeting	Presenting pilot study		Discussion of conclusion form	Decision form
			Contract signing

Ethical considerations

The department management approved the study. Participants were informed about the Delphi process in a briefing session concerning: the purpose of the project, the content, information about their role, and their right to withdraw from the project. Before the briefing session, participants received a short description about the project by email. The panellists gave written and oral informed consent during the briefing session, and complied with rules on confidentiality. An open and trustful atmosphere was pursued to secure loyalty to personal statements and content of the meetings. Researchers and participants were acquainted with each other, which might have impacted on their voluntariness. However, the request to reduce unnecessary interruptions came from the clinical nurses. In this way the project was a joined decision. All tape-recordings were deleted after completion of the study. Due to Danish law the study was exempt from approval by the local Human Research Ethics Committee and the Danish Data Protection Agency.

Rounds in the panel process

To develop consensus concerning interventions, we took 60 days and used four panel meeting rounds.

Before the first meeting: Preparation form 1 included 10 statements based on findings from the pilot study; this was emailed to the panellists, who scored the statements. The 10 statements are shown in Table 2.

The facilitator (LB) collected the preparation forms and composed scoring form 1. In addition, the panellists prepared a two-minute presentation on the open-ended question: how do interruptions impact on the working environment?^26–28

Table 2.

Ten statements from a pilot study on interruptions in nursing practice. The statements appeared in interviews and observations.

Interruptions affect patients’ safety

Interruptions affect overview

Interruptions impact on the psychological working environment

Interruptions affect nurses’ competence development

Interruptions affect professional competencies

Interruptions affect quality of nursing practice

Interruptions from own colleagues are problematic

Interruptions at the office are problematic

Interruptions in the hospital corridor are problematic

Interruptions in the medication room are problematic

First-round panel meeting: This meeting featured only reflection and no dialogue. The first round is typically qualitative and designed to ascertain the broadness of the topic for the panellists.^26,27 Initially, each participant gave a two-minute presentation. The other panellists were not allowed to comment or interrupt. Afterwards scoring form 1 was handed out and panellists were given 1.5 minutes to mark their agreement of the statements. Then each panellist’s scoring form was circulated among the panel for one minute each. Each panellist responded to the scorings of the other panellists by marking their level of agreement.

Before the second meeting: The facilitator listened to the tape-recording from the first meeting to extract the essence of panellist presentations, and counted the completed scoring forms, on the basis of which preparation form 2 was composed. Preparation form 2 was emailed to the panellists, who scored the statements. The facilitator counted the scorings and composed scoring form 2.

Second-round panel meeting: Scoring form 2 was handed out to the panellists to form the basis of a 30-minute plenum discussion. Afterwards each panellist wrote the two most important conclusions from the discussion. The panellists were given 1.5 minutes to mark agreement of the statements on scoring form 2. Then these scoring forms were circulated among the panellists for one minute each. Each panellist responded to the scorings of the other panellists by marking their level of agreement.

Before the third meeting: The facilitator listened to the tape-recording from the second meeting to extract the essence of the discussions, and counted the completed scoring forms, on the basis of which preparation form 3 was composed. Preparation form 3 was emailed to the panellists, who scored the statements. The facilitator counted these scorings and composed scoring form 3.

Third-round panel meeting: Scoring form 3 was handed out to the panellists for a 30-minute plenum discussion. Afterwards each panellist wrote suggestions and ideas for possible and relevant interventions to reduce unnecessary interruptions. The panellists were given 1.5 minutes to mark agreement of the statements on scoring form 3. Then these scoring forms were circulated among the panellists for one minute each. Each panellist responded to the scorings of the other panellists by marking their level of agreement.

Before the fourth meeting: The facilitator listened to the tape-recording from the third meeting, gathering suggestions and ideas for interventions, and counted the completed scoring forms 3. A conclusion form was composed and emailed to the panellists.

Fourth-round panel meeting: The facilitator presented the findings from the pilot study and the conclusion form was discussed. Afterwards interventions were determined and a decision form was composed. The panel process ended with all panellists signing a contract and by evaluating the panel process.

Results

Benefits of the panel process

The benefits of involving nursing staff in the panel process were mainly positive both personally and professionally:

It was a positive process with many eye-openers and extra benefits. (Head nurse of the department)

It has given more insight into other peoples’ working procedures. (Department nurse)

It has contributed to my own competence development. (RN)

As intended, the panel process created time, space and room for reflection and for presenting considerations about interruptions without being disturbed. This was pointed out as important for further reflection. Even though the panellists had different in-depth knowledge of interruptions, and although the process was time consuming, sharing these experiences qualified the results and created new angles to the topic. It created awareness of and legitimised the problem of interruptions. The process has been a revelation for managing complex problems:

The big blurred subject has become more tangible. (Department nurse)

I have become more curious; it is exciting, practice-related and applicable. (Department nurse)

Furthermore, the process has created alternative ways of thinking:

It was another way to meet, not only talking knowledgeably about things. (Department nurse)

There is recognisability related to my participation in an interview, where it then was more emotional and experience-based, it is now more concrete. (RN)

Interventions to reduce interruptions

Consensus was reached on four focus areas for interruptions in clinical nursing. Interventions were related to expectations, physical framework, work procedures, and medication room procedures (Table 3).

Table 3.

Focus areas and interventions to reduce interruptions in clinical nursing developed through a Delphi panel process.

Focus areas	Interventions
Expectations	Introducing time out
Expectations	Multi-disciplinary dialogue meetings
Physical framework	Office guide to make sure agreements were kept
Physical framework	Nursing coordinator to ‘filter’ all contacts with nursing staff
Work procedures	Use of present lamp in patient rooms
	Use of ‘nurse finder’ (a pocket pager)
	Focus on the use and structure of patient boards
	Remember to close the door when talking to patients and relatives
Medication room	Focus on the medication room as a quiet and safe place
Medication room	The use of ‘please do not disturb me’ vests

Time out was chosen to legitimise the right to put one’s foot down in situations when the nursing staff did not wish to be interrupted. Simultaneously, time out will be introduced in each shift to minimise interruptions related to progression of work. Dialogue meetings with other health professionals will be held to optimise the right time for e.g. the physiotherapist to ask about a patient’s condition or when to mobilise a patient.

Physical framework encompassed introducing an office guideline promoting a quiet atmosphere in the nursing office, reducing traffic jams in the hallway, optimising administrative working procedures, and reducing small talk at the office. The nursing coordinator will be used as a gatekeeper for interruptions e.g. by prioritising external calls, and making arrangements with doctors, physiotherapists or relatives.

For work procedures, we reached consensus about the use of lamps to indicate when nurses are caring for patients in patient rooms. Additionally, by operating a pocket pager in the team the nurses can triage patients’ calls as the pager selectively chooses the appropriate team members. The pocket pager also contributes to a quiet working environment (reducing background noise). The patient board will be more structured and used optimally, and encompass agreed-upon information. Moreover, we will focus on closing doors and using the occupied sign when having conversations with patients and relatives.

Finally, to minimise interruptions in the medication room, attention will be on displaying ‘do not disturb me during administration of medication’, and ‘do not disturb me’ vests will be implemented.

Discussion

In the Delphi panel process we aimed at selecting interventions to reduce unnecessary interruptions in nursing practice. We demonstrated that the process was a feasible way to involve nursing staff through dialogue and to achieve consensus about selected interventions. The outcome of the panel process was agreement on four focus areas to reduce interruptions: expectations, physical framework, work procedures, and medication room procedures.

We chose to highlight the panel process in Delphi. The process is described as multi-stage where each stage builds on the results of the previous stages.²⁸ This forecasting activity is defined to be iterative to achieve consensus as well as to determine, predict and explore panellists’ attitudes, requirements and main concerns.^26,28 Delphi has been used in different fields including nursing and healthcare.²⁶ However, Delphi is often not used in the same way, and modifications to the original version are extensive.²⁸

To avoid subject bias, anonymity offers an opportunity for each panellist to introduce and respond to ideas without bias based on the identities of others. Anonymity may thereby facilitate an open-minded and truthful approach. In this way each opinion carries the same power and is considered equally important when reaching consensus.²⁸ The panel process also includes plenum discussions. Most of the panellists knew each other and to prevent panellists feeling pressured, and to divide speaking time equally, the facilitators led the panel meetings, discussed the process after each meeting, and listened to the tape-recordings of the meetings. Hence, our process reflected quasi-anonymity rather than anonymity.^28,32

Earlier studies have focused on single interventions to increase patient safety by reducing interruptions related to medication.⁹ Multi-factorial interventions could result in a higher level of compliance since different approaches appeal to people in diverse ways. According to the Theory of Planned Behaviour, different background factors affect the ability to change habits beside reasoned modes of operation; these factors are impulsivity and ability to act on intentions.³³ Different interventions could motivate different individuals to change habits, as nurses adapt to changes in different ways.³⁴ However, individual learning and working styles have a considerable impact on the success of implementing the selected interventions. Willingness to adapt is personally and socially processed.³³ Since interruptions are related to habits, we believe the chosen interventions to reduce interruptions should involve changing habits in a social context. We assume that multi-factorial interventions have more impact on successful change of habits, because interruptions affect behaviour depending on the person, frequency, duration, content, complexity and context.⁸ While nurses have different attitudes towards interruptions it is important to involve all parties for whom Delphi was appropriate.²⁹ It is, however, unknown whether the chosen multi-factorial interventions can reduce interruptions; further prospective studies are needed to confirm this.

Obtaining consensus in our panel process does not necessarily mean that the right intervention to reduce interruptions was identified, since the result from the process cannot replace rigorous scientific work.²⁶ In this thinking Delphi can lead us to place larger confidence in our results than might be reasonable.²⁸ Yet, the role of the facilitator has been described as to ensure that all participants are able to express their understandings by controlling the dialogue, and to prevent a particular interest or predetermined opinion from dominating.³⁰ Hence, we aimed at not forcing through consensus, and allowed the participants to debate subjects letting opportunities arise for respondents to elaborate on their perceptions.

Jones and Hunter describe two levels of making agreements.³⁰ Firstly, the level to which each participant agrees with the question rated on a numerical or categorical scale. Secondly, the level to which respondents agreed with each other. During the panel process different competencies and interpretations about interruptions were clarified among the panellists (i.e. how listening to reflections of others affected own attitude and position among panellists). We thus used the process in both levels of agreement with consensus measurement and consensus development.³⁰ As a spin-off in the panel process, we made it possible to become more familiar with each other and learn how to work together.²⁹ Additionally, the panel process may be a useful revelation, which may be transferred to similar processes where teamwork is important.

Strengths and weaknesses

This study aimed at following the Delphi approach.²⁹ The 14 panellists represented leaders and nurses from all wards in the department with different competencies, interests and insights into the field of interruptions. Literature suggests that the panellists must at least be practising in the field under consideration.^26,30 Nurses being interviewed in the pilot study were involved; this may likely have impacted on their reflections about interruptions. However, there is always a risk of bias in the selection of participants, which may have affected the chosen intervention to reduce interruptions.³⁰ Nevertheless, having knowledge of and experience with interruptions does not automatically mean being an expert on the topic. Furthermore, the panellists can often not be both a representative sample of the nursing staff and experts – these characteristics may be mutually exclusive.²⁸ We assumed that the number of panellists was representative of different positions and attitudes that needed to be challenged. Regarding implementation, more panellists might have enhanced the chance of success. The panel process is based upon the hypothesis that more participants are less expected to reach a wrong conclusion than a single individual.²⁶ However, the evaluation showed agreement among the participants about their opinions being respected, the process being fair, and no one altered their responses or opinions as a consequence of being pressured. Furthermore, the use of successive rounds and the progression in the preparation forms may have helped us to enhance validity. Still, it is uncertain if a repetition of the panel process with other panellists would result in exactly the same interventions to reduce interruptions.^26,28

To reach consensus about statements in the panel we used a cut-off at 60% on the four-point Likert Scale. The Likert scale is frequently used in Delphi studies.³⁵ However, the methodology is criticised for being sub-optimal, since a transformation of nominal to numeric measurement can result in the panellists failing to recognise their contributions.²⁷ Different cut-off points and different scales are used in the literature. Some have made the criterion for consensus at 80% of the panellists’ scoring within two categories on a seven-point scale; others recommend that at least 70% of the panellists rate three or more on a four-point Likert Scale with a median at 3.25 or higher.^27,29,35 Nevertheless, the use of different scales and different cut-off points makes it difficult to compare our results with others.^26,36

The facilitators chose only to present the statements from the pilot study, not the findings of the interviews and observations, to avoid the impact of panellists’ pre-understanding of interruptions and to make the process as objective as possible. On the other hand, knowledge of the pilot study could have motivated and qualified the interventions during the panel process.

Some panellists were present at all meetings and others were not. This might have influenced the insight and continuity of the process, misinterpretations or loss of nuances and ownership.^26,37 The panel process is highly dependent upon the participants’ obligation in the process. It is central that those who have agreed to contribute sustain involvement until the process is completed.²⁶ To minimise cancellations, meetings were held after day shifts, and written information was delivered accompanying the first round. The panellists’ understanding of the purpose of the process was highlighted to encourage ownership. Still, although all agreed on the final decisions, the consequences of cancellations may be lack of ownership and thus commitment to prioritise interventions to reduce unnecessary interruptions.

The panel process was time-consuming, as we emphasised the importance of making room for reflection. This resulted in the process being educational both personally and professionally. Furthermore, to obtain an adequate level of consensus, four rounds were necessary as described in the classic original Delphi approach.^28,30 Four rounds have been criticised for creating sample fatigue.²⁶ We did not experience sample fatigue, as the process demanded reflection on each participant’s own response as well as the response from the other panellists.

Conclusion

In this structured Delphi panel process we created consensus about interventions to reduce interruptions in nursing practice. We found the process a feasible and transparent method to select multi-factorial interventions. The panel process has led to inspiration, competence development and enthusiasm among the panellists. It has created awareness of individual and group-based positions and behaviours, and has strengthened team spirit. The selected interventions were related to four areas: expectations, physical framework, work procedures, and medication room procedures. Further prospective studies are needed to explore whether these multi-factorial interventions can reduce unnecessary interruptions and improve patient safety.

Footnotes

Acknowledgements

We wish to thank the nurses from Department of Cardiothoracic and Vascular Surgery at Aarhus University Hospital for participating in the Delphi panel and for helping us in identifying and prioritising interventions to reduce interruptions in clinical nursing.

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Conflict of interest

The authors declare that there is no conflict of interest.

References

Biron

Loiselle

Lavoie-Tremblay

. Work interruptions and their contribution to medication administration errors: an evidence review. Worldviews Evid Based Nurs 2009; 6(2): 70–86.

Sorensen

Brahe

. Interruptions in clinical nursing practice. J Clin Nurs 2014; 23(9/10): 1274–1282.

Westbrook

Woods

Rob

. Association of interruptions with an increased risk and severity of medication administration errors. Arch Intern Med 2010; 170(8): 683–690.

Cloete

. Reducing medication errors in nursing practice. Nurs Stand 2015; 29(20): 50–59.

Tucker

Spear

. Operational failures and interruptions in hospital nursing. Health Serv Res 2006; 41(3 Pt 1): 643–662.

Brixey

Robinson

Turley

. The roles of MDs and RNs as initiators and recipients of interruptions in workflow. Int J Med Inform 2010; 79(6): e109–e115.

Kreckler

Catchpole

Bottomley

. Interruptions during drug rounds: an observational study. Br J Nurs 2008; 17(21): 1326–1330.

Speier

Valacich

Vessey

. The influence of task interruption on individual decision making: an information overload perspective. Decision Sci 1999; 30(2): 337–360.

Verweij

Smeulers

Maaskant

. Quiet please! Drug round tabards: are they effective and accepted? A mixed method study. J Nurs Scholarsh 2014; 46(5): 340–348.

10.

Beyea

. Noise: a distraction, interruption, and safety hazard. AORN J 2007; 86(2): 281–285.

11.

Liebowitz

. Interruption management: a review and implications for IT professionals. IT Professional 2010; 13(2): 44–48.

12.

Bowers

Lauring

Jacobson

. How nurses manage time and work in long-term care. J Adv Nurs 2001; 33(4): 484–491.

13.

Tomietto

Sartor

Mazzocoli

. Paradoxical effects of a hospital-based, multi-intervention programme aimed at reducing medication round interruptions. J Nurs Manag 2012; 20(3): 335–343.

14.

Yoder

Schadewald

. The effect of a safe zone on nurse distractions, interruptions, and medication administration errors. Western J Nurs Res 2012; 34(8): 1068–1069.

15.

King’s College, National Nursing Research Unit. Interruption to nurses during medication administration: are there implications for the quality of patient care? Policy + 2010; 22: 1–2.

16.

Kebinger J. Current research in workplace interruption management 2005. Paper Comp171.

17.

Hudson

Fogarty

Atkeson

. Predicting human interruptibility with sensors: a Wizard of Oz feasibility study. CHI 2003, pp. 257–264.

18.

Mark

Gonzales

Harris

. No task left behind? Examining the nature of fragmented work. CHI 2005, pp. 321–330.

19.

Spira

Feintuch

. The cost of not paying attention: how interruptions impact knowledge worker productivity, New York: Basex, 2005.

20.

Jett

George

. Work interrupted: a closer look at the role of interruptions in organizational life. Acad Manage Rev 2003; 28(3): 494–507.

21.

Wilson G. Emails ‘hurt IQ more than pot’. CNN, http://edition.cnn.com/2005/WORLD/europe/04/22/text.iq/index.html?iref=allsearch (2005, accessed 27 October 2013).

22.

Atkinson

Shiffring

Human memory: a proposed system and its control processes. In: Spence

Spence

(eds). The psychology of learning and motivation, New York: Academic Press, 1968, pp. 89–195.

23.

Pape

Guerra

Muzquiz

. Innovative approaches to reducing nurses’ distractions during medication administration. J Contin Educ Nurs 2005; 36(3): 108–116. quiz 41–42.

24.

Relihan

O’Brien

O’Hara

. The impact of a set of interventions to reduce interruptions and distractions to nurses during medication administration. Qual Saf Health Care 2010; 19(5): e52.

25.

McGillis Hall

Pedersen

Fairley

. Losing the moment: understanding interruptions to nurses’ work. J Nurs Adm 2010; 40(4): 169–176.

26.

Hasson

Keeney

McKenna

. Research guidelines for the Delphi survey technique. J Adv Nurs 2000; 32(4): 1008–1015.

27.

Clibbens

Walters

Baird

. Delphi research: issues raised by a pilot study. Nurse Res 2012; 19(2): 37–44.

28.

Keeney

Hasson

McKenna

. A critical review of the Delphi technique as a research methodology for nursing. Int J Nurs Stud 2001; 38(2): 195–200.

29.

Hsu

C-C

Sandford

. The Delphi technique: making sense of consensus. Pract Assess, Res Eval 2007; 12(10): 1–8.

30.

Jones

Hunter

. Consensus methods for medical and health services research. BMJ 1995; 311(7001): 376–380.

31.

Norman

. Likert scales, levels of measurement and the ‘laws’ of statistics. Adv Health Sci Educ Theory Pract 2010; 15(5): 625–632.

32.

McKenna

. The Delphi technique: a worthwhile research approach for nursing? J Adv Nurs 1994; 19(6): 1221–1225.

33.

Ajzen

From intentions to actions: a theory of planned behavior. In: Kuhl

Beckmann

(eds). Action-control: from cognition to behavior, Heidelberg: Springer, 1985, pp. 11–39.

34.

Dunn

. Understanding the Dunn and Dunn learning styles model and the need for individual diagnosis and prescription. J Read Writ Learn Disabil Int 1990; 6(3): 223–247.

35.

Beattie

Mackway-Jones

. A Delphi study to identify performance indicators for emergency medicine. Emerg Med J 2004; 21(1): 47–50.

36.

Jeffs

Law

Straus

. Defining quality outcomes for complex-care patients transitioning across the continuum using a structured panel process. BMJ Qual Saf 2013; 22(12): 1014–1024.

37.

Reavy

Tavernier

. Nurses reclaiming ownership of their practice: implementation of an evidence-based practice model and process. J Contin Educ Nurs 2008; 39(4): 166–172.