Implementing an Online Instrument to Measure Nurse Practitioner Workload: A Feasibility Study

Abstract

Introduction/Objectives:

Nurse practitioners (NPs) improve access to care in community-based primary care. Determining an appropriate workload for NPs is complex. The number of patients seen by NPs represents an important consideration. We sought to determine the feasibility, acceptability and appropriateness of implementing the online NP workload measurement index (NP-WI).

Methods:

Feasibility study supported by the Theoretical Framework of Acceptability, conducted across 3 health regions in Québec, Canada. Data were collected from January to July 2024 using the online NP-WI (n = 66), 8-item acceptability questionnaire (n = 47), weekly implementation team meetings with NPs and decision-makers (n = 11), field notes and interviews (n = 13). Data analysis completed using descriptive statistics and content analysis, with data integration using joint displays.

Results:

NPs indicated that the NP-WI was easy to use. Acceptability scores were positively rated. Daily data entry took 5 to 7 min to complete. NPs deemed a 4-week collection period sufficient to capture a representative workload sample. The NP-WI captured patient, provider and organizational characteristics and the number of patients seen by NPs.

Conclusions:

NP-WI implementation was feasible. The instrument can support healthcare workforce planning with more adequate estimations of NP workload in community-based primary care, and provide greater equity in resource allocation and distribution of NP workload.

Keywords

community-based primary care feasibility study measurement mixed methods nurse practitioners workload

Introduction

Access to primary healthcare services is a global challenge.¹ Nurse practitioners (NPs) are Master’s-prepared nurses with advanced clinical competencies.^2,3 NPs autonomously diagnose and treat health conditions in a wide range of community-based practice settings including primary care, long-term care, and home care.^2,3 NPs improve access to care in these settings.⁴ However, the number of patients seen by NPs is an important consideration.⁵ Wide variability in the number of patients seen by NPs has been noted internationally across primary care-based settings with average numbers ranging from 9 to 15 patients per day.⁵

Workload refers to the amount of time needed to complete an activity,⁶ and caseload is the number of patients under the care of a full-time provider.⁵ Determining an appropriate workload for NPs is complex because of the large number of factors to consider.⁷ Excessive workload and difficulties meeting patients’ needs cause distress for care providers and can increase their intent to leave.^8,9 Significant challenges have been identified to capture NP workload because patients are often rostered with physicians rather than NPs.⁵ In many jurisdictions, including Canada, most administrative databases do not capture NP activities.^10
-12 In other jurisdictions, such as the United States, some NP care (eg, patients under Medicare) is reimbursed at a reduced rate.¹³ Even though NPs in the United States are required to bill for services using their own unique identifier, such reimbursement practices can encourage organizations to bill under physicians, making NP practice invisible.^12,13 Direct observations can capture what NPs do.^14,15 Time and motion studies (n = 945 h 25 min) using direct observations were conducted with NPs in 2 provinces in Canada (ie, Québec and Ontario). These studies found statistically significant differences in the time spent by NPs in activities (ie, workload) after taking into account patient (eg, gender, health condition), NP (eg, gender), organizational (eg, practice setting), and system (eg, scope of practice) characteristics.^14,15 Direct observations represent the gold standard to capture NP activities.¹⁶ However, they are resource intensive undertakings that are difficult to complete when large numbers of clinicians are involved or if clinicians are located in several regions. In 2024, 1543 NPs were registered in Québec, making it difficult to capture workload across the province.¹⁷

A recent study conducted in Québec, Canada, by Landry et al^18
-20 aimed to describe the information needs of stakeholders (ie, NPs, physicians, other care providers, decision-makers, patients) for a workload measurement instrument and determine areas of consensus on the elements to include in the tool. Large areas of consensus were found around the types of appointments (eg, face-to-face), number of patients seen per day, and NP years of experience. However, no consensus was reached among participants for the time needed to enter data into the tool, and the number of days needed to accurately represent NP workload and caseload (ie, 2-, 4-weeks). Using these data, a static model was developed as a mock-up of the measure (ie, Nurse Practitioner Workload Index [NP-WI]).

Measuring NP workload is important. Umbrella reviews of indicators sensitive to NP care in community-based primary care settings⁴ and the global perspective of advanced practice nursing²¹ identified critical gaps in research examining NP workload. Weiner et al²² and Wang et al²³ argue that it is crucial to examine feasibility, acceptability and appropriateness of implementation, particularly with new technology. These authors define feasibility as the perception that an innovation can be used successfully. Acceptability represents perceptions that an innovation is agreeable or satisfactory. Appropriateness examines the perceived compatibility of an innovation with the setting or its ability to address a problem. An online workload measurement instrument to collect NP workload data is needed to support workforce planning in Québec and in other jurisdictions internationally.

Objectives

The overall aim is to determine the feasibility, acceptability and appropriateness of implementing an online workload measure for NPs. More specifically, aim (1) pilot-test an online version of the NP-WI to determine feasibility and appropriateness; and aim (2) explore stakeholder views of the acceptability of using the online tool.

Conceptual Framework

The Theoretical Framework of Acceptability (TFA)²⁴ supported the study. The TFA includes 7 components: (1) affective attitude describes how participants feel about an intervention; (2) burden represents the perceived effort to participate in the intervention; (3) ethicality embodies the fit between an intervention and participants’ values; (4) perceived effectiveness represents participants views that the intervention is likely to achieve its goal; (5) intervention coherence addresses participants’ understanding of the intervention; (6) self-efficacy examines participants’ perceived confidence in their ability to perform required behaviours; and (7) opportunity costs incorporate perceptions of benefits (profits, values) that must be given up by participants to engage in the intervention.²⁴

Methods

Design

A feasibility study was undertaken to implement the online NP-WI.^25,26 A participatory research approach guided the process and the implementation plan was adapted with participants.²⁷ Perspectives of different stakeholders (eg, NPs, patients, decision-makers, physicians) were examined to support long-term sustainability of the implementation.^24,28 Québec’s healthcare system is primarily publicly-funded and most NPs work as salaried employees of the government. The project was structured around elements known to support successful short team interventions identified following a systematic review.²⁹ They include (1) volunteer participation, (2) regular weekly meetings for the first 3 months of the project to quickly identify and address challenges, (3) discussions with participants to share experiences, (4) determination of implementation priorities, and (5) adaptation of scheduled meetings at participants’ request after the first 3 months to meet their needs (minimum once per month). These elements were tested empirically in 2 implementation studies with NPs and other stakeholders in home care and long-term care in Québec, Canada.^10,30,31

Instruments

The online NP-WI was developed using activities identified in time and motion studies with NPs in community-based primary care in Québec and Ontario, a qualitative descriptive study to outline information needs of stakeholders and a consensus building exercise using the TRIAGE Method.^14,15,18,32 The tool, available in English and in French, was launched in January 2024 to establish proof of concept. The instrument includes an NP profile page that is completed once when the NP starts to use the tool, and includes the NP’s years of experience, number of hours and days worked per week. Organizational characteristics include the main and secondary practice settings, presence of an electronic health record (EHR), clerical and nursing supports, presence of other NPs, team size, and level of collaboration in the team. Daily, NPs select the appropriate date and add information to describe the types of appointments and patient characteristics. Time spent in non-clinical role dimensions and NP perceptions of workload (1-very unsatisfied to 4-very satisfied) are indicated daily. Reports for NPs and decision-makers are generated using these data. Decision-makers receive aggregated data for all NPs in their service who enter data in the tool.

The Theoretical Framework of Acceptability (TFA) Questionnaire-Providers³³ consists of 8 items to assess the domains of acceptability of healthcare innovations described above. The measure takes 2 min to complete. Responses range from 1 to 5 with anchors for each item. Median scores are reported by item. Lower scores for burden and opportunity costs indicate increased acceptability.

Adequacy of Appointment Time Questionnaire-Patients: This 4-item questionnaire takes less than 2 min to complete online or a paper copy. Patients are asked their gender, age (above or below 65 years of age), length of appointment with NP adequate to meet their needs (yes/no), satisfaction with appointment (yes/no).

A semi-structured interview guide explored themes included in the conceptual framework (eg, burden) and perceptions of usability (eg, menus, navigation, content, colours).³⁴

Data Collection

We collected data from January to July 2024. Our target sample size of a minimum of 30 participants for continuous outcomes and 50 participants for binary outcomes was sufficient to capture a range of perspectives and experiences with the NP-WI and consistent with recommendations by Totton et al³⁵ and Kunselman³⁶ for sample sizes in the context of feasibility studies. The implementation team (n = 11) included NPs and decision-makers from the 3 health regions. The team met weekly. They were actively engaged by the research team that included a patient partner (AD) to share experiences and trouble shoot issues (eg, connectivity). The implementation team were asked to share concerns during regular team meetings and via email. Field notes recording meetings, issues, and impressions were circulated weekly to the implementation team to ensure accuracy. The implementation team provided feedback on the NP-WI and supporting material. NPs in each region completed data entry for the online tool. NPs and decision-makers were asked to complete the TFA questionnaire. Patients who were seen by NPs were asked to complete the 4-item questionnaire. Semi-structured interviews were completed with participants to identify different perspectives. Purposive sampling was used to identify participants with different characteristics (professional group, experience, gender, location, favour implementation/not).³⁷ For aim 1, in addition to the TFA questionnaire, we measured the number of complete and incomplete entries, and attrition rates to assess feasibility.³⁸ Space to write comments was provided in the NP-WI. For aim 2, we explored perceptions of burden when completing the tool (eg, time to enter data, task complexity, technical issues, 2- or 4-week data collection periods), and adequacy of the content of the online training module.³⁹

Data Analysis

For aim 1, descriptive statistics were generated for the TFA questionnaire and NP-WI outputs.

Patient characteristics include patients in vulnerable situations (yes/no), patient’s age <80; patient’s age <70; several people in the exam room (eg, patients needing a translator, family members present for the health visit), number of issues addressed (1 to ≥4), and patients without specific health needs. NP characteristics include NP experience (years), NP gender (male/female), NP perception of workload (1-very unsatisfying to 4-very satisfying), NP full-time work status (yes/no). Full-time status was determined by examining the number of hours worked per week and the number of days worked per week in the NP-WI. The NP level of experience was calculated by subtracting the total duration of leaves for illness (months), maternity/paternity or other work stoppages (months) from the NP years of experience. The values are categorized by the NP-WI as novice (<1 year), beginner (1 to <2 years), competent (2 to <5 years), and expert (≥5 years). Organizational characteristics include the presence of an EHR (yes/no), clerical support (yes/no), nursing support (yes/no), other NPs in the practice (yes/no), team size (small, medium, large), perceived collaboration with healthcare team (1-very dissatisfied to 4-very satisfied), location (urban/non-urban). No additional analyses (eg, inferential statistics) were planned. We did not aim to test specific hypotheses or identify causal mechanisms in this feasibility study.^36,40 Several authors have advised against this practice in the context of feasibility studies as small sample sizes influence the determination of effect sizes and statistical significance (eg, Arain et al,⁴¹ Lancaster,²⁵ Thabane et al,⁴² Teresi et al²⁶). We did not impute values for missing data but monitored non-responses as part of the assessment of feasibility. Pearson et al⁴³ argue that to examine an implementation process in the context of a feasibility study, proportions and types of providers are a more adequate measure to answer the research question.

Content analysis was used to analyze the qualitative data.⁴⁴ We remained sensitive to NP expressions of power imbalances or concerns about how the data from the NP-WI would be used by decisions-makers as a potential gender difference.⁴⁵ Interviews were transcribed verbatim. A deductive approach was used to code themes consistent with definitions in the framework and an inductive approach identified emerging themes in the data.⁴⁶ Each data source was analyzed separately and integrated at the analysis phase using joint displays.^47,48 Triangulation of data sources enhanced our understanding of this complex phenomenon.⁴⁹ Results are reported using the Standards for QUality Improvement Reporting Excellence (SQUIRE) 2.0 guidelines (Supplemental Material).⁵⁰

Results

Participants were located across the 3 health regions. Participants were primarily women (84%), with a mean age of 38.42 years (standard deviation [SD]: 7.26, median: 38.00, range: 24.00-58.00). On average, they were licensed in their role for 10.58 years (SD: 8.99, median: 8.50, range: 0.67-32.00). Almost all participants worked full-time, and had worked with their team an average of 4.12 years (SD: 3.72, median: 3.00, range: 0.33-13.00) (See Table 1).

Table 1.

Characteristics of Participants Who Completed the TFA Questionnaire (n = 47).

Variables	n	%	Mean	Standard deviation	Median	Minimum	Maximum
Gender
Male	7	15.6
Female	38	84.4
Age (years)	43		38.42	7.26	38.00	24.00	58.00
Professional group
Nurse practitioner	43	93.5
Decision-maker	3	6.5
Education
Master’s	44	95.7
Specialty certificate	2	4.3
Employment status
Permanent full-time	45	97.8
Permanent part-time	1	2.2
Licensed or registered in your professional role (years)	46		10.58	8.99	8.50	0.67	32.00
Current role in this organization (years)	46		5.19	4.86	3.00	0.42	25.00
Working with this team (years)	46		4.12	3.72	3.00	0.33	13.00

For aim 1, the team co-developed a training module including a 10-min online training video, user guides, frequently asked questions (FAQs) in English and in French. A communication plan was developed with participants to describe the instrument’s purpose, clarify how it would be used and circulated to NPs across the sites to support uptake. Participants reviewed the training material and provided feedback. Adjustments were made weekly or as needed to adapt the training material using the feedback and respond to questions.

NPs (n = 66, response rate: 42%) entered data using the NP-WI to pilot-test data entry, visualize fields included in the tool, and generate reports. NPs entered data for the NP-WI for 128 days, with 45 NPs (68%) completing data entry for at least 20 days (range: 1-60 days). The tool captured practice sites in primary care, long-term care and home care.

The percentage of time spent by NPs was primarily in the clinical (84%) dimension, followed by administration/leadership (8%), education (5%), and research (3%). On average, NPs saw 8.3 patients per day across community-based primary care settings (SD: 4.56, median: 8, range: 1-38) (see Table 2). The number of problems/patient concerns addressed per appointment ranged from 1 (67%), 2 (21%), 3 (8%), and 4 or more (4%). First appointments and follow-up for mental health conditions accounted for 10.5% of interventions. Most NPs (92.8%) indicated they were satisfied or very satisfied with the level of collaboration in the healthcare team, and approximately 2 thirds of NPs (65.8%) reported they were satisfied or very satisfied with their workload. NP perceptions of workload, organizational, team and patient characteristics are described in Table 3. Incomplete data entry was less than 1% except for NP perceptions of workload. Participants indicated that the check box was difficult to see which made it easy to pass over the item, particularly if data entry was completed on their portable phone. Patients (n = 420, 99.5%) overwhelmingly responded that they were satisfied with appointments times with NPs, and times were adequate to meet their needs. Reports were generated overall and for each site in table form or pie chart.

Table 2.

Mean Number of Patients Seen per Day per NP, by Setting.

Settings	Data collection days (n)	Number of patients seen per day per NP
Settings	Data collection days (n)	Mean	Standard deviation	Median	Minimum	Maximum
LTC	17	4.3	2.56	4	1	8
Home care	8	1.88	0.83	2	1	3
Ambulatory care	118	8.78	4.29	8	1	38
Mental health	37	5.42	2.45	5	1	15
Total	128	8.3	4.56	8	1	38

Abbreviations: LTC, long-term care; NP, nurse practitioner.

Table 3.

Description of NP Perceptions of Workload, Organizational, Team, and Patient Characteristics in the NP-WI.

Variables	Site 1		Site 2		Site 3		Total
Variables	n	%	n	%	n	%	n	%
NP perception of workload
Satisfied	1782	41.8	2406	60.4	1544	59.4	5732	52.8
Very satisfied	505	11.9	636	16.0	247	9.5	1388	12.8
Dissatisfied	516	12.1	253	6.4	506	19.4	1275	11.7
Very dissatisfied	92	2.2	12	0.3	81	3.1	185	1.7
Missing	1366	32.1	675	17.0	235	9.0	2276	21.0
Setting
Family medicine group	3337	78.3	2678	67.3	1189	45.5	7204	66.4
Local community service centre	235	5.5	357	9.0	842	32.2	1434	13.2
University-affiliated family medicine group	192	4.5	301	7.6	486	18.6	979	9.0
Medical clinic	207	4.9	445	11.2	0	0.0	652	6.0
Cooperative clinic	258	6.1	133	3.3	0	0.0	391	3.6
Residential and long-term care	0	0.0	41	1.0	75	2.9	116	1.1
Missing	32	0.8	27	0.7	21	0.8	80	0.7
Electronic health record (EHR)
Yes	4229	99.2	3277	82.3	2471	94.6	9977	91.9
No	0	0.0	678	17.0	121	4.6	799	7.4
Missing	32	0.8	27	0.7	21	0.8	80	0.7
Clerical support
Yes	4178	98.1	3460	86.9	2342	89.6	9980	91.9
No	51	1.2	495	12.4	250	9.6	796	7.3
Missing	32	0.8	27	0.7	21	0.8	80	0.7
Nursing support
Yes	3953	92.8	2823	70.9	2218	84.9	8994	82.8
No	276	6.5	1132	28.4	374	14.3	1782	16.4
Missing	32	0.8	27	0.7	21	0.8	80	0.7
Other NP(s) in your practice setting
Yes	2479	58.2	3123	78.4	2471	94.6	8073	74.4
No	1750	41.1	832	20.9	121	4.6	2703	24.9
Missing	32	0.8	27	0.7	21	0.8	80	0.7
Team size
Small (5 and less)	302	7.1	983	24.7	103	3.9	1388	12.8
Medium (5-9)	885	20.8	1234	31.0	908	34.7	3027	27.9
Large (more then 10)	3042	71.4	1738	43.6	1581	60.5	6361	58.6
Missing	32	0.8	27	0.7	21	0.8	80	0.7
Level of collaboration
Very satisfied	2332	54.7	1264	31.7	1681	64.3	5277	48.6
Satisfied	1897	44.5	2436	61.2	469	17.9	4802	44.2
Very dissatisfied	0	0.0	122	3.1	240	9.2	362	3.3
Dissatisfied	0	0.0	133	3.3	202	7.7	335	3.1
Missing	32	0.8	27	0.7	21	0.8	80	0.7
Appointment format
In-person	2769	65.0	2815	70.7	2125	81.3	7709	71.0
Telephone	1206	28.3	1031	25.9	438	16.8	2675	24.6
Virtual	254	6.0	109	2.7	29	1.1	392	3.6
Missing	32	0.8	27	0.7	21	0.8	80	0.7
Appointment type
Walk-in or advanced access	1795	42.1	2174	54.6	954	36.5	4923	45.3
Follow-up for chronic illness and/or comorbidities	872	20.5	1047	26.3	752	28.8	2671	24.6
Follow-up for mental health issue	638	15.0	200	5.0	195	7.5	1033	9.5
Follow-up for pregnancy	356	8.4	57	1.4	92	3.5	505	4.7
Periodic exam in children	133	3.1	198	5.0	161	6.2	492	4.5
Rostering/ registering a new patient	132	3.1	51	1.3	113	4.3	296	2.7
Reproductive and sexual health (eg, IUD insertion)	58	1.4	87	2.2	140	5.4	285	2.6
1st appointment for patient with chronic condition	85	2.0	40	1.0	41	1.6	166	1.5
1st appointment for mental health	60	1.4	22	0.6	23	0.9	105	1.0
STBBI appointment	50	1.2	46	1.2	15	0.6	111	1.0
1st appointment for pregnancy	49	1.1	12	0.3	27	1.0	88	0.8
LTC consultation	0	0.0	9	0.2	77	2.9	86	0.8
Home care consultation	1	0.02	12	0.3	2	0.1	15	0.1
Missing	32	0.8	27	0.7	21	0.8	80	0.7
Patients
Patients in vulnerable situations	977	21.2	404	9.6	550	18.9	1931	16.5
Patients aged 70 years or older	465	10.1	281	6.7	311	10.7	1057	9.0
Patients aged 80 years or older	182	3.9	295	7.0	156	5.4	633	5.4
Patients requiring a translator	45	1.0	62	1.5	45	1.5	152	1.3
Several people in the health visit (excluding well-baby appointment)	80	1.7	64	1.5	82	2.8	226	1.9
Clientele without specific health needs	64	1.4	23	0.5	15	0.5	102	0.9
Homeless clientele	54	1.2	5	0.1	0	0.0	59	0.5

Abbreviations: IUD, intra-uterine device; LTC, long-term care; NP, nurse practitioner; STBBI, sexually transmitted and blood-borne infection.

During the weekly implementation meetings, NPs and decision-makers noted that the number of problems/concerns that were addressed during the patient’s health visit was an important indicator of NP workload. It could be used to help clarify what makes NP practice distinct from other providers (eg, physicians). After noting the high percentage of patients who were seen with 4 or more problems in her caseload, a participant reflected: “Now I know why I feel overwhelmed!”

For aim 2, NPs and decision-makers found that the NP-WI was easy to use and included all the essential elements to measure NP workload. As indicated in Figure 1, the acceptability scores were positively rated with low scores noted for burden and opportunity costs. In the interviews, the NPs noted that data entry took 5 to 7 min each day and this was acceptable to them. They determined that a 4-week data collection period of NP activities was sufficient to capture a representative sample of NP workload. Two weeks were deemed insufficient to capture regular activities that occur less frequently (eg, team meetings). One NP noted: “The tool is easy to use!”

Figure 1.

Median scores of acceptability (n = 47). Item score ranges and corresponding anchors: Affective attitude (1. very uncomfortable—5. very comfortable), burden (1. none—5. huge), ethicality (1. very unfair—5. very fair), perceived effectiveness (1. strongly disagree—5. strongly agree), intervention coherence (1. strongly disagree—5. strongly agree), self-efficacy (1. very unconfident—5. very confident), opportunity costs (1. strongly disagree—5. strongly agree), general acceptability (1. completely unacceptable—5. completely acceptable).

Adjustments to the NP-WI included separating appointment types for walk-ins and advanced access, adding categories for appointment types related to follow-up of test results, completing forms, definitions to the user guide for patients in vulnerable situations, and adding white space around the item for NP perceptions of workload.

Discussion

A feasibility study was conducted in 3 health regions to implement an online NP-WI. NPs indicated that the NP-WI was easy to use. It took 5 to 7 min per day to complete data entry. The NP-WI was able to account for a wide range of patient, provider and organizational characteristics across community-based primary care settings, the time spent in clinical and non-clinical role dimensions, and the number of patients seen by NPs. Identifying the number of issues or problems addressed per visit may help to distinguish NP practice from other provider roles in healthcare teams. NPs determined that a 4-week collection period was sufficient to capture a representative sample of their workload. The study generated new knowledge to facilitate the uptake of an important health service innovation and identify areas for improvement prior to a wider roll-out.

There remains a paucity of research focussed on nurses and NP workload in primary care.^51,52 To address these concerns and ensure transparency in the use of the information, NPs were asked to assess their workload directly in the instrument. All participants assessed the adequacy of the instrument’s outputs. Subjective assessments of workload that incorporate professional judgement have been found to enhance appropriate resource allocation to meet patient needs.⁵³ The NP-WI can be used to support reflective practice for NPs. Decision-makers can more easily identify unevenly distributed workload. Professional development needs for NPs can also be pinpointed depending on the level of NP experience, changes in NP satisfaction of workload and changes in characteristics of patients seen by NPs (eg, increase in number of homeless persons). To support transparency, NPs and decision-makers have access to reports. These data can be used to promote discussions between NPs and decision-makers about their caseload to ensure that NP roles are used optimally, and to build relationships to improve staffing decisions.⁵⁴ The instrument provides NPs and decision-makers with information to improve NP workforce planning, account for the needs of patients with health issues (eg, home care, mental health, walk-in), optimize NP role implementation and increase access to care.

High workloads have deleterious effects on NPs and patients.⁵⁵ To determine what constitutes a reasonable workload, nurses cited effective workload management, modernizing processes, time management, communication and prioritization as key considerations.⁵⁶ Advanced practice nurses have expressed the need to go beyond the number of patients seen to measure clinical and non-clinical activities as well as distinguish between nursing and non-nursing activities (eg, clerical tasks).⁵⁷ Concerns about the adequacy of workload measurement were voiced as early as 1970.⁵⁸ To support decision-makers, these systems must capture the growing complexity of patient care and work environments to support decisions related to prospective planning of resources, determining current staffing adequacy or retrospectively reviewing past resource allocations and expenditures.^7,11,58,59

The use of a participatory approach allowed stakeholders to inform all steps of the implementation. Engaging users in the co-creation of technology can enhance uptake and sustainability.⁶⁰ To ensure that the patient’s voice was heard, our patient partner participated in all phases of the study from protocol development to publication. Our patient partner reviewed and discussed findings and generated recommendations. Also, offering both online and paper copy options for the 4-item patient questionnaire greatly increased the response rate and should be considered when patients are asked to complete a survey. We did not include interviews with patients at this stage of instrument development because we believed that it was premature to present the untested instrument to patients for their feedback.

Poor healthcare workforce planning is costly and leads to systemic inequities in care, particularly for underserved populations.^61,62 Additional research is needed to address the limitations of our research project. The study was conducted in 1 province in Canada with a small sample size and did not include patient outcomes. Research is needed to spread and scale up the NP-WI in other provinces or countries to capture more diverse NP roles, patient populations, and different care contexts.⁶³ These authors argued that scaling looks to expand the impact of an innovation and represents an effective strategy to reduce costs and waste in research. A larger sample size with participants in different jurisdictions will allow for a more in-depth analysis of the relationships between patient, NP, team, organization, and system level characteristics. There is a lack of research examining the link between NP staffing decisions and patient outcomes using validated tools.^5,64 Also, a comparison between online and observational data is needed to ensure the adequacy of the measure. Somensi et al⁶⁵ identified no significant differences between the measure but found that the online tool was a better reflection of real-time data.

Conclusions

It is feasible to implement the NP-WI. The measure captured patient, provider and organizational characteristics and the number of patients seen by NPs. The instrument can support health workforce planning with more adequate estimations of NP workload and caseload across community-based primary care settings and provide greater equity in resource allocation and distribution of workload for NPs. Ultimately, it will be possible to link decisions about workload and caseload to outcomes and address a long-standing critical gap in NP workforce planning.

Supplemental Material

sj-pdf-1-jpc-10.1177_21501319251321302 – Supplemental material for Implementing an Online Instrument to Measure Nurse Practitioner Workload: A Feasibility Study

Supplemental material, sj-pdf-1-jpc-10.1177_21501319251321302 for Implementing an Online Instrument to Measure Nurse Practitioner Workload: A Feasibility Study by Kelley Kilpatrick, Véronique Landry, Eric Nguemeleu Tchouaket, André Daigle and Mira Jabbour in Journal of Primary Care & Community Health

Footnotes

Acknowledgements

We thank all participants in this study for their time and feedback. We would also like to acknowledge the assistance of the Ministère de la Santé et des Services sociaux in the recruitment of participating sites, and of Ms R. Atallah in the preparation of the manuscript.

Author Contributions

Conceptualization: KK, VL, ENT, AD, MJ.

Data curation: KK, VL, ENT, AD, MJ.

Formal analysis: KK, VL, ENT, MJ.

Funding acquisition: KK, VL, ENT, AD, MJ.

Investigation: KK, VL, ENT, AD, MJ.

Methodology: KK, VL, ENT, AD, MJ.

Project administration: KK, MJ.

Software: MJ.

Supervision: KK.

Validation: KK, VL, ENT, AD, MJ.

Writing—original draft: KK, VL, ENT, MJ.

Writing—review & editing: KK, VL, ENT, AD, MJ.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Réseau de recherche en interventions en sciences infirmières du Québec (RRISIQ), Concours de soutien 2022 to 2023 aux projets innovants et structurants (award number 266683); the Ministère de la Santé et des Services sociaux du Québec (MSSS) (reference number 23-AU-01121); as well as the McGill University Faculty of Medicine and Health Sciences and the Newton Foundation via the Susan E. French Chair in Nursing Research and Innovative Practice held by KK. KK also holds a Fonds de recherche du Québec-Santé Research Scholar Senior salary award (award number 298573). ET holds the Canadian Research Chair Tier 2 in Economics of Infection Prevention and Control: ÉconoPCI (grant number CRC-2021-00321).

Ethical Considerations

The study was approved by the Research Ethics Committee (REC) of the Centre intégré universitaire de santé et de services sociaux (CIUSSS) de l’Est-de-l’Île-de-Montréal (approval #MP-12-2023-3358). Authorizations to carry out the project were obtained from the RECs of the CIUSSS de la Capitale-Nationale (#MEO-12-2024-2852), CISSS de Lanaudière (#MEO-12-2024-133), CIUSSS de l’Ouest-de-l’Île-de-Montréal (MEO-12-2024-899) and the Université de Moncton (file #2324-032). Participation was voluntary, and could be withdrawn at any time.

Consent to Participate

All participants were informed of study aims and provided written consent.

Consent for Publication

Not applicable.

ORCID iD

Kelley Kilpatrick

Data Availability Statement

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

Supplemental Material

Supplemental material for this article is available online.

References

World Health Organization. State of the World’s Nursing Report 2020- Investing in Education, Jobs and Leadership. 2020. Accessed January 28, 2025. https://www.who.int/publications/i/item/9789240003279

Canadian Council of Registered Nurse Regulators. National Entry-Level Competencies for Nurse Practitioners. 2023. Accessed January 28, 2025. https://www.ccrnr.ca/nurse-practitioners

International Council of Nurses., Schober

Lehwaldt

, et al. Guidelines on Advanced Practice Nursing 2020. 2020. Accessed January 28, 2025. https://www.icn.ch/system/files/documents/2020-04/ICN_APN%20Report_EN_WEB.pdf.

Kilpatrick

Tchouaket

Savard

, et al. Identifying indicators sensitive to primary healthcare nurse practitioner practice: a review of systematic reviews. PLoS ONE. 2023;18(9):e0290977.

Martin-Misener

Kilpatrick

Donald

, et al. Nurse practitioner caseload in primary health care: scoping review. Int J Nurs Stud. 2016;62:170-182.

Muldoon

Dahrouge

Russell

Hogg

Ward

How many patients should a family physician have? Factors to consider in answering a deceptively simple question. Healthc Policy. 2012;7(4):26-34.

MacPhee

Havaei

Bookey-Bassett

, et al. Commentary on the past, present, and future of nursing workload research. Nurs Res Rev. 2024;14:59-67.

Canadian Academy of Health Sciences. Canada’s Health Workforce: An Overview. 2023. Accessed January 28, 2025. https://cahs-acss.ca/wp-content/uploads/2023/04/E-HW-Overview.pdf

Splane

Horvath

Ziegler

, et al. Retention of Canadian advanced practice nurses: what will it take? Nurs Leadersh. 2023;36(1):16-32.

10.

Kilpatrick

Tchouaket

Jabbour

Hains

A mixed methods quality improvement study to implement nurse practitioner roles and improve care for residents in long-term care facilities. BMC Nurs. 2020;19:6.

11.

Housden

Tantongco

Jiang

Crowe

Optimizing patient care: a novel panel size calculator for primary care nurse practitioners. NPOJ Online First. 2024;1(1):33-42.

12.

Slade

DePriest

Commodore-Mensah

Samuel

Hanson

D’Aoust

State full practice authority regulations and nurse practitioner practice autonomy: evidence from the 2018 national sample survey of registered nurses. Med Care Res Rev. 2025;82(1):68-78.

13.

Poghosyan

Courtwright

Flandrick

, et al. Advancement of research on nurse practitioners: setting a research agenda. Nurs Outlook. 2023;71(5):102029.

14.

Kilpatrick

Landry

Al Hakim

, et al. Predictors of time spent by nurse practitioners on activities: time and motion studies. Nurs Pract Open J. 2023;3(2):66-79.

15.

Landry

Kilpatrick

Paquette

, et al. The scope of nurse practitioner roles in primary health care in different practice setting in Quebec: a time and motion study. Sci Nurs Health Pract. 2020;3(6):1-14.

16.

Lopetegui

Yen

Lai

Jeffries

Embi

Payne

Time motion studies in healthcare: what are we talking about?

J Biomed Inf. 2014;49:292-299.

17.

Nurse Practitioner Association of Canada. Nurses Practitioners Licensed by Province & Territory-2024. 2024. Accessed January 28, 2025. https://npac-aiipc.org/nps-in-canada/

18.

Landry

Kilpatrick

Essential elements of a workload measurement instrument for nurse practitioners. J Nurse Pract. 2024;20(10):105214.

19.

Landry

Kilpatrick

Co-construction of a workload measurement instrument for nurse practitioners in primary healthcare. Paper presented at: The 12th International Council of Nurses Nurse Practitioner/Advanced Practice Nursing Network Conference; August 21-24, 2022; Dublin, Ireland (Oral presentation).

20.

Landry

Kilpatrick

Développer un instrument de charge de travail des infirmières praticiennes: La voix des patients. Paper presented at: 8e Conférence du Secrétariat international des infirmières et infirmiers de l’espace francophone (SIDIIEF); October 19, 2022. Ottawa, Canada (Poster presentation).

21.

Kilpatrick

Savard

Audet

L-A

, et al. A global perspective of advanced practice nursing research: a review of systematic reviews. PLoS ONE. 2024;19(7):e0305008.

22.

Weiner

Lewis

Stanick

, et al. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci. 2017;12(1):108.

23.

Wang

Kenyon

McGilton

, et al. The time is now: a FASTER approach to generate research evidence for technology-based interventions in the field of disability and rehabilitation. Arch Phys Med Rehabil. 2021;102(9):1848-1859.

24.

Sekhon

Cartwright

Francis

JJ.

Acceptability of healthcare interventions: an overview of reviews and development of a theoretical framework. BMC Health Serv Res. 2017;17(1):88.

25.

Lancaster

Pilot and feasibility studies come to age!

Pilot Feasibil Stud. 2015;1(1):1.

26.

Teresi

Stewart

Hays

RD.

Guidelines for designing and evaluating feasibility pilot studies. Med Care. 2022;60(1):95-103.

27.

Roper

Skeat

Innovation through participatory design: collaborative qualitative methods in the development of speech-language pathology technology. Int J Speech Lang Pathol. 2022;24(5):527-532.

28.

Sekhon

Cartwright

Francis

JJ.

Acceptability of health care interventions: a theoretical framework and proposed research agenda. Br J Health Psychol. 2018;23(3):519-531.

29.

Kilpatrick

Paquette

Jabbour

, et al. Systematic review of the characteristics of brief team interventions to clarify roles and improve functioning in healthcare teams. PLoS ONE. 2020;15(6):e0234416.

30.

Kilpatrick

Jabbour

Tchouaket

Acorn

Donald

Hains

Implementing primary healthcare nurse practitioners in long term care teams: a qualitative descriptive study. J Adv Nurs. 2019;75(6):1306-1315.

31.

Savard

Jabbour

Tchouaket

Gauthier

Kilpatrick

Evaluating the influence of primary healthcare nurse practitioners’ interventions in home care on hospitalizations and emergency department transfers. J Eval Clin Pract. 2024;30(3):440-452.

32.

Landry

Kilpatrick

Jabbour

A workload measurement instrument for nurse practitioners: finding consensus with the TRIAGE method. Nurs Pract Open J. Under review.

33.

Sekhon

Cartwright

Francis

JJ.

Development of a theory-informed questionnaire to assess the acceptability of healthcare interventions. BMC Health Serv Res. 2022;22(1):279.

34.

Saad

Zia

Raza

Kundi

Haleem

A comprehensive analysis of healthcare websites usability features, testing techniques and issues. IEEEAccess. 2022;10:97701-97718.

35.

Totton

Lin

Julious

Chowdhury

Brand

A review of sample sizes for UK pilot and feasibility studies on the ISRCTN registry from 2013 to 2020. Pilot Feasibil Stud. 2023;9(1):188.

36.

Kunselman

AR.

A brief overview of pilot studies and their sample size justification. Fertil Steril. 2024;121(6):899-901.

37.

Kim

Sefcik

Bradway

Characteristics of qualitative descriptive studies: a systematic review. Res Nurs Health. 2017;40(1):23-42.

38.

Yajima

Bowe

Barber

Dains

Telehealth interventions for supportive management and early recognition of treatment-related symptoms in patients with hematologic malignancies. J Adv Pract Oncol. 2021;12(8):835-849.

39.

Pant Pai

Chiavegatti

Vijh

, et al. Measures and metrics for feasibility of proof-of-concept studies with human immunodeficiency virus rapid point-of-care technologies: the evidence and the framework. Point Care. 2017;16(4):141-150.

40.

Mailhot

Goulet

Maheu-Cadotte

Fontaine

Lequin

Lavoie

Methodological reporting in feasibility studies: a descriptive review of the nursing intervention research literature. J Res Nurs. 2020;25(5):460-472.

41.

Arain

Campbell

Cooper

Lancaster

GA.

What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Med Res Methodol. 2010;10:67.

42.

Thabane

Chu

, et al. A tutorial on pilot studies: the what, why and how. BMC Med Res Methodol. 2010;10:1.

43.

Pearson

Naylor

Ashe

Fernandez

Yoong

Wolfenden

Guidance for conducting feasibility and pilot studies for implementation trials. Pilot Feasibil Stud. 2020;6(1):167.

44.

Vaismoradi

Turunen

Bondas

Content analysis and thematic analysis: implications for conducting a qualitative descriptive study. Nurs Health Sci. 2013;15(3):398-405.

45.

Cederroth

Earp

Gómez Prada

, et al. Integrating gender analysis into research: reflections from the Gender-Net Plus workshop. eClinicalMedicine. 2024;24:1-7.

46.

Elo

Kyngäs

The qualitative content analysis process. J Adv Nurs. 2008;62(1):107-115.

47.

Plano Clark

. Meaningful integration within mixed methods studies: identifying why, what, when, and how. Contemp Educ Psychol. 2019;57(2019):106-111.

48.

Guetterman

Fetters

Creswell

JW.

Integrating quantitative and qualitative results in health science mixed methods research through joint displays. Ann Fam Med. 2015;13(6):554-561.

49.

Carter

Bryant-Lukosius

DiCenso

Blythe

Neville

AJ.

The use of triangulation in qualitative research. Oncol Nurs Forum. 2014;41(5):545-547.

50.

Ogrinc

Davies

Goodman

Batalden

Davidoff

Stevens

SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process. J Nurs Care Qual. 2016;31(1):1-8.

51.

Dickson

Cramer

Peckham

CM.

Nursing workload measurement in ambulatory care. Nurs Econ. 2010;28(1):37-43.

52.

Dye

Wells

Subjective and objective measurement of neonatal nurse practitioner workload. Adv Neonatal Care. 2017;17(4):E3-E12.

53.

Ferguson-Pare

Bandurchin

The Ontario nursing workload demonstration projects: rethinking how we measure, cost and plan the work of nurses. Nurs Leadersh. 2010;23:20-32.

54.

Waterfield

Barnason

The integration of care ethics and nursing workload: a qualitative systematic review. J Nurs Manage. 2022;30(7):2194-2206.

55.

Kim

Scott

Poghosyan

Martsolf

GR.

Burnout, job satisfaction, and turnover intention among primary care nurse practitioners with their own patient panels. Nurs Outlook. 2024;72(4):102190.

56.

Buttram

Stadtmiller

Research review and notes using appreciative inquiry to understand workload reasonability. AI Practitioner. 2024;26(2):100-116.

57.

Kapu

McComiskey

Buckler

, et al. Advanced practice providers’ perceptions of patient workload: results of a multi-institutional survey. J Nurs Adm. 2016;46(10):521-529.

58.

Griffiths

Saville

Ball

, et al. Nursing workload, nurse staffing methodologies and tools: a systematic scoping review and discussion. Int J Nurs Stud. 2020;103:103487.

59.

Scott

Cockerill

Wright

O’Brien Pallas

A survey of workload measurment systems: the occupational therapy manager’s perspective. Can J Occup Ther. 1993;60:17-22.

60.

MacNeil

Koch

Kuspinar

Juzwishin

Lehoux

Stolee

Enabling health technology innovation in Canada: barriers and facilitators in policy and regulatory processes. Health Policy. 2019;123(2):203-214.

61.

Bourgeault

Simkin

Chamberland-Rowe

Poor health workforce planning is costly, risky and inequitable. CMAJ. 2019;191(42):E1147-E1148.

62.

Canadian Academy of Health Sciences. Canada’s Health Workforce: Pathways Forward. 2023. Accessed January 28, 2025. https://cahs-acss.ca/assessment-on-health-human-resources-hhr/

63.

Corôa

Gogovor

Ben Charif

, et al. Evidence on scaling in health and social care: an umbrella review. Milbank Q. 2023;101(3):881-921.

64.

Kilpatrick

Tewah

Jabbour

Tchouaket

Adherence to best practice: validating a care quality index for primary healthcare nurse practitioners. Paper presented at: The 13th ICN Nurse Practitioner/Advanced Practice Nurse (NP/APN) Network Conference; September 11, 2024; Aberdeen, Scotland (Oral presentation). 2024.

65.

Somensi

Caregnato

RCA

Cervi

Flores

CD.

Workload: a comparison between the online and observational methods. Rev Bras Biol. 2018;71(4):1850-1857.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.09 MB