Referral quality and justification rate in MRI for the lower back: evaluation of an intervention to reduce imaging og low value

Abstract

Background

Lower back magnetic resonance imaging (MRI) is a common imaging procedure; however, it often has low value in patients with uncomplicated back pain. Measures to reduce low-value procedures are necessary for a sustainable imaging service.

Purpose

To evaluate the outcomes of the national intervention implemented in imaging centers by assessing referral quality and justification rate of lower back MRI in private imaging centers in Norway.

Material and Methods

This study used a before-after design to assess referrals retrospectively for justification and information quality to evaluate the intervention's effect. Four radiologists and four radiographers, all experienced in the field, assessed 360 referrals, half from before and half after the intervention. Descriptive, comparative, and Kappa statistics were used to analyze the data.

Results

The rate of justified referrals increased from 60% to 65% when comparing before and after the implementation. The rate of unjustified referrals was reduced from 24% before to 19% after. The rate of referrals that lacked information was unchanged. Further, referral quality slightly increased after the intervention; still, 88% of the referrals were considered low quality. Radiologists were stricter in their assessment of referrals compared to radiographers. The interrater agreement was moderate in justification assessment and very good in referral quality assessment.

Conclusion

The intervention slightly improved referral quality and justification rate. However, with a 20% unjustified rate after the intervention, further measures are needed to increase service quality and reduce wait times.

Keywords

Low back pain ethics medical magnetic resonance imaging radiologist radiologic technologists

Introduction

Musculoskeletal disorders, particularly low back pain, constitute a substantial health challenge for individuals, health service providers, and health policymakers worldwide, as in Norway (1,2). Magnetic resonance imaging (MRI) is the most commonly used imaging modality to diagnose lumbosacral conditions, e.g. radiculopathy, stenosis, infection, malignancy, and spinal hemorrhage. However, for many patients with uncomplicated low back pain, an MRI would not influence their treatment or health outcomes and thus would be of low value for the patient (3,4). Uncomplicated low back pain is defined as pain in the lumbar region without red flag symptoms that suggest underlying medical or surgical causes, such as radiculopathy, cancer, fracture, or infection (5). The use of imaging in patients with uncomplicated low back pain represents a substantial financial burden for healthcare services, generates wait times with potentially poorer outcomes for other patients, and reduces the quality and safety of services (6). Thus, addressing this issue is crucial for ensuring the cost-effectiveness of healthcare delivery (3,7). Several practice guidelines and recommendations have been developed internationally to reduce the unwarranted use of imaging for low back pain, such as the ESR iGuide (8) and the Choosing Wisely recommendations (9 –11).

Even with guidelines and recommendations, unnecessary imaging is still performed, and the geographical variation of imaging utilization in low back pain is large, indicating possible overutilization of imaging for these patients (4,12,13).

Earlier research has reported that referral quality affects justification rate, and good quality referrals should, according to Pitman (14), be legible and unambiguous, and give adequate clinical information to select the most appropriate procedure and the reason for doing the procedure with a clear diagnostic question. Repeated and continuous interventions targeting referring clinicians’ adherence to referral guidelines improve the overall quality of MRI referrals (15) and multi-component interventions are the most successful in reducing low-value imaging (16). If clinicians provide detailed patient clinical information in referrals, they assist radiologists and radiographers in justifying imaging, and unwarranted imaging can be avoided (14,17).

A multi-component intervention was piloted at two private imaging centers in Norway and showed no change in referral quality or justification rate (18). Learning from the piloting, the intervention was developed and extended nationally to all imaging centers (19). The aim of the present study was to evaluate the outcomes of the national intervention implemented in imaging centers by assessing referral quality and justification rate of lower back MRI in private imaging centers in Norway.

Material and Methods

The Regional Committees for Medical Research Ethics in Central-Norway approved this study in 2022 (approval no. 378396). The requirement for patient consent was waived.

Context and intervention

This study uses a before–after design to assess referrals retrospectively for justification and quality as a measure of the intervention's effect. The imaging centers included in this study are private providers contracted with public regional health authorities to perform outpatient imaging on public terms. The centers also perform imaging on self-paying and insured patients. Private imaging centers do not have access to patient records and thus rely on good-quality referrals for justification assessment.

MRI referrals at the centers are received from general practitioners, specialists, chiropractors, or manual therapists and are assessed primarily by experienced radiographers. If the radiographers need support in their assessment, a radiologist will assess the referral.

The development and implementation of the intervention evaluated in this study are described in detail in Hofmann et al. (19). In sum, the intervention consisted of three components: implementing a new procedure for referral assessment according to the Choosing Wisely recommendations, sending an informative and polite return letter to referrers when imaging was found unjustified according to the recommendations, and an information campaign for patients, professionals, and the public. The Norwegian version of the Choosing Wisely campaign states the following recommendation for advanced imaging for uncomplicated low back pain: avoid diagnostic imaging for low back pain in adults without red flags; red flags include fever or other signs of infection, history of injury, or recent spinal puncture, accompanying general symptoms, fresh urinary difficulties, or other severe or progressive neurological deficits (which may suggest epidural abscess or other infections, epidural hematoma, fracture, cancer, or cauda equina syndrome). In case of suspected uncomplicated disc herniation or uncomplicated spinal stenosis, imaging diagnostics are only indicated after 4–6 weeks of conservative treatment and if surgery is being considered (20). The intervention started on 1 October 2022 and lasted until the end of June 2023.

Data collection

A total of 360 referrals for lumbar MRI in adult patients with low back pain were collected from six imaging centers operated by the two major private imaging providers in Norway, where the intervention was implemented, located in different parts of the country. The data collected included referral text, patient sex and age, center name, and year received. Each center provided 60 referrals, with 30 received before and 30 after the intervention. The referrals were received at the centers in January and February 2022 and 2023, before and after the implementation. Referrals were collected in March and April 2024.

The referrals were reviewed manually to remove identifiable information such as name, telephone number, geographical location, place of work, and name of referrer, and to ensure they were all related to low back pain. Data were then coded to blind the assessors for the time the referral was received and the name of the imaging center and put in a random order in an assessment form.

The assessment form in Excel included the referral text and the patient's age and sex, providing one row per patient. The assessors were to assess the information in the referrals in seven different categories based on the Choosing Wisely recommendations, six of which were ranked yes/no and one defining duration of symptoms. In addition, the assessors should give an overall conclusion on justification with three alternatives: Yes/No/Need more information. A free-text column was provided so the assessors could add necessary comments. The categories of the assessment form are illustrated in Table 1.

Table 1.

Assessment form developed for the referral assessment.

The referral contains enough relevant information on:
Tentative diagnoses	Patient history	Clinical examination	Red flags	Conservative treatment given	Considering surgery?	Duration of symptoms	Imaging justified?
Yes	Yes	Yes	Yes	Yes	Yes	<4 weeks	Yes
No	No	No	No	No	No	>4 weeks	No
–	–	–	–	–	–	Undefined	Need more information
–	–	–	–	–	–	Not given	–

Assessors

Eight assessors were recruited from private imaging providers, not necessarily from the centers providing referrals. Four radiologists and four radiographers who usually assess referrals in clinical practice were invited and consented to participate.

Data analysis

To evaluate the quality, the ratings were scored as Yes = 1 and No = 0 in the following categories: tentative diagnosis, patient history, clinical examination, red flags, conservative treatment, and considering surgery. In the category of duration of symptoms, specification of more or less than 4 weeks gave 1 point, and Undefined/No gave 0 points. This gave a maximum possible score of 7 points. 0–4 points were considered a low-quality referral, while a referral with 5–7 points was considered a high-quality referral. All assessors reviewed each referral; thus, the observations are not independent. Statistics were conducted in Microsoft Excel version 2406 (Microsoft Corp., Redmond, WA, USA) and SPSS Statistics version 28 (IBM Corp., Armonk, NY, USA). Kappa statistics, Gwet's AC2, were used to assess interrater reliability. The Kappa statistic was interpreted as follows: <0.20 = poor, 0.21–0.40 = fair, 0.41–0.60 = moderate, 0.61–0.80 = good, 0.81–1 = very good (21).

Results

The eight assessors assessed all 360 referrals (2880 assessments), 50% from before and 50% after the implementation. There were no missing data. The mean rate of justification before and after the intervention is presented in Fig. 1. The justified rate changed from 60% to 65% when comparing before and after the implementation. The rate of unjustified referrals was reduced from 24% before to 19% after. The rate of referrals that lacked information was unchanged.

Fig. 1.

Mean percentage rate of referrals assessed as justified, unjustified, or need more information before and after intervention implementation.

All assessors agreed that 22.5% of the referrals were justified and that 3% were unjustified. In none of the referrals did all assessors agree that more information was needed. Gwet's AC2 showed a moderate interrater agreement of 0.59 (95% confidence interval [CI] = 0.53–0.64).

Referral quality

The assessors were asked to rate seven categories of information in the referrals. Table 2 gives an overview of the assessment of referral information. Less than 10% of the referrals included information on whether surgery was under consideration or red flags, and only 22% included information on conservative treatment. About half of the referrals included a tentative diagnosis. Most referrals included patient history and duration of symptoms, and findings from clinical examinations were included in just above 60% of the referrals.

Table 2.

Overview of the mean number of referrals with or without information given in the seven categories of referral information.

	Tentative diagnoses	Patient history	Clinical examination	Red flags	Conservative treatment	Considering surgery?	Information about the duration of symptoms
Yes	172 (48)	321 (89)	231 (64)	33 (9)	80 (22)	12 (3)	251 (70)
No/undefined	189 (52)	39 (11)	129 (36)	327 (91)	281 (78)	348 (97)	109 (30)

Values are given as mean n (%).

The referrals had a mean quality score of 3 (range = 0–5). Only 25 referrals had a mean score of 5 and were thus assessed as good quality. Fig. 2 gives an overview of the scores given by individual assessors (range = 0–7) before and after intervention (Fig. 2). The most common scores were 3–4, in the upper part of low-quality referrals. Of the referrals, 88% had a low-quality score (<5), indicating a potential for increased referral quality. Referrals after the intervention were more often rated with a higher score. Interrater variability was very good, 0.90 (95% CI = 0.89–0.91).

Fig. 2.

Individual referral quality score of referrals before and after the intervention.

Differences among professions

Radiographers conducted the primary assessment of referrals in the centers included in the intervention, only assisted by radiologists when clarification was needed. Thus, comparing radiographers’ and radiologists’ assessments are relevant in this study. As shown in Table 3, radiologists assessed, on average, 59% of referrals as justified, 26% as unjustified, and 15% as needed for more information. The radiographers assessed 65% as justified, 18% as unjustified, and in 17% of the referrals, there is a need for more information. The interrater variability was moderate in both groups: 0.56 (95% CI = 0.49–0.62) among the radiologists and 0.60 (95% CI = 0.53–0.66) among the radiographers.

Table 3.

Referrals who were justified, unjustified, and need more information.

Assessor	Justified	Unjustified	Need more information
Radiologist
1	215 (59.7)	69 (19.2)	76 (21.1)
2	265 (73.6)	84 (23.3)	11 (3.1)
3	176 (48.9)	84 (23.3)	100 (27.8)
4	194 (53.9)	140 (38.9)	26 (7.2)
Mean	212.5 (59.0)	94.3 (26.2)	53.3 (14.8)
Radiographer
5	260 (72.2)	41 (11.4)	59 (16.4)
6	202 (56.1)	83 (23.1)	75 (20.8)
7	292 (81.1)	40 (11.1)	28 (7.8)
8	178 (49.4)	96 (26.7)	86 (23.9)
Mean	233 (64.7)	56 (18.1)	62 (17.2)

Values are given as n (%).

The radiographers scored the referrals higher than the radiologists in the referral quality scoring, with more referrals given scores of 4 and 5 compared to the radiologists. An overview of the quality scoring per profession is shown in Fig. 3. However, radiographers more often gave referrals a score of 0. At the same time, only radiologists assessed referrals to have a top score. The interrater agreement was very good in both professions.

Fig. 3.

Quality scoring per referral comparing radiologist and radiographer.

Free-text comments

The free-text option of the form received 353 comments, 129 from radiologists and 224 from radiographers. Most commonly, the comment asked for more information on the duration of symptoms or clinical information (n = 190) or why the assessors had made their conclusion (n = 127). In two referrals, radiographers commented that they wanted to discuss the case with a radiologist.

Discussion

Earlier research showed a high rate (>50%) of negative findings in lower back MRI examinations and that imaging seldom changes patients’ treatment (4). Implementing this multi-component intervention slightly reduced unnecessary MRI in low back pain. Still, 20% of the referrals were assessed to be unjustified after the intervention.

The interrater agreement in justification among the assessors in this study was moderate. In only 25% of the referrals, all assessors came to the same conclusion. Thus, more training and better guidelines are needed to maintain consistency in referral assessment. However, providing correct and sufficient information in the referral would also help radiographers and radiologists in their assessment, thus improving the justification rate and, subsequently, the patient's treatment. For the assessors in private imaging centers to distinguish between patients with and without red flags, the information given in the referral is crucial, as they do not have access to any other information on the patient (5). In this study, referrals often lacked information on red flags, planned surgery, and conservative treatment, three of the most important factors in the Choosing Wisely recommendation (20). The assessors often commented on a lack of information. With a high rate of low-quality referrals (88%), referral assessment is challenging. Earlier research has shown that rates of unnecessary imaging due to low-quality referrals are in the range of 20%–75% (22). Thus, there is a need for referrers to be trained to improve their referrals.

The radiographers in the private imaging centers involved in the intervention primarily assess MRI referrals. The analysis showed that radiographers assessed a higher rate as justified and had a lower rate of unjustified referral assessments. Further, they rated the referrals as higher quality than the radiologists. As radiologists have higher competencies in imaging and medicine, one would rate their conclusions as more accurate than those of radiographers. Still, the interrater agreement was moderate among both groups. Only in two referrals did a radiographer comment that they wanted to discuss the case with a radiologist, indicating that the radiographers were comfortable with their decisions. According to Chilanga et al. (23), 57% of radiographers complied with guidelines when assessing referrals, and radiographers with postgraduate education and leading positions were more accurate in referral assessment. Thus, there seems to be a need for more training among radiographers assessing referrals in clinical practice to reduce the unnecessary use of imaging.

The referrals were collected from the imaging centers as the first 30 received in each period, before and after the intervention. Without information about the referrer, we do not know whether the after period included referrals sent from referrers who had received return letters or how many had received letters. However, the information campaigns targeted all referrers and thus assessed the intervention as a whole, not just the effect of the letters. This makes it challenging to determine the reason for the changes shown. To provide evidence-based referral assessment procedures, further research is needed on the clinical outcome of low-back pain MRI examinations in these private imaging centers. Due to dependent data, no formal hypothesis testing was performed; thus, results were presented descriptively. Statistical precision is therefore limited, and the conclusion should be interpreted with caution.

Further, this study focuses on private imaging centers, which may limit its transferability to public health systems. However, private imaging centers in Norway are largely commissioned by the public health authority. As private imaging centers cannot access patient records, the referral assessment process strongly depends on good-quality referrals. Transferability to other countries and settings would depend on how similar the clinical and system settings are to the Norwegian setting.

The research team developed the referral information form and quality scoring system in collaboration with radiologists. However, there might be other relevant information we should have asked the assessors to consider that would have been relevant in referral assessment. Further, we kept all at a 0 or 1 score as we considered all information relevant in the assessment. Using a more accurate scoring method, such as weighted scoring, could have revealed if some information might be more relevant than others. However, our scoring system was unweighted, as all information was considered equally relevant for referral assessment and image reporting and was expected to be included in all referrals.

The assessors were all employed at private imaging centers, and one could argue that experts working in radiological departments in public hospitals could or should have been used. However, as we evaluated the change in the context of private centers, we found good reasons to use experts at private imaging centers as assessors. Moreover, it would have been interesting to know which referrals led to examinations, but this is not crucial to investigating the justification rate and referral quality. We assessed whether referrals were justified according to whether they satisfied Norwegian and international guidelines criteria, including information content. This may, of course, vary from country to country.

In conclusion, the intervention led to a modest change in justification rate and referral quality. However, 20% of the referrals were still assessed as unjustified, and 88% were considered low quality. Thus, there is a need for further improvement. Radiologists were stricter in their assessment than radiographers; radiographers’ assessments may lead to unnecessary MRI procedures. Better training of radiographers could, therefore, improve the quality and safety of the imaging services. Further measures should be taken to improve referral quality and justification rates in private imaging centers in Norway. These measures could include referrer and imaging staff training, better access to patient information across health service levels, and a referral and imaging funding change.

Footnotes

Data availability

The data supporting this study's findings are available from the corresponding author upon reasonable request.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Norwegian Research Council (Norges Forskningsråd) (grant no. 302503).

ORCID iDs

Elin Kjelle

Ingrid Øfsti Brandsæter

Jan Porthun

Bjørn Morten Hofmann

References

Kinge

Knudsen

Skirbekk

, et al. Musculoskeletal disorders in Norway: prevalence of chronicity and use of primary and specialist health care services. BMC Musculoskelet Disord 2015;16:75.

Vos

Flaxman

Naghavi

, et al. Years lived with disability (YLDs) for 1160 sequelae of 289 diseases and injuries 1990–2010: a systematic analysis for the global burden of disease study 2010. Lancet 2012;380:2163–2196.

Rao

Scuderi

, et al. The use of imaging in management of patients with low back pain. J Clin Imaging Sci 2018;8:30.

Kjelle E, Andersen ER, Krokeide AM, et al. Characterizing and quantifying low-value diagnostic imaging internationally: a scoping review. BMC Med Imaging 2022;22:73.

Ruzycki

Jasaui

Mortazhejri

, et al. Understanding patients’ perceptions of uncomplicated low back pain: a theory-informed qualitative study using the common-sense self-regulation model. BMC Prim Care 2025;26:83.

Hofmann B, Brandsæter IØ, Kjelle E. Variations in wait times for imaging services: a register-based study of self-reported wait times for specific examinations in Norway. BMC Health Serv Res 2023;23:1287.

Kjelle E, Brandsæter IØ, Andersen ER, et al. Cost of low–value imaging worldwide: a systematic review. Appl Health Econ Health Policy. Epub ahead of print March 1, 2024. DOI: 10.1007/s40258–024–00876–2.

ESR iGuide. ESR | European Society of Radiology. Available at: https://www.myesr.org/eu-international-affairs/policy-topics/esr-iguide/ (accessed December 12, 2024).

Hong

Ross-Degnan

Zhang

, et al. Clinician-Level predictors for ordering low-value imaging. JAMA Intern Med 2017;177:1577–1585.

10.

Levin

Rao

. Reducing inappropriate use of diagnostic imaging through the choosing wisely initiative. J Am Coll Radiol 2017;14:1245–1252.

11.

Rao

Levin

. The overuse of diagnostic imaging and the choosing wisely initiative. Ann Intern Med 2012;157:574–576.

12.

Hofmann BM, Brandsæter IØ, Andersen ER, et al. Temporal and geographical variations in diagnostic imaging in Norway. BMC Health Serv Res 2024;24:463.

13.

Pakpoor

Raad

Harris

, et al. Use of imaging during emergency department visits for low back pain. AJR Am J Roentgenol 2020;214:395–399.

14.

Pitman

. Quality of referral: what information should be included in a request for diagnostic imaging when a patient is referred to a clinical radiologist? J Med Imaging Radiat Oncol 2017;61:299–303.

15.

Mesner

Foster

French

. Implementation interventions to improve the management of non-specific low back pain: a systematic review. BMC Musculoskelet Disord 2016;17:258.

16.

Kjelle E, Andersen ER, Soril LJJ, et al. Interventions to reduce low–value imaging – a systematic review of interventions and outcomes. BMC Health Serv Res 2021;21:983.

17.

Mendelson

. Diagnostic imaging: doing the right thing. J Med Imaging Radiat Oncol 2020;64:353–360.

18.

Chilanga CC, Heggelund M, Kjelle E. Assessing MRI referrals' appropriateness for low back pain post a radiology-initiated intervention. Radiography 2024;30:1277–1282.

19.

Hofmann B, Andersen ER, Brandsæter IØ, et al. Success factors for interventions to reduce low-value imaging. Six crucial lessons learned from a practical case study in Norway. Curr Probl Diagn Radiol 2024;52:6.

20.

Norwegian Society of radiologists. Gjør kloke valg – Radiologi. Available at: https://www.legeforeningen.no/foreningsledd/fagmed/norsk-radiologisk-forening/artikler/fag-og-utdanningsstoff-fra-noraforum/gjor-kloke-valg-radiologi/ (accessed August 16, 2024).

21.

McHugh

. Interrater reliability: the kappa statistic. Biochem Med (Zagreb) 2012;22:276–282.

22.

Chilanga

Lysdahl

. The radiographers’ opinion on assessing radiological referrals. Radiography 2024;30:605–611.

23.

Chilanga

Olerud

Lysdahl

. Radiographers’ actions and challenges when confronted with inappropriate radiology referrals. Eur Radiol 2022;32:4210–4217.