System-Level Variation in Multiple Sclerosis Care Outcomes: Initial Findings from the Multiple Sclerosis Continuous Quality Improvement Research Collaborative

Abstract

Multiple sclerosis (MS) is a “3C” (complex, chronic, costly) condition that is a common and disabling neurological illness affecting approximately 1 million adults in the United States. MS has been studied at the basic science, individual, and population levels, but not at the system level to assess small-area variation effects on MS population health outcomes. System-level effects have been observed in other 3C conditions including cystic fibrosis, rheumatoid arthritis, and inflammatory bowel disease. The authors report here on system-level variation findings from the baseline period during the first year of the Multiple Sclerosis Continuous Quality Improvement (MS-CQI) study. Stepwise binary logistic regression analyses were conducted to investigate system-level (small-area variation) effects on MS relapses (exacerbations), disease-modifying therapy (DMT) utilization, and brain MRI utilization, controlling for demographics (age and sex) and other potential confounders. Significant differences were observed in people with MS (PwMS) between centers for a number of demographic and disease characteristics, including sex, age, and MS subtype. Controlling for these factors, significant system-level effects were observed on outcomes, including DMT utilization, MRI utilization, and relapses. Significant relationships also were observed between outcomes and urgent care utilization, including emergency department visits and hospitalizations. This initial study provides evidence establishing the presence of system-level variation effects on MS outcomes in a multicenter population study – where PwMS get their care can influence their outcomes. Results support continued systems-level research and improvement initiatives to optimize MS population health outcomes in this challenging and costly complex chronic condition.

Introduction

Multiple sclerosis (MS) is a complex, chronic, and costly condition that is among the most common and disabling neurological diseases in adults. MS affects nearly 1 million Americans,¹ with a prevalence of 309 per 100,000 adults, and generates substantial burden on people with MS (PwMS), the health care system, and society as a whole, with estimated annual individual cost to a PwMS of $8528-$54,244.^2

–7 MS has been investigated at the basic science, individual, and population levels of analysis, including large data registry initiatives such as the North American Research Committee on MS (NARCOMS),⁸ MS Partners Advancing Technology and Health Solutions,⁹ the MS Leadership and Innovation Network,¹⁰ and the North American Registry for Care and Research in MS (NARCRIMS).¹¹

However, in the new era of health care reform and increasing pressure to demonstrate value in the management of conditions such as MS, these approaches, while necessary and important, are insufficient. A paradigm shift toward the inclusion of systems-level approaches will be required to study and improve MS care and to demonstrate its value at the population level. “Systems-level” refers to a unit of analysis that is aggregated at a higher level than that of the individual clinician (eg, physician, nurse practitioner) and that is focused at the level of the service delivery system (eg, clinic, department, hospital), but at a lower level than that of epidemiological studies of populations (eg, all MS patients in the United States).

The Institute of Medicine (IOM) reports¹² on quality and safety deficiencies in the US health care system and the Institute for Healthcare Improvement Triple Aim¹³ have called for a new systems-oriented focus and continuous improvement culture. Wennberg's research on small-area geographic variation,¹⁴ which established the Dartmouth Atlas of Health Care,¹⁵ demonstrates that local practice culture and patterns can displace evidence-based care and influence unwarranted utilization and increased costs without associated benefits to population health outcomes.^16,17 Finally, the Affordable Care Act has influenced a shift from delivery- and production-based models to value-based reimbursement (accountable care).¹⁸

Quality measurement

There has been limited discourse regarding quality indicators in MS care^19
–21 including a concise set of clinical and process measures developed by the American Academy of Neurology focused on the provision of key MS clinical services and related outcomes. This study used Nelson's balanced measures Clinical Value Compass framework to specify this study of process, outcomes, and utilization measures.²² Nelson's model has 4 categories: (1) clinical outcomes (eg, relapse rate, magnetic resonance imaging [MRI] status, symptoms); (2) functional health (eg, self-efficacy, quality of life, Social Security disability status); (3) patient experience and satisfaction; and (4) utilization (eg, MRI utilization, appointment frequency, and emergency department [ED] utilization). Categories 1–3 are considered to be quality measures, and category 4 represents cost/utilization. Use of Nelson's framework can enable calculations of system-level value indices (quality/cost).^22,23

Learning health systems for quality improvement and population health research

Rigorous systems-level quality performance measurement can inform effective systems-level quality improvement (QI) and facilitate linked population health research efforts, especially when a Learning Health System (LHS) approach is used. Popularized by the IOM and most recently described by Nelson and colleagues,²⁴ a LHS is empowered by a dynamic information environment and capable of generating real-time learning, improvement, and research. LHS approaches include feed-forward data systems that can inform clinicians of patient needs ahead of clinical care encounters to optimize care, and feedback data about system performance and outcomes aggregated at system and population levels that can inform benchmarking for QI and research efforts simultaneously.²⁵

Systems-based QI approaches, such as lean/Six Sigma²⁶ and clinical microsystems²⁷ are currently used in some health care settings across the United States. Regional and national QI collaboratives using these improvement methodologies have demonstrated significant results. The Northern New England Cardiovascular Network Disease Study Group used a shared data registry and QI methods to reduce morbidity and mortality across cardiac surgery centers in the northeastern United States.²⁸ The Blue Cross Blue Shield of Michigan Program has established similar efforts across multiple conditions and populations.²⁹ On an even larger scale, more than 110 Cystic Fibrosis Foundation (CFF) centers have participated in national-level QI collaboratives that use a shared systems-level registry and QI methods. The CFF Learning and Leadership Collaboratives have reduced mortality, improved life expectancy, reduced morbidity, and improved a number of process quality indicators.^30

–33 Other prominent large-scale improvement collaborative efforts include: (1) the Vermont Oxford Network,³⁴ a network of more than 1300 hospitals working together to improve neonatal outcomes; (2) Improve Care Now, a multicenter collaborative focused on improving pediatric inflammatory bowel disease (IBD) care and outcomes,³⁵ and (3) IBD Qorus, a national multicenter improvement collaborative supported by the Crohn's and Colitis Foundation focused on improving adult IBD care.³⁶

Although large MS data registries such as NARCOMS and NARCRIMS have made important strides investigating MS care at the population level of analysis, they unfortunately do not focus on system-level aspects of MS care, and there has not yet been an investigation at the systems level regarding geographic variation, quality, and value of MS care. Additionally, there has not yet been a documented effort of this kind in MS, including the use of QI collaboratives, aimed at bettering MS-related performance quality, outcomes, or value. The Multiple Sclerosis Continuous Quality Improvement (MS-CQI) collaborative aims to make initial strides in this new area of study in MS care. Established in 2017, MS-CQI is the first multicenter improvement research collaborative for MS. MS-CQI uses a LHS model, modern QI methods, and a prospective, randomized research design to study and improve system-level performance and outcomes. This report describes baseline system-level variation in outcomes in the first year of the study to establish a rationale for systems-level QI intervention to improve MS care.

Methods

MS-CQI is a multicenter, prospective, stepped-wedge randomized study³⁷ designed to investigate system-level variation and test the comparative effects of QI intervention versus usual practice on performance quality and outcomes. The study is 3 years in duration with a 1-year baseline period followed by 2 randomization steps (QI intervention vs. usual care control), each occurring once annually. MS-CQI is a collaborative research network, composed of 4 participating MS care centers that provide de-identified data on MS outpatient care encounters abstracted from electronic health records (EHRs) to a data analytic center for management and analysis, which in turn is supervised by a central research oversight team. PwMS followed by MS-CQI centers also may consent to provide individual-level patient-reported outcomes data.

Setting and sample

Because the majority of MS care occurs in the outpatient setting, this study of system-level EHR-abstracted outpatient clinical data was conducted for the first year of the study (the baseline/preintervention “current state” period). All outpatient encounters for adults with MS ages 18 years or older presenting to any of the 4 centers participating in MS-CQI during the first year of the study (July 1, 2017-June 30, 2018) were included. Only 2 encounters with missing or incorrectly coded data were excluded.

Measures and data management

De-identified EHR data were abstracted by research coordinators from each of the 4 participating MS-CQI centers for all outpatient encounters, input into a secure RedCap database management system, and later extracted by the data analytics center and converted to SAS 9.4 (SAS Institute Inc., Cary, NC)³⁸ format for statistical analyses. Priority quality measures included proportion of patients on MS disease-modifying therapy (DMT), proportion of patients with at least 1 brain MRI scan, and proportion of patients experiencing a MS relapse (exacerbation). Priority outcome measures include the number of hospitalizations, ED utilization, all-cause hospitalizations, urgent care (UC) utilization, and MS exacerbations (relapses). Each encounter was stripped of patient identifying information and tracked by a unique study identification number.

Data analytic plan

Descriptive analyses were conducted, including frequency distributions for categorical variables and means for continuous variables reported for each center and the MS-CQI collaborative in aggregate. For categorical variables, P values were calculated using chi-square tests. For continuous variables, P values were calculated using analysis of variance. Significance was set at an α level of P < 0.05. The power analyses established a minimum β level of 0.80.

Stepwise binary logistic regression was used to investigate the influence of quality, utilization, and demographic variables on priority outcomes of MS relapses (exacerbations), DMT utilization, and brain MRI utilization.³⁹ All variables were initially included in the model, but those that were not significant were removed using a stepwise procedure. The model also includes centers as a group variable to investigate the association between system-level variation and outcomes. For this statistical analysis, a causal inference assumption is implied for the relapse outcome and utilization of health care services. For example, a likely causal directional relationship has been assumed that relapses cause ED visits. To ensure internal validity of the research and control for external influences, potential confounders were included in the statistical models such as demographics (age and sex), system (center), phenotype, and selected quality and utilization variables: ED visit, MRIs, hospitalization, DMT, and relapse.

One of the MS centers was designated to be the referent group for the logistic regression analyses using a “highest volume criteria,” as the largest contributor to the overall distribution is highly likely to be closest to that of the overall aggregate. The overall MS-CQI collaborative (aggregate) group could not be used as the referent group because it is not mutually exclusive from the other groups, each of which is a part of that overall (aggregated) group. Center A was selected here because it is the highest volume (largest N) center in MS-CQI and the proportion of the MS phenotype affecting a majority of PwMS (relapsing-remitting MS [RRMS]) is 82.9% compared to the MS-CQI overall, which is 80.9%. All analyses were performed using SAS 9.4.

Ethics review

The MS-CQI study was approved as a minimal risk research study by the Committee for Protection of Human Subjects (CPHS) at Dartmouth College.

Results

A total of 3111 outpatient clinical encounters were included in the analysis (Table 1). The 4 participating centers (A-D), represent varying contexts including: (1) an urban academic MS center; (2) a large urban private practice setting; (3) a rural academic MS center; and (4) a rural community hospital MS center. A total of 3111 unique PwMS were followed by MS-CQI in the first year of the study, and encounter volumes varied substantively across centers from a low of 471 in Center B to a high of 1053 in Center A. Demographic and clinical characteristics varied significantly (P < 0.05) across centers, including MS diagnosis type, age, and sex.

Table 1.

Characteristics of Participating Centers (N = 3111)

	Center A (rural hospital)	Center B (urban academic hospital)	Center C (rural academic center)	Center D (urban private practice)	MS-CQI (total)	P Value
Patients (n, %)	1053 (33.9)	471 (15.1)	863 (27.7)	724 (23.3)	3111 (100)
Age (mean, SD)	52.2 (12.0)	48.7 (13.2)	51.7 (12.8)	48.8 (12.8)	50.76 (12.7)	<0.0001
Female (n, %)	821 (78.0)	340 (72.2)	629 (72.9)	563 (77.8)	2353 (75.6)	<0.0001
MS Phenotype (n, %)						<0.0001
RRMS	873 (82.9)	365 (77.4)	613 (71.0)	665 (91.8)	2516 (80.9)
SPMS	99 (9.4)	47 (9.9)	147 (17.0)	29 (4.0)	322 (10.3)
PPMS	74 (7.0)	36 (7.6)	62 (7.2)	18 (2.5)	190 (6.1)
PRMS	0 (0)	0 (0)	0 (0)	9 (1.2)	9 (0.29)
Other	7 (0.66)	23 (4.8)	41 (4.7)	3 (0.42)	74 (2.4)

MS, multiple sclerosis; MS-CQI, Multiple Sclerosis Continuous Quality Improvement research collaborative; PPMS, primary progressive MS; PRMS, progressive relapsing MS; RRMS, relapsing remitting MS; SD, standard deviation; SPMS, secondary progressive MS.

System-level variation

Univariate analyses identified numerous differences in center-level outcomes (Table 2). There were significant system-level differences across centers for DMT, brain MRI, cervical MRI, and thoracic MRI (P < 0.01). There were significant differences across centers for MS exacerbations (relapses) (P < 0.01), corresponding to an overall annualized relapse rate of approximately 0.2. Annualized rates varied significantly across MS centers (P < 0.01). The proportion of patients with at least 1 episode of acute care utilization (UC visits, ED visits, or hospitalizations) for MS-CQI were relatively low overall (UC = 3.6%, ED = 9.9%, and hospitalizations = 9.7%), and significant system-level variation in performance was observed here as well. Utilization in these categories also was calculated in terms of absolute utilization rates, which also demonstrated statistically significant variation (P < 0.01) across centers.

Table 2.

Center-Level Variation in Outcomes (Year 1 Baseline)

	Center A (n = 1053)	Center B (n = 471)	Center C (n = 863)	Center D (n = 724)	MS-CQI (N = 3111)	P Value
Patients on at least 1 DMT (n,%)	783 (74.3)	347 (73.7)	482 (55.9)	547 (75.6)	2159 (69.4)	<0.0001
Brain MRI (n,%)	649 (61.6)	298 (63.3)	400 (46.3)	371 (51.2)	1718 (55.2)	<0.0001
C-Spine MRI (n,%)	419 (39.8)	108 (22.9)	168 (19.5)	185 (25.5)	880 (28.3)	<0.0001
T-Spine MRI (n,%)	205 (19.5)	84 (17.9)	65 (7.5)	127 (17.5)	481 (15.5)	<0.0001
Hospitalizations (n,%)
None	966 (91.7)	438 (93.0)	756 (87.6)	652 (90.0)	2812 (90.4)
1	59 (5.6)	32 (6.8)	82 (9.5)	62 (8.6)	235 (7.6)
≥2	28 (2.7)	1 (0.21)	25 (2.9)	10 (1.4)	64 (2.1)
Mean, SD	0.13 (0.56)	0.07 (0.27)	0.19 (0.74)	0.12 (0.37)	0.13 (0.55)	0.0018
ED visits (n,%)
None	927 (88.0)	415 (88.1)	778 (90.2)	683 (94.3)	2803 (90.1)
1	74 (7.0)	46 (9.8)	56 (6.5)	40 (5.5)	216 (6.9)
≥2	52 (4.9)	10 (2.1)	29 (3.4)	1 (0.14)	92 (3.0)
Mean, SD	0.22 (0.82)	0.15 (0.43)	0.18 (0.85)	0.06 (0.49)	0.16 (0.69)	<0.0001
UC visits (n,%)
None	1012 (96.1)	454 (96.4)	820 (95.0)	715 (98.8)	3001 (96.4)
1	36 (3.4)	11 (2.3)	31 (3.6)	8 (1.1)	86 (2.8)
≥2	5 (0.47)	6 (1.3)	12 (1.4)	1 (0.14)	24 (0.77)
Mean, SD	0.05 (0.28)	0.05 (0.30)	0.07 (0.35)	0.01 (0.13)	0.05 (0.30)	0.0011
Relapses (n,%)
None	935 (88.8)	407 (86.4)	823 (95.4)	603 (83.3)	2768 (89.0)
1	74 (7.0)	56 (11.9)	35 (4.06)	69 (9.5)	234 (7.5)
≥2	44 (4.2)	8 (1.7)	5 (0.58)	52 (7.2)	109 (3.5)
Mean, SD	0.19 (0.67)	0.16 (0.42)	0.05 (0.25)	0.30 (0.80)	0.17 (0.59)	<0.0001

DMT, disease-modifying therapy; ED, emergency department; MS-CQI, Multiple Sclerosis Continuous Quality Improvement research collaborative; MRI, magnetic resonance imaging; SD, standard deviation; UC, urgent care.

Adjusted models predicting associations with priority outcomes

Stepwise logistic regression analyses were conducted controlling for potential confounders to identify site-level effects on outcomes for 3 priority outcomes: (1) DMT utilization; (2) brain MRI utilization; and (3) relapses (Table 3). Goodness of fit (C statistic) levels were greater than 0.70 for all models reported, with a majority achieving 0.8 or higher. C values of 0.7 to 0.8 indicate acceptable discrimination and values of 0.8 or above indicate excellent discrimination.³⁹

Table 3.

Adjusted Center-Level Variation in Relapses

	Odds ratio (95% confidence interval)	P Value
All MS Types (n = 3108)
Age^*	0.959 (0.949–0.968)	<0.0001
Center B vs A	1.345 (0.943–1.917)	0.1017
Center C vs A	0.509 (0.343–0.756)	0.0008
Center D vs A	1.871 (1.382–2.534)	<0.0001
At least 1 ED visit in baseline year (Yes/No)	2.395 (1.669–3.436)	<0.0001
At least 1 DMT in baseline year (Yes/No)	2.034 (1.458–2.837)	<0.0001
At least 1 hospitalization in baseline year (Yes/No)	1.537 (1.047–2.257)	0.0282
At least 1 b-MRI in baseline year (Yes/No)	1.406 (1.017–1.944)	0.0393
At least 1 t-MRI in baseline year (Yes/No)	1.644 (1.192–2.267)	0.0024
At least 1 c-MRI in baseline year (Yes/No)	2.014 (1.449–2.799)	<0.0001
RRMS Only (n = 2516)
Age^*	0.961 (0.95–0.972)	<0.0001
Center B vs A	1.628 (1.116–2.377)	0.0115
Center C vs A	0.574 (0.369–0.892)	0.0136
Center D vs A	1.959 (1.426–2.692)	<0.0001
At least 1 ED visit in baseline year (Yes/No)	2.233 (1.509–3.305)	<0.0001
At least 1 DMT in baseline year (Yes/No)	2.017 (1.418–2.87)	<0.0001
At least 1 hospitalization in baseline year (Yes/No)	1.762 (1.173–2.646)	0.0064
At least 1 t-MRI in baseline year (Yes/No)	1.634 (1.157–2.308)	0.0054
At least 1 c-MRI in baseline year (Yes/No)	2.203 (1.592–3.049)	<0.0001
Non-RRMS (n = 592)
Age^*	0.948 (0.918–0.98)	0.0014
Center B vs A	0.492 (0.164–1.473)	0.2049
Center C vs A	0.24 (0.091–0.629)	0.0037
Center D vs A	0.883 (0.236–3.303)	0.8528
At least 1 ED visit in baseline year (Yes/No)	3.587 (1.546–8.322)	0.0029
At least 1 DMT in baseline year (Yes/No)	2.994 (1.148–7.812)	0.025
At least 1 c-MRI in baseline year (Yes/No)	4.94 (2.238–10.904)	<0.0001

Center A: rural hospital; Center B: urban academic hospital; Center C: rural academic center; and Center D: urban private practice.

Odds of relapse decrease by X% for each year of increasing age.

ED, emergency department; DMT, disease-modifying therapy; MRI, magnetic resonance imaging; MS, multiple sclerosis; RRMS, relapsing remitting MS.

Relapses

Significant system-level variation in relapses was observed in adjusted analyses (Table 3). Having ≥1 ED visits, being on DMT, having ≥1 hospitalization, receiving a thoracic MRI, a cervical MRI, or a brain MRI in the baseline year were all associated with increased odds of a relapse. Increasing age was associated with decreased odds of a relapse (P < 0.0001). Compared to Center A (referent), results showed greater risk of relapse at Center D (P < 0.0001), and comparatively decreased odds of a relapse at Center C (P ≤ 0.001). Analyses limited to patients with RRMS yielded similar results. In persons with other forms of MS (non-RRMS), including secondary progressive MS (SPMS), having ≥1 ED visits, being on DMT, or receiving a cervical MRI in the baseline year was associated with increased risk of a relapse. Increasing age was associated with decreased odds of relapse (P < 0.01).

DMT utilization

System-level variation was observed in DMT utilization (Table 4) in all MS types after controlling for potential confounders and modifiers including age, sex, and MS phenotype (RRMS vs. non-RRMS). Factors associated with increased odds of DMT included ≥1 brain MRI in the baseline year and ≥1 relapse. Factors that significantly reduced odds of DMT utilization included ≥1 thoracic MRI, female sex, and increasing age. Center A was the reference group (highest volume). In comparison to patients at Center A (referent), those at Center C had decreased odds of receiving a DMT.

Table 4.

Adjusted Center Level Variation in Disease-Modifying Therapy Utilization

	Odds ratio	P value
	(95% confidence interval)	P value
All MS Types (n = 3108)
Age ^*	0.986 (0.980–0.993)	<0.0001
Female sex	0.824 (0.680–0.998)	0.048
Center B vs A	0.865 (0.667–1.123)	0.2762
Center C vs A	0.473 (0.385–0.581)	<0.0001
Center D vs A	1.12 (0.889–1.410)	0.3375
At least 1 relapse in baseline year (Yes/No)	1.686 (1.221–2.329)	0.0015
At least 1 brain MRI in baseline year (Yes/No)	3.565 (2.985–4.257)	<0.0001
At least 1 t-MRI in baseline year (Yes/No)	0.639 (0.495–0.826)	0.0006
RRMS Only (n = 2516)
Center B vs A	0.763 (0.562–1.037)	0.0837
Center C vs A	0.406 (0.319–0.518)	<0.0001
Center D vs A	0.986 (0.763–1.273)	0.9115
At least 1 ED visit in baseline year (Yes/No)	1.625 (1.149–2.297)	0.006
At least 1 hospitalization in baseline year (Yes/No)	0.721 (0.522–0.994)	0.0461
At least 1 brain MRI in baseline year (Yes/No)	3.722 (3.033–4.567)	<0.0001
At least 1 t-MRI in baseline year (Yes/No)	0.649 (0.486–0.867)	0.0034
Non-RRMS (n = 592)
Age^*	0.954 (0.938–0.970)	<0.0001
Female sex	0.641 (0.436–0.944)	0.0242
Center B vs A	1.979 (1.163–3.368)	0.0118
Center C vs A	0.882 (0.570–1.364)	0.5731
Center D vs A	1.409 (0.734–2.703)	0.3028
At least 1 relapse in baseline year (Yes/No)	2.910 (1.113–7.612)	0.0294
At least 1 brain MRI in baseline year (Yes/No)	2.640 (1.833–3.800)	<0.0001

Center A: rural hospital; Center B: urban academic hospital; Center C: rural academic center; and Center D: urban private practice.

Odds of relapse decrease by X% for each year of increasing age.

ED, emergency department; MRI, magnetic resonance imaging; MS, multiple sclerosis; RRMS, relapsing remitting MS.

Significant system-level variation in DMT utilization also was observed in stratified analyses of persons with RRMS. Factors associated with increased odds of DMT included ≥1 brain MRI in the baseline year and ≥1 relapse in the past year. Utilization of ≥1 thoracic MRI and ≥1 hospitalization significantly decreased odds of receiving a DMT. Compared to Center A (referent), there were decreased odds of receiving a DMT at Center C.

Similar analyses in patients with other forms of MS (primary progressive MS [PPMS] or SPMS) found significant system-level variation. Experiencing ≥1 relapse and using ≥1 brain MRI were both associated with increased odds of DMT utilization. Factors that significantly decreased odds of DMT utilization included female sex and increasing age. Compared to Center A (referent), greater odds of receiving a DMT at Center B were observed.

Brain MRI utilization

System-level variation was observed in brain MRI utilization (Table 5) in patients with all MS types after controlling for potential confounders. Having ≥1 relapses, being on DMT treatment, having an ED visit, and having ≥1 hospitalization increased the odds of receiving a brain MRI. Compared to Center A (referent), there were decreased odds of receiving a brain MRI at Center C and Center D. Similar results were observed limiting to persons with RRMS. Compared to Center A (referent), there were decreased odds of receiving a brain MRI at Centers C and D. In persons with PPMS and SPMS, having >1 relapses, being on DMT, or >1 ED visits increased odds of brain MRI utilization. Increasing age was associated with decreased odds of brain MRI utilization.

Table 5.

Adjusted Center-Level Variation in Brain Magnetic Resonance Imaging Utilization

	Odds ratio	P value
	(95% confidence interval)	P value
All MS Types (n = 3108)
Age ^*	0.977 (0.971–0.983)	<0.0001
Center B vs A	0.984 (0.773–1.252)	0.8948
Center C vs A	0.644 (0.528–0.785)	<0.0001
Center D vs A	0.558 (0.454–0.686)	<0.0001
At least 1 relapse in baseline year (Yes/No)	2.230 (1.678–2.964)	<0.0001
At least 1 DMT in baseline year (Yes/No)	3.294 (2.783–3.899)	<0.0001
At least 1 hospitalization in baseline year (Yes/No)	1.474 (1.090–1.992)	0.0116
At least 1 ED visit in baseline year (Yes/No)	1.925 (1.412–2.625)	<0.0001
RRMS Only (n = 2516)
Age^*	0.979 (0.972–0.986)	<0.0001
Center B vs A	1.105 (0.836–1.459)	0.4826
Center C vs A	0.628 (0.500–0.789)	<0.0001
Center D vs A	0.561 (0.450–0.699)	<0.0001
At least 1 relapse in baseline year (Yes/No)	2.062 (1.525–2.789)	<0.0001
At least 1 DMT in baseline year (Yes/No)	3.409 (2.806–4.413)	<0.0001
At least 1 hospitalization in baseline year (Yes/No)	1.615 (1.142–2.283)	0.0067
At least 1 ED visit in baseline year (Yes/No)	1.963 (1.369–2.815)	0.0002
Non-RRMS (n = 592)
Age^*	0.983 (0.968–0.998)	0.0251
At least 1 relapse in baseline year (Yes/No)	3.633 (1.515–8.713)	0.0039
At least 1 DMT in baseline year (Yes/No)	2.558 (1.787–3.663)	<0.0001
At least 1 ED visit in baseline year (Yes/No)	2.322 (1.356–3.974)	0.0021

Center A: rural hospital; Center B: urban academic hospital; Center C: rural academic center; and Center D: urban private practice.

Odds of relapse decrease by X% for each year of increasing age.

DMT, disease-modifying therapy; ED, emergency department; MS, multiple sclerosis; RRMS, relapsing remitting MS.

Discussion

This study found significant system-level variation in baseline systems-level outcomes across MS centers in the first year (baseline/preintervention period) of the MS-CQI study. Adjusting for individual-level factors (age, sex, and MS phenotype), significant center-level effects on MS care outcomes persisted for DMT utilization, MRI utilization, and relapses, suggesting that system-level effects are substantively influencing these outcomes in their own right.

Individual level-factors

Expected associations were observed between outcomes and demographic factors. Increasing age demonstrated a small but significant association with reduced odds of relapse overall, and for RRMS and non-RRMS subgroups alone.^40

–45 This also was observed for MRI, which also would be expected to be used more frequently in younger persons with MS. Increasing age was significantly associated with decreased DMT utilization overall and in persons with PPMS and SPMS, but not for individuals with RRMS. This aligns with the expectation that younger persons and persons with RRMS are more likely to be on DMT.^46

–50 Beginning treatment at an early age also is important because treatment is most effective longitudinally when started soon after diagnosis. Gross and Watson reported a significant difference in reported DMT use between individuals with SPMS and persons with RRMS.⁴⁸ Sex did not have any substantial influence on relapses, DMT utilization, or MRI utilization in RRMS, but it did overall and in the non-RRMS subgroup. Females were 18% less likely to be on DMT overall (P < 0.05) and women with PPMS or SPMS were 36% less likely to be on DMT (P < 005).

Perhaps the most interesting individual-level finding in this study was that DMT utilization was associated with significantly higher odds of relapse, which held overall and for RRMS and non-RRMS subtypes. This observation runs counter to an overwhelming trend established in the literature supporting a relationship between DMT treatment and decreased relapse rates.^49

–55 In contrast, the expected associations between DMT use and imaging are intuitive and DMT utilization was not generally associated with increased hospitalizations or UC utilization, except in the RRMS subgroup, which had higher odds of ED utilization. Thomas et al demonstrated that DMT treatment was associated with decreased ED⁵⁶ and Burks et al reported that DMT adherence was associated with a 42% reduction in relapses, 52% reduction in hospitalizations, and 38% reduction in emergency care visits.⁵⁷ Unlike these, the present study accounted for system-level effects and did not replicate these results.

PwMS experiencing relapses were more likely to have higher MRI utilization overall and in those with RRMS, likely associated with an increased and appropriate utilization of imaging to assess for radiographic progression and guide treatment decisions related to relapse events. This finding corresponds with Consortium of MS Centers guidelines, which recommend the use of MRI for the initial diagnosis of MS and observation of changes and/or progression over time.⁵⁹ MRI also was significantly associated with DMT treatment, suggesting appropriate use of MRI for monitoring of DMT treatment. However, the rate of utilization of MRI utilization varied greatly across centers, from a low of 46.3% in Center C to a high of 63% in Center B (Table 2). Finally, relapses were associated with substantially higher odds of costly UC utilization, including ED and hospitalizations, and this association held overall and in stratified analyses of RRMS and non-RRMS subgroups. These findings align with recent research associating disease severity, relapses, and health care system characteristics with utilization and cost.^58

–61

It is suspected that the cross-sectional design of this analysis, which was intended primarily for establishing system-level variation differences at baseline and not for establishing longitudinal causal relationships between DMT treatment and outcomes, likely contributes to the DMT relapse relationship finding here. Specifically, the established association between DMT treatment and relapses is likely more representative of a relationship between people on DMT being more likely to have relapsing forms of MS than DMT causing increased relapse rates, or that PwMS with higher relapse frequency may be more strongly encouraged to be on DMT or to change from an ineffective DMT to a new DMT.

System-level variation demonstrated in adjusted analyses

The chief contribution of this study is the provision of initial evidence that substantial system-level variation effects influence MS population health outcomes and that these effects persist after controlling for significant individual-level factors (eg, age, sex, MS subtype) that are known to influence these outcomes. Compared to the referent (highest volume) center, a community hospital-based MS center (Table 3), PwMS followed by Center D (an urban large private practice) had nearly 2-fold higher odds of relapse overall (odds ratio [OR] = 1.87, P < 0.0001), and in the RRMS subgroup (OR = 1.96, P < 0.0001). Center C (a rural academic center), demonstrated nearly 50% lower odds of relapse overall (OR = 0.51, P < 0.001) and in RRMS (OR = 0.58, P < 0.05) and non-RRMS subgroups (OR = 0.24, P < 0.01). Center C also demonstrated significantly lower odds of MRI utilization overall and in the RRMS subgroup (Table 4), and over 50% lower odds of DMT utilization overall (OR = 0.47, P < 0.0001) and in stratified analysis of their RRMS subgroup (OR = 0.41, P < 0.0001). The proportion of PwMS with RRMS differed significantly across MS-CQI sites (Table 1), which could influence the overall population findings, but not the stratified RRMS group findings.

These findings have important improvement and research implications. First, establishing that small-area variation effects exist in MS care, just as have been previously observed in many other conditions, supports a rationale for the ongoing system-level study and improvement of population health outcomes. Second, the study of system-level variation can be used to benchmark performance and identify top performers and best practices at specific MS centers, which can then be studied, shared, and replicated throughout MS-CQI using a learning collaborative mechanism. Finally, the discovery of areas of substantial variation across centers, such as those observed here for relapses and DMT utilization in the RRMS population, can identify important potential targets for focused research and improvement efforts to address.

Limitations

The most substantial limitation of this study is that MS-CQI is designed as a QI research study, and not as a generalizable population-level epidemiological study. The cross-sectional design of this analysis of baseline current state findings limits the inference to associations. Causation or directional relationships cannot be inferred here, and the analysis is more vulnerable to confounding than longitudinal designs.

Finally, although MS-CQI does not include all important measures, and some known effect modifiers are missing, including Extended Disability Status Scale, Patient-Determined Disease Steps, objective measures of cognition, functional assessments such as the MS Functional Composite, and patient-reported measures of fatigue and depression.^62

–66 These are potential unmeasured confounders or effect modifiers. However, although these measures are often used in research settings, they are not always standard of care in clinical practice and are not captured uniformly. For this reason, they were not included in the clinical (EHR) data set in the real-world design of this study.

Conclusion

MS-CQI is the first improvement research study investigating system-level effects on MS population health outcomes. This study provides initial evidence of system-level effects on important MS population health outcomes such as relapses, MRI utilization, DMT utilization, and UC utilization in a real-world multicenter prospective study. Establishing that system-level variation is present, and localizing where that variation is occurring, supports a developing rationale for a continued systems-level improvement and research agenda in MS and can inform the selection of priority improvement targets for the next steps in that journey.

Footnotes

Authors' contributions

Dr. Oliver led and oversaw all aspects of study design, conduct, data analysis, and manuscript preparation. Dr. Walsh served as the lead data analyst and data custodian for the research database, oversaw the data analytic plan, data analyses, and participated in the interpretation of results and preparation of the manuscript. Mr. Messier served as a study co-investigator and improvement coach and participated in reviews and revisions of the manuscript. Ms. Mehta served as the project manager for the MS-CQI Collaborative and participated in manuscript reviews. Dr. Cabot, Ms. Pagnotta, Dr. Klawiter, and Dr. Solomon served as site investigators at participating MS centers and participated in reviews of study results and reviews and revisions of the manuscript. Dr. England participated as a scientific liaison and participated in reviews of the manuscript.

Acknowledgments

The MS-CQI research collaborative is an improvement and research community of practice comprised of dedicated researchers, clinicians, coordinators and PwMS. Findings disseminated from the MS-CQI study are authored acknowledging the combined efforts of the larger community of MS-CQI Investigators, which we wish to formally acknowledge below:

Dartmouth (research leadership team, Hanover, NH): Brant Oliver (principal investigator), Fal Mehta (project manager, 2019–2020), Randal Messier (improvement coach and co-investigator), Cathy Alexander (co-investigator), Hasna Hakim (co-investigator), Mary Smith (research assistant), Chandlee Bryan (research associate), Lucy Fennesy (research associate), Amy Hall (project manager, 2017–2019), Troi Perkins (research assistant, 2017–2019), and James Curtin (database administrator).

Jefferson College of Population Health (data analytics center, Philadelphia, PA): Karen Walsh (chief data analyst), Dexter Waters (AHEOR fellow, 2019–2020), Laetitia N'Dri (AHEOR fellow, 2019–2020), Arianna Kee (AHEOR fellow, 2019), Albert Crawford (researcher, 2019), and Marianna LaNoue (Director of Research)

Massachusetts General Hospital MS Clinic (Boston, MA): Eric Klawitter (neurologist/site investigator), Anna Vaeth (study coordinator)

University of Vermont Medical Center MS Center (Burlington, VT): Andrew Solomon (neurologist/site investigator), Emily Azalone (study coordinator)

Concord Hospital Neurology MS Specialty Care Program (Concord, NH): Ann Cabot (neurologist/site investigator), Rick Lavallee (study coordinator, IT administrator).

MS Center of Greater Orlando (Maitland, FL): Tricia Pagnotta (MS specialist nurse practitioner/site investigator), Kelly Holley, RN (study coordinator)

MS-CQI Research & Improvement Advisory Committee (RIAC): The RIAC meets twice annually to advise and monitor the progress of the MS-CQI collaborative. The authors wish to thank the RIAC members for their important contributions and guidance: Heather Wishart (RIAC chair, MS neuropsychologist, Dartmouth-Hitchcock Health), Cy Jordan (MD), Randy Messier (MT, MSA, PCMH-CCE), Dean Lea (PhD), Chris Rovinski-Wagner (MSN, APRN), Ann Cabot (DO), Elizabeth R. McLure (PwMS), Alex Hoyt (PhD, RN), Jody Karp (PwMS), Steven Triedman (PwMS)

The authors also would like to thank colleagues at the Department of Community and Family Medicine at Dartmouth-Hitchcock for the following specific contributions to this work: (1) Peter DiMillia, MPH, for key assistance in the preparation of the tables; and (2) Tiffany D'cruze, BA, for key assistance with library research, literature reviews and contributions to the narrative in the section of the manuscript.

Data Statement

The database for this study is de-identified and held via a secure RedCap repository by the Dartmouth research lead site and also locally at the Jefferson College of Population Health data analytics center per institutional review board-approved protocol and will be maintained securely for a period of 5 years post study completion or longer as required by regulatory policies. This data set is not publicly available. Inquiries concerning the research data set used in this study should be directed to the MSCQI principal investigator (Dr. Oliver).

Author Disclosure Statement

The authors declare that there are no conflicts of interest.

Funding Information

The MS-CQI research study was supported by a 3-year research grant from Biogen (2017–2020).

References

Wallin

, Culpepper

, Campbell

, et al. The prevalence of MS in the United States: A population-based estimate using health claims data. Neurology, 2019; 92:e1029–e1040.

Hartung

, Bourdette

, Ahmed

, Whitham

. The cost of multiple sclerosis drugs in the US and the pharmaceutical industry: Too big to fail? Neurology 2015;84:2185–2192.

Adelman

, Rane

, Villa

. The cost burden of multiple sclerosis in the United States: a systematic review of the literature. J Med Econ, 2013; 16:639–647.

Aitken

, Klienrock

Medicines use and spending in the U.S.: A review of 2016 and outlook to 2021. Available: https://www.iqvia.com/insights/the-iqvia-institute/reports/medicines-use-and-spending-in-the-us-a-review-of-2016. Accessed June 10, 2021 .

Owens

GM.

Economic burden of multiple sclerosis and the role of managed sare organizations in multiple sclerosis management. Am J Manag Care, 2016; 22(6 Suppl):s151–158.

Zimmermann

, Brouwer

, Tice

, et al. Disease-modifying therapies for relapsing-remitting and primary progressive multiple sclerosis: a cost-utility analysis. CNS Drugs, 2018; 32:1145–1157.

Pyenson

, Fredericks

, Berrios

, Mastroianni

, Han

. Multiple sclerosis: new perspectives on the patient journey. Milliman

Inc

. 2016:40. https://us.milliman.com/en/insight/multiple-sclerosis-new-perspectives-on-the-patient-journey-2019-update# Accessed June 7, 2021 .

Home Page: ABOUT. NARCOMS Registry for Multiple Sclerosis. https://www.narcoms.org/ Accessed November 12, 2020 .

Our Motivation; Our Approach; Our

Responsibility

. MS

PATHS

. www.mspaths.com Accessed November 12, 2020 .

10.

Gisler

. EMD serono launches

MS-LINK research network to improve patient care

. 2019. https://multiplesclerosisnewstoday.com/news-posts/2019/05/07/emd-serono-launches-ms-link-a-community-to-advance-ms-science-and-improve-patient-lives/ Accessed November 12, 2020 .

11.

Home Page | NARCRMS. NARCRMS North American registry for care and research in multiple sclerosis. www.narcrms.org/ Accessed November 12, 2020 .

12.

Institute of Medicine (US) Committee on Quality of Health Care in

America

. Crossing the quality chasm: a new health system for the 21st century. National Academies Press (US), 2001. www.ncbi.nlm.nih.gov/books/NBK222274/ Accessed November 13, 2020 .

13.

The IHI Triple Aim | IHI—Institute for Healthcare Improvement. www.ihi.org/Engage/Initiatives/TripleAim/Pages/default.aspx Accessed November 13, 2020 .

14.

Wennberg

, Gittelsohn null. Small area variations in health care delivery. Science, 1973; 182:1102–1108.

15.

Trustees of Dartmouth College. The Dartmouth atlas of health care. 2015. https://www.dartmouthatlas.org/ Accessed November 13, 2020 .

16.

Welch

, Miller

, Welch

, Fisher

, Wennberg

. Geographic variation in expenditures for physicians' services in the United States. N Engl J Med, 1993; 328:621–627.

17.

Fisher

, Wennberg

. Health care quality, geographic variations, and the challenge of supply-sensitive care. Perspect Biol Med, 2003; 46:69–79.

18.

The Affordable Care Act. U.S. Department of Health and Human Services. Published 2013. https://www.hhs.gov/healthcare/about-the-aca/index.html Accessed November 13, 2020 .

19.

Cheng

, Crandall

, Bever

, et al. Quality indicators for multiple sclerosis. Mult Scler, 2010; 16:970–980.

20.

American Academy of Neurology. Multiple Sclerosis Quality Measures. 2014. https://www.aan.com/siteassets/home-page/policy-and-guidelines/quality/quality-measures/14msmeasureset_pg.pdf Accessed June 7, 2021 .

21.

Traboulsee

, Simon

, Stone

, et al. Revised recommendations of the consortium of MS centers task force for a standardized mri protocol and clinical guidelines for the diagnosis and follow-up of multiple sclerosis. AJNR Am J Neuroradiol, 2016; 37:394–401.

22.

Nelson

, Batalden

, Godfrey

, Lazar

. Value by design: developing clinical microsystems to achieve organizational excellence. New York: Wiley, 2011. https://www.wiley.com/en-us/Value+by+Design%3A+Developing+Clinical+Microsystems+to+Achieve+Organizational+Excellence-p-9780470385340 Accessed November 13, 2020 .

23.

Nelson

, Greenfield

, Hays

, Larson

, Leopold

, Batalden

. Comparing outcomes and charges for patients with acute myocardial infarction in three community hospitals: an approach for assessing “value”. Int J Qual Saf Healthcare, 1995; 7:95–108.

24.

Nelson

, Dixon-Woods

, Batalden

, et al. Patient focused registries can improve health, care, and science. BMJ, 2016; 354:i3319.

25.

Oliver

, Nelson

, Kerrigan

. Turning feed-forward and feedback processes on patient-reported data into intelligent action and informed decision-making: case studies and principles. Med Care, 2019; 57 Suppl 5 Suppl 1:S31–S37.

26.

Lean Six Sigma. The Power of Lean Six Sigma. 2014. http://engineering.dartmouth.edu/sixsigma/ Accessed November 13, 2020 .

27.

Institute for Excellence in Health and Social Systems in the College of Health and Human Services. Clinical Microsystems at University of New Hampshire. 2020. https://clinicalmicrosystem.org/ Accessed November 13, 2020 .

28.

Published Literature. Northern New England Cardiovascular Network Disease Study Group. 2014. www.nnecdsg.org/pub_lit_2.htm Accessed November 13, 2020 .

29.

Blue Cross Blue Shield of Michigan Program. https://www.valuepartnerships.com/programs/collaborative-quality-initiatives/ Accessed April 13, 2021 .

30.

Godfrey

, Oliver

. Accelerating the rate of improvement in cystic fibrosis care: contributions and insights of the learning and leadership collaborative. BMJ Qual Saf, 2014; 23:i23–i32.

31.

Marshall

, Nelson

. Accelerating implementation of biomedical research advances: critical elements of a successful 10 year Cystic Fibrosis Foundation healthcare delivery improvement initiative. BMJ Qual Saf, 2014; 23(Suppl 1):i95–i103.

32.

Mogayzel

, Dunitz

, Marrow

, Hazle

. Improving chronic care delivery and outcomes: the impact of the cystic fibrosis Care Center Network. BMJ Qual Saf, 2014; 23 Suppl 1:i3–8.

33.

Sabadosa

, Batalden

. The interdependent roles of patients, families and professionals in cystic fibrosis: a system for the coproduction of healthcare and its improvement. BMJ Qual Saf, 2014; 23 Suppl 1:i90–94.

34.

Home - Vermont Oxford Network

VON.

https://public.vtoxford.org/ Accessed November 13, 2020 .

35.

ImproveCareNow. https://www.improvecarenow.org/ Accessed November 13, 2020 .

36.

Quality of Care: IBD Qorus | Crohn's & Colitis Foundation. https://www.crohnscolitisfoundation.org/research/ibd-qorus Accessed November 13, 2020 .

37.

Brown

, Lilford

. The stepped wedge trial design: a systematic review. BMC Med Res Methodol, 2006; 6:54.

38.

SAS Viya |

SAS.

https://www.sas.com/en_us/software/viya.html Accessed November 13, 2020 .

39.

Hosmer

, Lemeshow

. Applied logistic regression, second edition. New York: Wiley & Sons, Inc., 2000. DOI:10.1002/0471722146.

40.

Tremlett

, Zhao

, Joseph

, Devonshire

, the UBCMS Clinic

Neurologists

. Relapses in multiple sclerosis are age- and time-dependent. J Neurol Neurosurgery Psychiatry, 2008; 79:1368–1374.

41.

Nazareth

, Rava

, Polyakov

, et al. Relapse prevalence, symptoms, and health care engagement: patient insights from the Multiple Sclerosis in America 2017 survey. Mult Scler Relat Disord, 2018; 26:219–234.

42.

Goodin

, Frohman

, Garmany

, et al. Disease modifying therapies in multiple sclerosis: report of the therapeutics and technology assessment subcommittee of the American Academy of Neurology and the MS Council for Clinical Practice Guidelines. Neurology, 2002; 58:169–78.

43.

Lublin

, Reingold SC; National Multiple Sclerosis Society (USA) Advisory Committee on Clinical Trials of New Agents in Multiple

Sclerosis

. Defining the clinical course of multiple sclerosis: results of an international survey. Neurology, 1996; 46:907–911.

44.

Confavreux

, Vukusic

, Moreau

, Adeleine

. Relapses and progression of disability in multiple sclerosis. N Engl J Med, 2000; 343:1430–1438.

45.

Steinman

Immunology of relapse and remission in multiple sclerosis. Annu Rev Immunol, 2014; 32:257–281.

46.

Kister

, Bacon

, Chamot

, et al. Natural history of multiple sclerosis symptoms. Int J MS Care, 2013; 15:146–158.

47.

Granqvist

, Boremalm

, Poorghobad

, et al. Comparative effectiveness of rituximab and other initial treatment choices for multiple sclerosis. JAMA Neurol, 2018; 75:320–327.

48.

Gross

, Watson

. Characteristics, burden of illness, and physical functioning of patients with relapsing-remitting and secondary progressive multiple sclerosis: a cross-sectional US survey. Neuropsychiatr Dis Treat, 2017; 13:1349–1357.

49.

Kapica-Topczewska

, Collin

, Tarasiuk

, et al. Clinical and epidemiological characteristics of multiple sclerosis patients receiving disease-modifying treatment in Poland. Neurol Neurochir Pol, 2020; 54:161–168.

50.

Hua

, Harris

, Conway

, Thompson

. Changes in patient-reported outcomes between continuers and discontinuers of disease modifying therapy in patients with multiple sclerosis over age 60. Mult Scler Relat Disord, 2019; 30:252–256.

51.

Cook

, Dhib-Jalbut

, Dowling

, et al. Use of magnetic resonance imaging as well as clinical disease activity in the clinical classification of multiple sclerosis and assessment of its course: a report from an International CMSC Consensus Conference, March 5–7, 2010. Int J MS Care, 2012; 14:105–114.

52.

Hart

, Bainbridge

. Current and emerging treatment of multiple sclerosis. Am J Manag Care, 2016; 22(6 Suppl):s159–170.

53.

Wingerchuk

, Carter

. Multiple sclerosis: current and emerging disease-modifying therapies and treatment strategies. Mayo Clin Proc, 2014; 89:225–240.

54.

National Multiple Sclerosis Society. Disease Modifying Therapies for Multiple Sclerosis. https://www.nationalmssociety.org/For-Professionals/Clinical-Care/Managing-MS/Disease-Modification Accessed June 7, 2021 .

55.

Cohen

, Coyle

, Leist

, Oleen-Burkey

, Schwartz

, Zwibel

. Therapy optimization in multiple sclerosis: a cohort study of therapy adherence and risk of relapse. Mult Scler Relat Disord, 2015; 4:75–82.

56.

Thomas

, Curkendall

, Farr

, Yu

, Hurley

. The impact of persistence with therapy on inpatient admissions and emergency room visits in the US among patients with multiple sclerosis. J Med Econ, 2016; 19:497–505.

57.

Burks

, Marshall

, Ye

. Adherence to disease-modifying therapies and its impact on relapse, health resource utilization, and costs among patients with multiple sclerosis. Clinicoecon Outcomes Res, 2017; 9:251–260.

58.

Nicholas

, Boster

, Wu

, et al. Comparison of disease-modifying therapies for the management of multiple sclerosis: analysis of healthcare resource utilization and relapse rates from us insurance claims data. Pharmacoecon Open, 2018; 2:31–41.

59.

Kobelt

, Thompson

, Berg

, et al. New insights into the burden and costs of multiple sclerosis in Europe. Mult Scler, 2017; 23:1123–1136.

60.

Bruno

, Marc

, Ouarda

, et al. Economic burden of multiple sclerosis in France estimated from a regional medical registry and national sick fund claims. Mult Scler Relat Disord, 2019; 36:101396.

61.

Boster

, Nicholas

, Wu

, et al. Comparative effectiveness research of disease-modifying therapies for the management of multiple sclerosis: analysis of a large health insurance claims database. Neurol Ther, 2017; 6:91–102.

62.

Meyer-Moock

, Feng

Y-S

, Maeurer

, Dippel

F-W

, Kohlmann

. Systematic literature review and validity evaluation of the expanded disability status scale (EDSS) and the multiple sclerosis functional composite (MSFC) in patients with multiple sclerosis. BMC Neurol, 2014; 14:58.

63.

Højsgaard Chow

, Schreiber

, Magyari

, et al. Progressive multiple sclerosis, cognitive function, and quality of life. Brain Behav, 2018; 8:e00875.

64.

Ayache

, Chalah

. Fatigue in multiple sclerosis—insights into evaluation and management. Neurophysiol Clin, 2017; 47:139–171.

65.

Fisk

, Pontefract

, Ritvo

, Archibald

, Murray

. The impact of fatigue on patients with multiple sclerosis. Can J Neurol Sci, 1994; 21:9–14.

66.

Patten

, Marrie

, Carta

. Depression in multiple sclerosis. Int Rev Psychiatry, 2017; 29:463–472.