Psychometric Properties of the Pediatric Acute-Onset Neuropsychiatric Syndrome Global Impairment Score in Children and Adolescents with Pediatric Acute-Onset Neuropsychiatric Syndrome

Abstract

Objectives:

This study validates the caregiver-rated Pediatric Acute-onset Neuropsychiatric Syndrome (PANS) Global Impairment Score (GIS), a single-item, 0–100 scale, for use in PANS.

Methods:

We collected longitudinal data from community patients meeting PANS criteria. We included 128 patients with 1926 GISs, each from a unique clinic visit. To assess discriminant validity, we compared GISs from patients with PANS with scores from a population of healthy controls. To evaluate external validity, we compared global impairment with a clinician-reported global measure—the Child Global Assessment Scale (CGAS)—using the Bland–Altman plots and correlation coefficients. Then, we evaluated associations between the PANS GIS and symptom-specific disease severity variables by fitting mixed models with repeated measures.

Results:

The GIS shows excellent discriminant validity, distinguishing patients with PANS from healthy controls. The scores on the GIS show an acceptable level of agreement with the clinician-reported CGAS. The regression line in the Bland–Altman plot had a positive slope, indicating that parents tend to report higher disease severity than clinicians at higher levels of disease severity. Correlation was higher during disease remissions than during disease flares (r = −0.69 vs. r = −0.48). All disease severity scales predicted GIS in the expected direction.

Conclusion:

The GIS has excellent discriminant validity and acceptable construct validity.

Introduction

Pediatric Acute-onset Neuropsychiatric Syndrome (PANS) is defined by the abrupt-onset of obsessive-compulsive disorder (OCD) and/or food restriction with at least two other equally debilitating neuropsychiatric symptoms from the following categories: anxiety; emotional lability or depression; irritability, aggression, and/or severely oppositional behaviors; deterioration in school performance; sensory or motor abnormalities; and somatic signs and symptoms, including sleep disturbances, enuresis, or urinary frequency (Swedo et al. 2012; Chang et al. 2015). Patients often follow a relapsing-remitting course, intermittently experiencing neuropsychiatric flares and remissions. The illness is often complicated by comorbidities, including pain, malaise, fatigue, as well as other autoimmune, dermatologic, and neurodevelopmental disorders. PANS onset often coincides with infections such as sinusitis, streptococcal pharyngitis, and mycoplasma pneumoniae or another inflammatory condition. The subgroup associated with Streptococcal infections is called pediatric autoimmune neuropsychiatric syndrome associated with Streptococcal infections (PANDAS) (Swedo et al. 1998, 2012; Murphy et al. 2012).

The diversity of inciting events, PANS symptoms, and severity complicates treatment. In response, some PANS practitioners treat patients using multiple modalities in a multidisciplinary clinic in which psychiatrists treat neuropsychiatric symptoms while other specialists address infections and inflammation (Frankovich et al. 2015; Cooperstock et al. 2017; Thienemann et al. 2017). PANS symptom heterogeneity also complicates research because it is difficult to compare disease severity across patients and within a patient over time. For example, two patients with different symptoms may have equivalent impairment: one patient's impairment may be driven by severe OCD symptoms, whereas the other's impairment is driven by severe generalized anxiety. Equally impairing symptoms may change flare to flare within an individual (e.g., an individual experiences severe OCD during one disease flare and severe generalized anxiety with little OCD during the second flare). Symptom-specific ratings alone would not capture the same impairment level. Without a global measure of patient status, it is difficult to track disease impact.

Also, some existing global measures of psychiatric disease severity focus on the clinician's assessment. The caregiver and the clinician may not perfectly agree on a child's disease severity because they bring to the clinic different sets of knowledge. While the clinician carries the knowledge of how this disease has manifested itself in previous patients, the caregiver—especially if he or she is a parent—holds intimate knowledge of how the disease has affected this patient in particular. These sets of knowledge can complement each other even as (or especially because) they can conflict with each other.

To address these issues, experienced PANS/PANDAS clinicians Dr. James Leckman of the Yale University and Dr. Tanya Murphy of the University of South Florida developed the PANS Global Impairment Score (GIS), a single-item caregiver- or patient-reported assessment of impairment that has been previously reported but never described or validated (Brown et al. 2017). The score is not designed to replace symptom-specific measures of disease severity or clinician reports. Rather, as a caregiver-reported score, it is designed to measure a caregiver's perspective of a child's PANS illness, which could add to symptom-specific measures and to a clinician's global assessment. The GIS, if valid and reliable, could be useful clinically or potentially in research settings as a component of an outcome in prospective studies. The GIS would also be useful for basic science studies as a continuous measure of patient status filled out on the same day the patient has biological examination, such as a blood draw or brain magnetic resonance imaging. Finally, the GIS is easy for caregivers to fill out, lending itself to clinical use.

This study seeks to validate the GIS for clinical use in PANS by assessing the discriminant validity, construct validity, and sensitivity to change. Specifically, we aimed to (1) assess how well the parent-reported GIS agreed with the clinician-reported Child Global Assessment Score and (2) test the assumption that most—if not all—PANS symptoms would influence the PANS GIS.

Methods

Setting

Our multidisciplinary PANS clinic serves community-based families affected by PANS and PANDAS. Between September 2012, when the clinic opened, and December 2017, the clinic has accepted and tracked 287 patients. Patients are diagnosed with PANS, PANDAS, both, or neither based on published research criteria (Swedo et al. 1998, 2012, 2017). One important difference between PANS and PANDAS is that PANDAS requires a preceding streptococcal infection, whereas PANS is agnostic to preceding events. In the Stanford PANS clinic, treatment is coordinated between practitioners of various disciplines (psychiatry, psychology, primary care, rheumatology, immunology, ENT, and occasionally, neurology), as well as a social work psychotherapist and an education specialist (Frankovich et al. 2015).

Data sources

We draw measures of disease severity from a questionnaire that the patient or parent fills out before each appointment with the PANS clinic. Patients have different numbers and frequencies of visits, so we have variable amounts of data for each patient.

The questionnaire has included the following scales: (1a) the Child Yale-Brown Obsessive Compulsive Scale (Scahill et al. 1997), (1b) a 0- to 15-point OCD scale (likelihood scale), (2) the Yale Global Tic Severity Scale (Leckman et al. 1989), (3) an Avoidant/Restrictive Food Intake symptom scale (unpublished data), (4) the Columbia Impairment Score (Bird et al. 1993), (5) the Caregiver Burden Inventory (Novak and Guest 1989), (6) the Modified Overt Aggression Scale (Sorgi et al. 1991), and (7) a Fibromyalgia scale measuring pain, fatigue, and malaise (Wolfe et al. 2010; Ting et al. 2016). All scales have previously been validated with the exception of the OCD and the Avoidant/Restrictive Food Intake symptom scales, both of which were adapted from prior studies, but neither of which was formally validated (Murphy et al. 1982; Insel et al. 1983). A symptom severity checklist was added to the questionnaire in April 2017. This checklist prompts parents to rate the severity during the past week of 30 symptoms on a 0–4 scale. Two other measures of disease severity are clinician-reported: disease flare status (yes/no) and the Child Global Assessment Scale (CGAS) (Shaffer et al. 1983; Brown et al. 2017). The CGAS measures child and adolescent psychosocial function due to psychiatric illness. Scores range from 100 (lowest impairment) to 1 (highest impairment). A score of 41–50 reflects a “moderate degree of interference in functioning in most social areas or severe impairment of functioning in one area.”

We show the PANS GIS prompt in Table 1. A short statement first primes caregivers to think about certain aspects of a child's disease. Five anchor statements (one every 20 points) then describe patient functioning within each category of scores. Caregivers report a score from 0 to 100 using a horizontal slide bar. The GIS scaled from 0 to 50 until February 2015, and then, it was changed to scale from 0 to 100. Zero represents no impairment and 100 represents the highest level of impairment. For these analyses, we scaled all scores to range from 0 to 100 by multiplying older scores by two. A score between 41 and 60 reflects “Moderate impairment (due to any PANS symptom, psychiatric, or nonpsychiatric) with a definite decrease in ability to function on their own. Symptoms would lead to seeking psychiatric and medical treatments and services.” In our clinic, the global impairment question is only available in English.

Table 1.

Before Reporting a Global Impairment Score from 0 to 100, Caregivers of Patients with Pediatric Acute-Onset Neuropsychiatric Syndrome Read the Following Prompt, Which Includes Anchors Every 20 Points

Global impairment symptoms
Please rate patient's overall Global Impairment in the past week using the scale below. Input number (0–100) in the space provided. This will include impairment from motor issues, psychiatric issues, headaches, abdominal pain, other pain issues, fatigue, exercise intolerance, etc.
0 No impairment
1–20 Minimal impairment. Functional and social capabilities within the range of normal but may include some degree of social withdrawal or performance below normal levels for the person.
21–40 Mild impairment. More difficulties with daily activities. Friends and relatives may notice difficulties.
41–60 Moderate impairment. Definite decrease in ability to function on their own. Symptoms would lead to seeking psychiatric and medical treatments and services.
61–80 Severe impairment, patient unable to function without substantial help.
81–100 Very severe impairment, patient requires 1:1 supervision at all times and for all self-care (e.g., brushing teeth, eating, bathing, dressing).

We also define disease flare status (flare, remission, or unclear) at each clinic visit as previously described (Brown et al. 2017). A research assistant defined flare status for visits between September 1, 2012, and Spring 2017. Since Spring 2017, clinicians have defined flare status at the end of each clinic visit, basing their decisions on a comprehensive clinical visit that includes review of the patient questionnaire, which includes the GIS. Briefly, a flare is defined as a discrete episode of neuropsychiatric symptoms that meet the PANS/PANDAS criteria.

Stanford University's Human Subjects Institutional Review Board approved this research. Parents provided written informed consent, and patients provided written assent or consent.

Inclusion/exclusion criteria

We show an inclusion flowchart in Figure 1. Inclusion criteria for this study were as follows: at least one GIS, diagnosis of PANS and/or PANDAS, patient younger than 18 years at first clinic visit, and family lives within 90 miles of clinic and in one of seven counties surrounding the San Francisco Bay (Marin, Contra Costa, San Francisco, Alameda, San Mateo, Santa Clara, and Santa Cruz). We also included data from 63 healthy controls recruited from the surrounding community through fliers and by asking families in our clinic to recruit friends and neighbors of approximately the same age as their child(ren). All controls filled out an extensive questionnaire.

FIG. 1.

Inclusion/exclusion criteria for a study seeking to validate the global impairment score for use in children and adolescents diagnosed with PANS. Before entering our PANS clinic, patients are prescreened for likelihood of having PANS by the following: family patient questionnaire, review of outside records and/or discussion with primary medical doctor. Patients must have had an established primary medical doctor for more than 3 years to gain entry into the PANS clinic. PANS, Pediatric Acute-onset Neuropsychiatric Syndrome.

Missing data

Our patient questionnaire was changed in Spring 2017. A few symptom scales used since the beginning of our clinic were replaced, and additional scales were added. Each variable was measured for unique—often overlapping—populations of patients with PANS. This reality limits the type of analysis that we can run; germanely, it prevents us from running multivariate regressions with all symptom scales as predictors.

Statistical analyses

We present descriptive statistics and preliminary normative data for GISs during disease flare and disease remission (Table 2). We assessed the discriminant validity by comparing GIS in our population of PANS patients with scores in a population of healthy controls (using t-tests for normally distributed data and Wilcoxon rank-sum tests for non-normally distributed data). We expected that GIS would clearly distinguish the two groups.

Table 2.

Demographics of 128 Patients with 1926 Visits at a Community-Based Clinic Serving Patients with Pediatric Acute-Onset Neuropsychiatric Syndrome

Variable	Statistic
Age at psychiatric symptom onset,^a mean (SD)	8.5 (3.5)
Age at first PANS appointment, mean (SD)	10.4 (3.5)
Male, n (%)	78 (60.9)
Race/ethnicity, n (%)
Asian	2 (1.6)
White (non-Hispanic)	76 (59.4)
Hispanic/Latino	15 (11.7)
Unknown	35 (27.3)
Criteria, n (%)
PANS	72 (56.3)
PANS+PANDAS	52 (40.3)
PANDAS only	4 (3.1)
Course of illness,^b n (%)
Single episode	11 (8.6)
Relapsing/remitting	82 (63.6)
Chronic/static	27 (20.9)
Progressive	8 (6.2)
Visits with global impairment score, median (IQR)	12.5 (6.5–20.5)
Global impairment (0–100) at the first flare, mean (SD)	48.4 (24.1)
Global impairment (0–100) at the first remission, median (IQR)	20 (10–40)
Follow-up time^c (years), median (IQR)	1.8 (1.0–3.0)

Psychiatric symptom onset is defined as the date of the first event of sudden-onset psychiatric symptoms, which may have met PANS criteria, but, in many cases, we cannot say with certainty. These data were determined through conversations with parents and primary medical doctors and through chart review.

A PANS course of illness can change over time. For instance, after one PANS flare that has subsided, all courses are, by definition, single episode. Also, relapsing/remitting courses can become chronic/static. For the purpose of these data, we used the most recent information to categorize the patients.

Follow-up time is the time from the patient's first clinic visit to the latest visit at the time of data analysis.

IQR, interquartile range; PANDAS, Pediatric Autoimmune Neuropsychiatric Disorders Associated with Streptococcal Infections; PANS, Pediatric Acute-onset Neuropsychiatric Syndrome; SD, standard deviation.

Our external validity analyses draw from the validation of the Bath Ankylosing Spondylitis Global Scale, which assessed correlations between the scale and similar but not identical clinical measures (Jones et al. 1996; Calin et al. 1999). The study then evaluated the scale longitudinally, analyzing test–retest reliability and sensitivity to change (Jones et al. 1996). This validation study followed good psychometric practice: validation studies should clearly define, a priori, the theoretical construct measured. Then, the study should statistically assess how well the measure approximates the construct (Borsboom 2006; Holmbeck and Devine 2009). If the measure is used longitudinally, a validation study should analyze the reliability and sensitivity to change.

We hypothesized that the PANS GIS measures a similar, but not identical, construct to the CGAS, a practitioner-reported measure of psychosocial functioning. To compare the two variables, we calculated the Spearman rank-order correlation coefficients separately for flare and remission, using data from the first flare and first remission for each patient for whom we have data on both the GIS and CGAS. We visualized the agreement between the GIS and CGAS in the Bland–Altman plot (Fig. 2). Bland–Altman plots are often used to assess the agreement between two measures. Each pair of measurements on the same unit of observation is included as a point in a scatter plot, where the x-axis is the average of the two measurements and the y-axis is the difference between measurements. Generally, studies using Bland–Altman plots report the average y-value and 95% limits of agreement (the y-values within which 95% of observations are expected to fall). If two measurements agree perfectly (and both measurements are on the same scale), all points will fall on y = 0. The closer the average y-value is to 0, the better the average agreement. Narrower limits of agreement indicate better range of agreement.

FIG. 2.

The Bland–Altman plot comparing global impairment with the practitioner-reported Child Global Assessment Scale. Data come from visits (n = 95 patients with 471 visits) with data on global impairment and Child Global Assessment Scale. In this Bland–Altman plot, the y-axis represents the difference between the global impairment score and a scaled version of the Child Global Assessment Scale. Negative y-values indicate that the clinician rated the patient as more impaired than the caregiver; positive values indicate that the clinician rated the patient as less impaired than the caregiver. A value of y = 0 indicates perfect agreement. Values farther from 0 indicate less agreement. The x-axis is the average of the two scores. Larger x-axis values indicate more impairment. The regression line shows the association between the y-axis (difference in ratings) and x-axis (impairment). In this case, as impairment increases, the difference in ratings becomes more positive, indicating that the clinician rates patients as less impaired than the caregiver at higher values of average score. The 95% LOAs are the limits within which the data predict that 95% of observations will fall. The 95% LOA account for correlations within patient. In this case, accounting for these correlations widens the limits. LOAs, limits of agreement.

Our Bland–Altman plot includes all observations for which we have data on both the GIS and CGAS (471 observations for 95 patients). We accounted for correlations within patient by calculating 95% Limits of Agreement using the Bland and Altman's algorithm for assessing the agreement with multiple observations per individual (Bland and Altman 2007). Since the CGAS runs from 100 to 1, whereas GIS runs from 0 to 100, we scaled CGAS to range from 0 to 100 using Equation (1). \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*}{ \rm Scaled \ CGAS} = 100 - ( { \rm CGAS} - 1 ) \times \left( {100 / 99} \right) . \tag{1} \end{align*} \end{document}

We then assessed the agreement using categorized scores. Since the GIS and CGAS have similar anchor points, we categorized the scores based on the anchor statement into which the score falls. We gave a value of 0 to GIS of 0 and CGAS scores of 100; we gave a value of 1 to GIS of 1–20 and CGAS scores of 99–80 and so on. Using the categorical data, we report the number and percentage of observations in which (1) caregiver and clinician agree, (2) caregiver reports greater severity, and (3) clinician reports greater severity. We then report correlations between disagreement and other variables (Table 3). We hypothesized, a priori, that the following variables are associated with disagreement: male patient sex, high caregiver social burden, and chronic/static disease course. After initial analyses, we hypothesized (post hoc) that two Fibromyalgia scores (Symptom Severity Scale and Widespread Pain Index) would be associated with greater disagreement because the CGAS theoretically only responds to psychiatric impairment, whereas the global impairment responds to all impairments (including pain and fatigue). In these analyses, the outcome is categorical agreement—agree, parent higher, or clinician higher. Outcomes can be either categorical or continuous. All analyses used the generalized estimating equation models with repeated measures.

Table 3.

Agreement Between Parent and Clinician Assessment of Child Impairment from Pediatric Acute-Onset Neuropsychiatric Syndrome

Agreement	Flare	Remission
Perfectly agree on category of impairment	94/271 (34.7%)	93/183 (50.8%)
Clinician rating higher	118/271 (43.5%)	69/183 (37.7%)
Parent rating higher	59/271 (21.8%)	21/183 (11.5%)
Total	271	183

Data come from clinic 454 visits during which PANS flare status was defined, the clinician gave a Child Global Assessment Scale rating, and the parent gave a global impairment rating. Here, we present the agreement between the two measures separately by flare and remission. These data are repeated measurements of 95 patients. We do not take correlations within patient into account for this table because these data are presented to generate hypotheses, not to test hypotheses.

PANS, Pediatric Acute-onset Neuropsychiatric Syndrome.

Our second main expectation was that multiple symptoms influence the GIS; therefore, all disease scales should predict the GIS. One provider hypothesized that male sex predicts increased GIS (due to the clinical experience). We also hypothesized that age at symptom onset does not predict the GIS, but the time between psychiatric symptom onset and first clinic appointment does due to the hypothesis that the disease is more difficult to treat if treatment is delayed following symptom onset. To test these hypotheses, we fitted mixed models with repeated measures in which the outcome was GIS. To facilitate interpretation, we standardized all predictor variables (with two exceptions, explained in the footnote of Table 4) to have a mean of 0 and a standard deviation of 1. From each mixed model, we report the number of observations used and the effect estimate and 95% confidence interval (CI) (Table 4). In this analysis, we modeled intercept as random due to theoretical considerations (impairment at entry into clinic is heterogeneous) and because doing so resulted in a smaller Akaike's Information Criterion, Corrected (AICC). We used an unstructured covariance structure because it resulted in the smallest AICC of all covariance structures tested.

Table 4.

Results of Longitudinal Mixed Models Evaluating Predictors of Global Impairment Score in a Multidisciplinary Pediatric Acute-Onset Neuropsychiatric Syndrome Clinic

Variable ^a	No. of observations used ^b	Effect estimate (95% CI)^c	p
Years in clinic	1926	−4.7 (−3.7 to −5.6)	<0.0001
Flare	1778	16.6 (14.9 to 18.3)	<0.0001
Disease Severity Scales
CY-BOCS	443	12.3 (10.6 to 14.0)	<0.0001
OCD	1534	14.3 (13.4 to 15.2)	<0.0001
YGTSS	156	6.5 (2.7 to 10.3)	0.0009
ARFI	1515	9.4 (8.2 to 10.5)	<0.0001
CIS	488	12.1 (10.4 to 13.9)	<0.0001
CBI	1135	13.1 (11.6 to 14.6)	<0.0001
MOAS	550	5.0 (4.0 to 6.1)	<0.0001
SSS	1823	11.6 (10.5 to 12.8)	<0.0001
WPI	1926	1.8 (0.6 to 2.9)	0.003
Other PANS Symptom Scales
Separation anxiety	528	6.8 (5.1 to 8.6)	<0.0001
Other anxiety	528	9.3 (7.7 to 11.0)	<0.0001
Behavioral regression	528	4.1 (2.3 to 5.9)	<0.0001
Mood disturbance	528	9.6 (8.0 to 11.1)	<0.0001
Oppositionality	528	9.6 (7.9 to 11.2)	<0.0001
School difficulties	528	9.2 (7.7 to 10.8)	<0.0001
Sleep disturbance	528	8.1 (6.3 to 9.8)	<0.0001
Urinary frequency	528	5.2 (3.5 to 6.8)	<0.0001
Delusions	528	7.2 (5.5 to 8.9)	<0.0001
Pain	528	7.9 (6.2 to 9.7)	<0.0001
Demographics
Age at first clinic visit	1926	4.0 (0.9 to 7.1)	0.01
Age at symptom onset	1926	2.9 (−0.3 to 6.0)	0.07
Time between onset and first clinic visit	1926	1.6 (−1.2 to 4.5)	0.3
Male sex	1926	−0.4 (−7.0 to 6.2)	0.9

All models evaluate the predicted effect that an increase in the variable in the first column has on the global impairment score.

Number of observations is highly variable because the clinic has collected different disease severity variables over the course of the past 5 years.

For all variables except years in clinic and flare, the effect estimate refers to the change in global impairment score predicted by a one standard deviation increase in the predictor variable.

ARFI, Avoidant or Restrictive Food Intake; CBI, Caregiver Burden Inventory; CI, confidence interval; CIS, Columbia Impairment Scale; CY-BOCS, Child Yale-Brown Obsessive Compulsive Scale; MOAS, Modified Overt Aggression Scale; OCD, Obsessive Compulsive Disorder; PANS, Pediatric Acute-onset Neuropsychiatric Syndrome; SSS, Fibromyalgia symptom severity scale; WPI, Widespread Pain Index; YGTSS, Yale Global Tic Severity Scale.

We stored data on a RedCap server, and we performed analyses on the SAS University Edition (Cary, NC) (Harris et al. 2009).

Results

The final study sample included 128 children and adolescents with PANS with a total of 1926 PANS GISs (median 12 scores, interquartile range [IQR] 7–20 scores) and 63 healthy controls (Table 1). Seventy-eight of 128 patients (61%) were male compared with 26 of 59 healthy controls (44%) with data on sex; average age at onset of psychiatric symptoms was 8.5 years (standard deviation 3.5 years). Average age at first clinic appointment was 10.4 years (standard deviation 3.6 years). In healthy controls, average age at data collection was 11.8 years (standard deviation 4.2 years).

The GIS during the first flare was high, approximating a normal distribution (mean = 48.4, standard deviation = 24.1); the GIS during the first remission was rated less severe, with a right-skewed distribution (median = 20, IQR 10–40). Five of 118 (4%) patients had a score of 0 during their first remission. In our healthy child population, however, the median score was 0 (IQR 0–0). Caregivers of 60 of the 63 (95%) healthy children reported a score of 0. It is highly unlikely that we would have found this difference by chance alone (p < 0.0001).

The GIS correlated moderately with the CGAS during the first flare captured in our clinic (Spearman's correlation coefficient = −0.48, p < 0.0001) and with the first remission captured in our clinic (Spearman's correlation coefficient = −0.69, p < 0.0001). We graphed the agreement in the Bland–Altman plot (Fig. 2). On average, scaled CGAS values were larger than GISs, indicating that the clinician rated the patient as more impaired than the caregiver did, on average. However, the regression line in Figure 2 indicates that at higher disease severity levels, GISs were higher than scaled CGAS scores. There was no association between disagreement and patient sex, course of illness, the Widespread Pain Index, or caregiver's social burden (data not shown). However, malaise score was associated with disagreement. Caregivers reporting a higher score than clinician predicted a 1.2 point increase in malaise score compared with parent agreeing with clinician (95% CI 0.3–2.1) and a 1.5 point increase in malaise score compared with clinician reporting a higher score than caregiver (95% CI 0.6–2.5).

In our mixed models, the GIS decreased linearly with increasing years engaged in clinic. Each additional year in clinic predicted a 4.7-point decrease in the GIS (95% CI −5.6 to −3.7).

All disease severity variables predicted GIS in the expected direction (Table 4). Flare status predicted GIS most strongly; a flare, compared with remission, predicted a 16.6-point increase in the GIS (95% CI 14.9 − 18.3). The OCD score had the largest effect on the GIS of any disease scale; each standard deviation increase in OCD score predicted a 14.3-point increase in the GIS (95% CI 13.4 − 15.2). All individual symptoms also predicted GIS in the expected direction (Table 4). Of demographic variables, increased age at first clinic visit predicted higher GIS, but no effect was found for age at symptom onset, time between onset and first clinic visit, and patient sex (Table 4).

Discussion

Holmbeck and Devine (2009) provide a checklist for the development of a new measure, which lists four items: (1) establishing a scientific need, (2) “building in” content validity, (3) establishing reliability, and (4) analyzing validity. In this discussion, we respond to each item in turn.

The PANS GIS is scientifically necessary because PANS is a heterogeneous disease that impacts different facets of functioning, which can complicate both treatment and research. The GIS draws from all sources of impairment, outputting a single score that can be used to track overall functioning over time. Similar scales exist in the rheumatology and psychiatry literatures for diseases without objective indices of disease activity (e.g., biomarkers, images, and tissue samples). In rheumatology, researchers often try to measure the current activity of an underlying disease using single-item disease activity indices, which have been employed in juvenile idiopathic arthritis, juvenile ankylosing spondylitis, and lupus (Bombardier et al. 1992; Consolaro et al. 2009; Weiss et al. 2014). Psychiatric research uses disorder-specific and global functional impairment scales (e.g., in studies involving depression, schizophrenia, self-esteem, and childhood mental health issues; Shaffer et al. 1983; Bird et al. 1987; Robins et al. 2001; Kadouri et al. 2007). These measures can help clinicians to monitor the disease activity and function. They can also gauge outcomes in research studies, especially in diseases with heterogeneous symptoms.

The GIS was developed to address this need in PANS/PANDAS research. As it is a single-item scale, we cannot perform confirmatory factor analysis to assess the reliability; therefore, this study focused on discriminant and external validity.

To assess the discriminant validity, we compared GISs in our population with a population of healthy children. Most caregivers of healthy children report a GIS of 0; therefore, the measure discriminates between healthy children and patients with PANS.

We then assessed the external validity, first by evaluating the agreement between the caretaker-rated GIS and the clinician-rated CGAS. The two measures agreed modestly. We found a correlation of −0.69 during disease remission; for reference, a previous study found a correlation of −0.65 between the CGAS and the behavior problem score on the Child Behavioral Checklist (Bird et al. 1987). Interestingly, correlation was tighter during disease remission than flare. To understand this relationship, we graphed the agreement between global impairment and CGAS on the Bland–Altman plot, finding that GISs, on average, were 4.8 points less than a scaled version of the CGAS (possible difference ranges from 100 points less than to 100 points greater than). The 95% confidence limits for agreement ranged from −39 points to +30 points. This confidence limit was wider than expected. Interestingly, a regression line in Figure 2 sloped upwards, indicating that the CGAS reports more severe impairment than the GIS when the average score is low, whereas GISs tend to be higher than CGAS scores when average scores are high. One explanation for this finding could be that parents and clinicians have different banks of knowledge. When a child is doing poorly, the parent may never have seen their child so impaired, leading them to rate the child at the high end of the GIS; however, the clinician has likely seen patients with the same disease doing worse, so they may report more moderate scores. If true, this could explain the tighter correlation found during disease remissions compared with disease flares when children have higher GISs.

Our 95% limits of agreement between clinicians and parents were wider than agreement between health-related quality-of-life scales filled out by parents and their children with skin disease, which found 95% limits of agreement between −25.5/100 and 26.7/100 (Beattie and Lewis-Jones 2006). However, Beattie and Lewis-Jones evaluated identical constructs, whereas we evaluated slightly different constructs. While the GIS prompts caretakers to think about all sources of impairment (including pain and fatigue), the CGAS prompts clinicians to only consider psychosocial impairment due to psychiatric symptoms. Therefore, discordance between the CGAS and GIS may be explained by symptoms such as pain and fatigue, which clinicians often observe in PANS (Leibold et al., under review). We found some evidence to support this hypothesis: when the caretaker reported a higher score than clinician, malaise score, an indicator of pain and fatigue, increased. Clinicians do not typically consider pain, fatigue, and other malaise symptoms when scoring the CGAS. Although pain and fatigue may be a part of the PANS illness, patients can also experience pain or fatigue from another cause. In one example, a patient had abdominal pain due to recent appendicitis. Since the GIS prompts the caregiver to consider pain, the caregiver gave a high score (>50 points). The clinicians noted the high score before their clinic visit and prepared for a moderately impaired child; however, the patient was doing quite well from a PANS standpoint. The clinicians were confused until they learned about the recent appendicitis.

Discordance could also be explained by the fact that caregivers and clinicians have different perspectives on a child's illness. Some evidence suggests that parents and clinicians agree comparably with children on measures of the quality of life (Bastiaansen et al. 2004); however, parents and clinicians do have different perspectives of a child's illness, so their ratings could systematically differ (Achenbach et al. 1987). Therefore, information from multiple informants should be interpreted in concert to provide a comprehensive view of child functioning. Given the above arguments, we believe that there is an acceptable level of agreement between the GIS and CGAS, suggesting that the GIS agrees somewhat with clinician impression of functioning, but from a parental perspective, which the GIS is useful in clinical settings both for getting a sense of the child's impairment and for improving parent–clinician communication regarding the child's impairment. However, we believe that the GIS is not sufficiently refined for use in clinical trials.

The GIS tends to decrease throughout the course of a patient's engagement with the Stanford PANS clinic (Table 4). A decrease of 4.7 points per year is on the lower end of clinical significance. More research is needed on the kinetics of the GIS. Clinical observation does suggest that the disease severity tends to decrease over time (probably through some combination of regression to the mean and effective clinical treatment). This result provides initial evidence that the GIS is reliable as a longitudinal measure.

As expected, increases in all disease severity variables predicted increased GIS. Clinician-defined patient flare status and caregiver-reported OCD score predicted the largest increases in the GIS. Other variables that predicted increased GIS were caregiver-rated Child Yale-Brown Obsessive Compulsive Scale, an Avoidant/Restrictive Food Intake symptom score, the Columbia Impairment Score, the Caregiver Burden Inventory, the Modified Overt Aggression Scale, and Fibromyalgia scales (the Symptom Severity Scale and the Widespread Pain Index). These scales are all caregiver-derived, so common method variance may explain some of the association (Holmbeck and Devine 2009). However, the fact that flare status and CGAS are both clinician-derived discounts this claim.

Based on clinical experience, one provider had expected male patients to have higher GISs than female patients; however, we found no difference based on sex. We also found no effect on the GIS for time between psychiatric symptom onset and first clinic visit, which was surprising considering the hypothesis that the treatment is less effective, the longer it is delayed after PANS onset (Frankovich et al. 2017).

Limitations

Our population, as reported previously, has high socioeconomic status (Farmer et al. 2018). Also, anecdotally, most of the caregivers who filled out the patient questionnaire are mothers. It is possible that caregiver assessment of child functional impairment depends on socioeconomic status and/or caregiver identity.

Other scales could have been used in place of the GIS. For instance, the Clinical Global Impressions scale has been used in a similar context. It is possible that the Clinical Global Impressions scale is more appropriate than the GIS for research studies; however, the purpose of the GIS is to assess the parent's view of their child's illness.

Our correlation between flares and the GIS suffers from circular logic—flare status is determined at least in part by the GIS, so some correlation between the two variables is expected.

We modeled changes in the GIS as linear; however, PANS has been described as a relapsing/remitting disease. We were unable to take into account more complex changes in the GIS in this study.

Future directions

We plan to use the GIS as a continuous measure of patient status measured on the same day a patient gives a biological specimen. This work will allow us to determine whether an immune profile—for example, TNF expression—tracks with global impairment due to PANS.

To further validate the concept of a PANS flare, an inter-rater reliability study could be performed using data from two clinicians who attended the same clinic visit.

Conclusion

The PANS GIS agrees acceptably with clinician-derived measures of disease severity, is responsive to all sources of child impairment, and discriminates between patients with PANS and healthy controls.

Clinical Significance

The GIS assesses a caretaker's perspective of PANS severity and is a useful clinical tool for tracking patients over time. The tool can also be used to improve parent–clinician communication regarding a patient's functional status.

Disclosures

No competing financial interests exist.

Footnotes

Acknowledgments

The authors acknowledge Cristan Farmer, PhD, for her statistical advice, the PANDAS Physicians Network, the Stanford SPARK program, Dr. Susan Swedo, MD, the National Institute of Mental Health, and all the staff and faculty at the Lucile Packard Children's Hospital who make caring for these children possible. We thank Kiki Chang, MD, for his critical review of this article.

References

Achenbach

, McConaughy

, Howell

: Child/adolescent behavioral and emotional problems: Implications of cross-informant correlations for situational specificity. Psychol Bull, 101:213–232, 1987.

Bastiaansen

, Koot

, Bongers

, Varni

, Verhulst

: Measuring quality of life in children referred for psychiatric problems: psychometric properties of the PedsQL 4.0 generic core scales. Qual Life Res, 13:489–495, 2004.

Beattie

, Lewis-Jones

: A comparative study of impairment of quality of life in children with skin disease and children with other chronic childhood diseases. Br J Dermatol, 155:145–151, 2006.

Bird

, Canino

, Rubio-Stipec

, Ribera

: Further measures of the psychometric properties of the Children's Global Assessment Scale. Arch Gen Psychiatry, 44:821, 1987.

Bird

, Shaffer

, Fisher

, Gould

: The Columbia Impairment Scale (CIS): Pilot findings on a measure of global impairment for children and adolescents. Int J Methods Psychiatr Res, 3:167–176, 1993.

Bland

, Altman

: Agreement between methods of measurement with multiple observations per individual. J Biopharm Stat, 17:571–582, 2007.

Bombardier

, Gladman

, Urowitz

, Caron

, Chang

: Derivation of the SLEDAI. A disease activity index for lupus patients. Arthritis Rheum, 35:630–640, 1992.

Borsboom

: The attack of the psychometricians. Psychometrika, 71:425–440, 2006.

Brown

, Farmer

, Mark Freeman

, Spartz

, Farhadian

, Thienemann

, Frankovich

: Effect of early and prophylactic nonsteroidal anti-inflammatory drugs on flare duration in pediatric acute-onset neuropsychiatric syndrome: An observational study of patients followed by an academic community-based pediatric acute-onset neuropsychiatric syndrome clinic. J Child Adolesc Psychopharmacol, 27:619–628, 2017.

10.

Calin

, Nakache

J-P

, Gueguen

, Zeidler

, Mielants

, Dougados

: Defining disease activity in ankylosing spondylitis: Is a combination of variables (Bath Ankylosing Spondylitis Disease Activity Index) an appropriate instrument?. Rheumatology, 38:878–882, 1999.

11.

Chang

, Frankovich

, Cooperstock

, Cunningham

, Latimer

, Murphy

, Pasternack

, Thienemann

, Williams

, Walter

, Swedo

; PANS Collaborative Consortium: Clinical evaluation of youth with pediatric acute-onset neuropsychiatric syndrome (PANS): Recommendations from the 2013 PANS Consensus Conference. J Child Adolesc Psychopharmacol, 25:3–13, 2015.

12.

Consolaro

, Ruperto

, Bazso

, Pistorio

, Magni-Manzoni

, Filocamo

, Malattia

, Viola

, Martini

, Ravelli

; Paediatric Rheumatology International Trials Organisation: Development and validation of a composite disease activity score for juvenile idiopathic arthritis. Arthritis Rheum, 61:658–666, 2009.

13.

Cooperstock

, Swedo

, Pasternack

, Murphy

: Clinical management of Pediatric Acute-Onset Neuropsychiatric Syndrome: Part III—Treatment and prevention of infections for the PANS/PANDAS Consortium. J Child Adolesc Psychopharmacol, 27:594–606, 2017.

14.

Farmer

, Thienemann

, Leibold

, Kamalani

, Sauls

, Frankovich

: (2018). Psychometric evaluation of the caregiver burden inventory in children and adolescents with PANS. J Pediatr Psychol, 43:749–757, 2018.

15.

Frankovich

, Swedo

, Murphy

, Dale

, Agalliu

, Williams

, Daines

, Hornig

, Chugani

, Sanger

, Muscal

, Pasternack

, Cooperstock

, Gans

, Zhang

, Cunningham

, Bernstein

, Bromberg

, Willett

, Brown

, Farhadian

, Chang

, Geller

, Hernandez

, Sherr

, Shaw

, Latimer

, Leckman

, Thienemann M; PANS/ PANDAS

Consortium

: Clinical management of pediatric acute-onset neuropsychiatric syndrome: Part II—Use of immunomodulatory therapies. J Child Adolesc Psychopharmacol, 27:574–593, 2017.

16.

Frankovich

, Thienemann

, Pearlstein

, Crable

, Brown

, Chang

: Multidisciplinary clinic dedicated to treating youth with pediatric acute-onset neuropsychiatric syndrome: Presenting characteristics of the first 47 consecutive patients. J Child Adolesc Psychopharmacol, 25:38–47, 2015.

17.

Harris

, Taylor

, Thielke

, Payne

, Gonzalez

, Conde

. Research electronic data capture (REDCap)—A metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform, 42:377–381, 2009.

18.

Holmbeck

, Devine

: Editorial: an author's checklist for measure development and validation manuscripts. J Pediatr Psychol, 34:691–696, 2009.

19.

Insel

, Murphy

, Cohen

, Alterman

, Kilts

, Linnoila

Obsessive-compulsive disordera double-blind trial of clomipramine and clorgyline. Arch Gen Psychiatry, 40:605–612, 1983.

20.

Jones

, Steiner

, Garrett

, Calin

: The Bath Ankylosing Spondylitis Patient Global Score (BAS-G). Rheumatology, 35:66–71, 1996.

21.

Kadouri

, Corruble

, Falissard

: The improved Clinical Global Impression Scale (iCGI): Development and validation in depression. BMC Psychiatry, 7:7, 2007.

22.

Leckman

, Riddle

, Hardin

, Ort

, Swartz

, Stevenson

, Cohen

: The Yale Global Tic Severity Scale: Initial testing of a clinician-rated scale of tic severity. J Am Acad Child Adolesc Psychiatry, 28:566–573, 1989.

23.

Murphy

, Pickar

, Alterman

: Assessment of affective disorders. In: Quantitative Techniques for the Evaluation of the Behavior of Psychiatric Patients. Edited by Burdock

, Swdilovsky

, Gershon

. New York, Marcel Dekker, 1982, pp. 355–392.

24.

Murphy

, Storch

, Lewin

, Edge

, Goodman

: Clinical factors associated with pediatric autoimmune neuropsychiatric disorders associated with streptococcal infections. J Pediatr, 160:314–319, 2012.

25.

Novak

, Guest

: Application of a multidimensional caregiver burden inventory. Gerontologist, 29:798–803, 1989.

26.

Robins

, Hendin

, Trzesniewski

: Measuring global self-esteem: construct validation of a single-item measure and the Rosenberg Self-Esteem Scale. Pers Soc Psychol Bull, 27:151–161, 2001.

27.

Scahill

, Riddle

, McSwiggin-Hardin

, Ort

, King

, Goodman

, Cicchetti

, Leckman

. J Am Acad Child Adolesc Psychiatry. 36:844–852, 1997.

28.

Shaffer

, Gould

, Brasic

, Ambrosini

, Fisher

, Bird

, Aluwahlia

: A Children's Global Assessment Scale (CGAS). Arch Gen Psychiatry, 40:1228, 1983.

29.

Sorgi

, Ratey

, Knoedler

, Markert

, Reichman

: Rating aggression in the clinical setting. A retrospective adaptation of the Overt Aggression Scale: Preliminary results. J Neuropsychiatry Clin Neurosci, 3:S52–S56, 1991.

30.

Swedo

, Frankovich

, Murphy

: Overview of treatment of pediatric acute-onset neuropsychiatric syndrome. J Child Adolesc Psychopharmacol, 27:562–565, 2017.

31.

Swedo

, Leckman

, Rose

: From research subgroup to clinical syndrome: Modifying the PANDAS criteria to describe PANS (pediatric acute-onset neuropsychiatric syndrome). Pediatr Therapeut, 2:2, 2012.

32.

Swedo

, Leonard

, Garvey

, Mittleman

, Allen

, Perlmutter

, Lougee

, Dow

, Zamkoff

, Dubbert

: Pediatric autoimmune neuropsychiatric disorders associated with streptococcal infections: clinical description of the first 50 cases. Am J Psychiatry, 155:264–271, 1998.

33.

Thienemann

, Murphy

, Leckman

, Shaw

, Williams

, Kapphahn

, Frankovich

, Geller

, Bernstein

, Chang

, Elia

, Swedo

: Clinical management of pediatric acute-onset neuropsychiatric syndrome: Part I-Psychiatric and behavioral interventions. J Child Adolesc Psychopharmacol, 27:566–573, 2017.

34.

Ting

, Barnett

, Lynch-Jordan

, Whitacre

, Henrickson

, Kashikar-Zuck

: 2010 American College of Rheumatology adult fibromyalgia criteria for use in an adolescent female population with juvenile fibromyalgia. J Pediatr, 169:181.e1–187.e1, 2016.

35.

Weiss

, Colbert

, Xiao

, Feudtner

, Beukelman

, DeWitt

, Pagnini

, Wright

, Wallace

: Development and retrospective validation of the juvenile spondyloarthritis disease activity index. Arthritis Care Res, 66:1775–1782, 2014.

36.

Wolfe

, Clauw

, Fitzcharles

M-A

, Goldenberg

, Katz

, Mease

, Russell

, Winfield

, Yunus

: The American College of Rheumatology preliminary diagnostic criteria for fibromyalgia and measurement of symptom severity. Arthritis Care Res, 62:600–610, 2010.