Psychometric Properties of 7- and 30-Day Versions of the PROMIS Emotional Distress Item Banks in an Australian Adult Sample

Abstract

This study aimed to examine the psychometric properties of the PROMIS depression, anxiety, and anger item banks in a large Australian population-based sample. The study tested for unidimensionality; evaluated invariance across age, gender, and education; assessed local independence; and tested item bank scores as an indicator for clinical criteria. In addition, equivalence of the 7-day time frame against an alternative 30-day time frame was assessed. A sample of 3,175 Australian adults were recruited into the study through online advertising. All three item banks showed strong evidence of unidimensionality and parsimony, with no items showing local dependence. All items were invariant across age, gender, and education. The item banks were accurate in detecting clinical criteria for major depressive disorder, generalized anxiety disorder, and panic disorder, although legacy measures designed for this purpose sometimes performed marginally better. Responses to the 30-day time frame were highly consistent with the original 7-day time frame. The study provided support for the validity of the PROMIS emotional distress item banks as measures of depression, anxiety, and anger in the Australian population, supporting the generalizability of the measures. The time frame chosen for assessing mental health outcomes using these item banks should be based on pragmatic considerations.

Keywords

depression anxiety anger validation item banks time frame

More rapid and precise assessment of mental and physical health may lead to improved health outcomes. The Patient-Reported Outcomes Measurement Information System (PROMIS®), a collaborative initiative funded by the National Institutes of Health, has developed self-reported outcome measures for assessing a broad range of health-related constructs (Cella et al., 2007). Using item response theory (IRT) methods, the PROMIS initiative has developed and calibrated three item banks assessing emotional distress: depression, anxiety and anger (Pilkonis et al., 2011). These item banks offer a flexible approach to the measurement of emotional distress, and have been used to develop new static and adaptive scales that are highly efficient and precise (Choi, Reise, Pilkonis, Hays, & Cella, 2010; Pilkonis et al., 2011). Item banks provide more thorough assessment of health constructs than single measures, as they build from a systematic approach to pool items together before successive rounds of revision and reduction (Batterham, Brewer, et al., 2015; Pilkonis et al., 2011). Through this process, only the most informative, content valid and psychometrically robust items are retained. Item banks also enable linkages to be made between measures used in population and clinical settings (Pilkonis et al., 2011). Item banks can be used to form common metrics that link a network of existing scales for easy comparison across different measures using in population and clinical settings. However, evidence for their validity across the populations in which these measures are to be used is vital. In addition, as the item banks may be used for multiple purposes within population-based and clinical settings, an examination of the utility of different time frames is also warranted.

Psychometric Properties of the PROMIS Emotional Distress Item Banks

Few independent studies have assessed the validity of the PROMIS emotional distress item banks outside the United States. International testing of the item banks is important as scales may perform differently as a function of culture, health systems, and demographic compositions. One European study has examined the performance of the PROMIS depression measures in a clinical sample (Vilagut et al., 2015), reporting excellent measurement properties. To our knowledge, no independent population-based test of the psychometric properties of the PROMIS emotional distress measures has occurred. Identifying whether the item banks form distinct unidimensional constructs, testing whether items remain accurate across subgroups of the population, and assessing parsimony in an independent sample from outside the United States would provide further evidence for the use of the PROMIS emotional distress item banks in population settings.

In addition, the PROMIS initiative has focused on continuous constructs of health as opposed to categorical constructs. This focus is valuable and has resulted in highly flexible measures that suit a number of settings, including community settings and patients with physical and/or mental health problems. However, the field of psychiatry tends to value mental health scales on the basis of how well they distinguish a specific disorder, while better measurement can inform the clinical utility of diagnostic categorizations (First et al., 2004). The PROMIS measures have not been tested for use as clinical screeners. Therefore, the development of empirically derived cut-points and testing the PROMIS depression and anxiety tools for screening major depressive disorder (MDD), generalized anxiety disorder (GAD), and panic disorder (PD) holds promise of yielding additional support for their use.

Time Frame in Assessment of Mental Health

In assessing mental health symptoms, time frame is an important consideration. Choosing an appropriate time period may depend on a number of factors, including the frequency that symptoms occur, consistency with clinical definitions, consideration of recall biases and periods in which change might be observed (Watson, 1988). Specifically, based on Diagnostic and Statistical Manual of Mental Disorders, 5th Edition (DSM-5; American Psychiatric Association, 2013) criteria, MDD require symptoms to occur over a 2-week time period, while GAD requires longer term symptoms with 6-month duration. Mental health scales typically adopt similar time periods to clinical definitions. Time period may be particularly sensitive to less common symptoms. For example, anxiety symptoms such as heart palpitations might occur episodically, such that an individual may report absence of such symptoms in the past week but more frequently in the past month. Recall biases place upper limits on the time period, as individuals often forget or fail to report distant events including depressive episodes (Wells & Horwood, 2004). Consideration of realistic time frames for change may also influence the selection of a time period. For example, a scale that assesses symptoms over the past year may not be useful in a clinical setting where change over weeks may be of interest.

A small body of research has tested the effects of time period on self-report mental health data. The focus has primarily been on short- versus long-term measurement of affect. Findings have tended to suggest that time frame has little systematic effect on mood measurement, with little impact on the validity of affect measures (Luhmann, Hawkley, Eid, & Cacioppo, 2012; Watson, 1988). Endorsement of particular items may be higher in the long term, which may reflect a greater degree of emotionality in long term recall (Ready, Robinson, & Weinberger, 2006; Terry, Stevens, & Lane, 2005) or simply more time for noticing less common feelings or behaviors. It is important to quantify the impact of different time frames to increase the utility of scales such as the PROMIS item bank for use in a range of settings with differing duration requirements.

The Present Study

This study aimed to test the psychometric properties of the PROMIS emotional distress item banks (depression, anxiety, anger) in an Australian population-based sample. The study consisted of four elements: (1) testing whether the items in the banks fit a unidimensional structure; (2) evaluating invariance across age, gender, and education; (3) identifying items that may be redundant (locally dependent); and (4) testing whether items from the banks can be used to accurately identify individuals who meet clinical criteria for major depression disorder or generalized anxiety disorder. In addition, two versions of each item bank (the original 7-day recall period and a 30-day recall period) were administered in a subset of participants to test the influence of time frame on responses to the items.

Method

Participants and Procedure

Respondents were recruited from the online social media website Facebook using a series of advertisements targeted to all Australian adults aged 18 year or older during August to December 2014. The target population of Facebook users aged 18 years or older was 8.8 million, representing approximately 45% of the total Australian population in that age bracket. The advertisements resulted in 39,945 users clicking on the advertisement. A total of 10,082 adults consented to participate in the survey with 5,011 (50%) completing the survey.

As the survey covered a broad range of disorders with lengthy scales for each, participants were given the option of completing a brief form of the survey (three versions, each ~30 minutes), or a full form (three versions with different item orders, each ~60 minutes). For consistency across analyses, the present study only used the subgroup of participants who completed the full version of the survey (n = 3,175). A subset of these were administered the 7-day versions of the depression (n = 1,066), anxiety (n = 1,075) and anger (n = 1,034) item banks. The survey was conducted online using LimeSurvey software, with data stored on a secure server at the Australian National University, Canberra. The study received ethics approval from the Australian National University Human Research Ethics Committee (Protocol No. 2013/509).

Measures

Participants completed the PROMIS item banks, legacy scales assessing each mental health condition and a checklist of DSM-5 criteria. Other measures focusing on a range of other mental health problems, functioning and service use were assessed but not included in the present study.

PROMIS item banks for depression, anxiety, and anger consist of 28, 29, and 29 items, respectively, each rated on a 5-point categorical response scale from never to always (Pilkonis et al., 2011).

Legacy scales used to assess depression, generalized anxiety and panic disorder were the 9-item Patient Health Questionnaire (PHQ-9; Spitzer, Kroenke, & Williams, 1999), the 7-item Generalized Anxiety Disorder scale (GAD-7; Spitzer, Kroenke, Williams, & Lowe, 2006), and the Panic Disorder Screener (PADIS; Batterham, Mackinnon, & Christensen, 2015), respectively. Symptom scores on these scales can range from 0 to 27, 0 to 21, and 0 to 13, respectively, with higher scores indicating greater symptom severity. These scales have previously been shown to be accurate in screening for risk of disorder and had high internal consistency in the present sample (Cronbach’s α = .93, .92, and .90 for the PHQ-9, GAD-7, and PADIS, respectively).

The DSM-5 symptom checklist was developed by the authors as a proxy for clinical diagnosis. The checklist queried respondents about the presence or absence of symptoms based directly on DSM-5 definitions for each disorder of interest. There were 15 items used to assess MDD (including items to exclude hypomania), 14 for GAD and 21 for PD. Each item reflected a single DSM-5 criterion for the disorder of interest (e.g., for GAD: “During the past six months, have you found it difficult to control your worries?”), although some criteria were probed across multiple questions and additional items were used to exclude alternative diagnoses. The DSM-5 symptom checklist is available online as Supplemental Material (https://journals-sagepub-com.web.bisu.edu.cn/doi/suppl/10.1177/1073191116685809). The checklist was designed along similar principles to the electronic version of the Mini International Neuropsychiatric Interview (MINI; Zbozinek et al., 2012) in terms of structure (binary and categorical self-report items with conditional skip logic) and response burden. However, the checklist used in the current study was developed independently from the MINI, nonproprietary and based on DSM-5 rather than DSM-IV (American Psychiatric Association, 1994) criteria.

Background characteristics including age group, gender, and educational attainment were also assessed.

Psychometric Properties of Item Banks

Examination of the psychometric properties of the PROMIS item banks using IRT proceeded in four stages, reflecting assumptions about the composition of the item banks. First, the unidimensional structure of each item bank was tested using confirmatory factor analysis (CFA), on the basis of limited information weighted least squares with mean and variance adjustment. This estimator is suitable for the analysis of categorical/ordinal data using polychoric correlations. Good model fit was determined using established cut-off values of ≥0.95 for comparative fit index and Tucker–Lewis index and values ≤0.08 for root mean square error of approximation based on the large sample size and under the assumption that these statistics are scaling-corrected (Hu & Bentler, 1998, 1999). Items with a factor loading on a single factor solution <0.4 were identified as poorly fitting (Brown, 2015).

The second stage involved testing whether the items exhibited local independence, that is, the assumption that items correlated with one another only through the shared relationship with the latent variable. To identify locally dependent items, modification indices of residual correlations were inspected. Modification indices associated with significant residual correlations between item pairs were identified and the content across the item pairs were inspected. Items with expected parameter change (fully standardized) ≥0.2 were identified as violating the assumption of local independence (Whittaker, 2012).

The third stage involved testing whether the underlying factor structure and the item response parameters estimated by the model were invariant on the basis of age, gender, and education. Evidence of invariance is a necessary prerequisite for group comparisons. Three key sociodemographic variables were tested using a multigroup CFA approach: age (18-55 years vs. 56+ years), gender (males vs. females), and education level (post–high school education vs. high school or lower). This approach proceeded in a series of steps by fitting nested models that estimated increasing levels of measurement invariance separately across age, gender, and education level. Specifically, nested configural (freely varying across groups) and scalar (constrained loadings and thresholds) models were compared using a chi-square difference test implemented by the DIFFTEST function in Mplus. A nonsignificant chi-square test (p > .05) would imply that the scalar model does not provide a significant decrement in model fit and all the items can be assumed invariant. Items with expected parameter change (fully standardized) ≥0.2 were identified as demonstrating noninvariance across age, gender or education (Whittaker, 2012). Data were then compared to the original U.S. calibration data for the PROMIS item banks (Pilkonis et al., 2011) accessed through Harvard Dataverse (https://dataverse.harvard.edu), with invariance by country tested using identical criteria. Demographic differences between the two samples were also assessed based on chi-square statistics.

The item banks were calibrated using the two-parameter graded response model suitable for ordinal data (Muthén & Muthén, 2010) by estimating a unidimensional CFA with a full information robust maximum likelihood estimator and a logit link function. This method takes into account the statistical similarities between IRT and factor analysis for ordered categorical items when converting item factor loadings and thresholds to the respective IRT discrimination and difficulty parameters (Takane & De Leeuw, 1987). As the sample was not representative of the general population, with higher levels of psychopathology and overrepresentation of females, a weighting scheme was applied to the data for the derivation of normative IRT estimates. This scheme was designed to make the sample more representative of the general population in terms of age, gender, and psychopathology distributions, to account for these potential imbalances (Batterham, Sunderland, Carragher, Calear, Mackinnon, et al., 2016). The weighting scheme used Australian data on the population prevalence of anxiety, affective, and substance use disorders, accounting for comorbidity between these disorder categories in each age and gender group. Data on mental disorders was obtained from the 2007 Australian National Survey of Mental Health and Wellbeing (Slade, Johnston, Oakley Browne, Andrews, & Whiteford, 2009), while data on age and gender distributions of the general population were obtained from Australian Bureau of Statistics population estimates (Australian Bureau of Statistics, 2016).

Finally, item banks were compared with DSM-5 criteria for MDD, GAD, and PD to test whether the item banks can provide robust screening for these specific disorders. The accuracy of the PROMIS items in identifying caseness for MDD, GAD, and PD (area under the receiver operating characteristic curve [AUC], sensitivity and specificity) was compared with the accuracy of the legacy scales (PHQ-9, GAD-7, PADIS). For comparability of length, existing short forms of the PROMIS depression and anxiety item banks (Pilkonis et al., 2011) were also tested against clinical criteria. Youden indices were used to identify optimal cut points where none were available.

Comparison of Time Frame

The 7- and 30-day time frame questions were compared using three approaches. First, correlations between the 7- and 30-day items were calculated, to assess whether correlations between equivalent items were at least as great as correlations between nonequivalent items. Second, differences in mean scores on each item were compared with identify whether there were systematically higher responses in the 30-day versions. Third, standardized factor loadings from confirmatory factor analyses were descriptively compared with identify whether items had a similar influence on the underlying construct in the 7- versus 30-day version of the item bank.

IRT analyses were conducted using Mplus version 7.2 (Muthén & Muthén, 2013). Remaining analyses were conducted using SPSS v23.0 (IBM Corp., Chicago, IL).

Results

Sample Characteristics

Table 1 displays the sociodemographic and mental health characteristics of the sample. There were elevated levels of psychopathology in the sample and females were over-represented compared to the general population. The weighting scheme reduced these biases, and was applied in the estimation of normative IRT parameters.

Table 1.

Characteristics of the Complete Sample (N = 3,175).

Characteristic	Unweighted		Weighted
Characteristic	Count	Percentage	Count	Percentage
Age (years)
18-25	371	12	442.3	14
26-35	277	9	582.7	18
36-45	506	16	558.6	18
46-55	830	26	549.6	17
56-65	842	27	522.6	17
≥66	349	11	519.2	16
Gender
Male	648	20	1556.9	49
Female	2,527	80	1618.1	51
Highest level of education
Less than high school	382	12	335.7	11
High school	440	14	507.1	16
Certificate/diploma	963	30	835.3	26
Bachelor degree	630	20	703.6	22
Higher degree	747	24	777.0	25
Prefer not to answer	13	<1	16.3	1
Employment status
Full-time	856	27	1001.5	32
Part-time/casual	819	26	712.9	23
Unemployed	383	12	358.4	11
Not working (study, maternity, retirement)	1,070	34	1052.8	33
Prefer not to answer	47	2	49.4	2
Location of residence
Metropolitan	1,421	45	1552.1	49
Regional	1,279	40	1204.4	38
Rural/remote	475	15	418.5	13
Language spoken at home
English only	2,962	93	2888.7	91
English and other	199	6	268.0	8
Other language only	14	<1	18.3	1
Met DSM-5 criteria for:
PD	186	6	72.7	2
MDD	362	11	201.7	6
GAD	645	20	244.8	8

Note. PD = panic disorder; MDD = major depressive disorder; GAD = generalized anxiety disorder.

Unidimensionality, Local Dependence, and Invariance of Item Banks

All three item banks showed strong evidence of unidimensionality. Single factors accounted for 71%, 63%, and 58% of the variance in depression, anxiety, and anger respectively. All items had significant loadings, with the depression item loadings >0.78, anxiety item loadings >0.61, and anger item loadings >0.63. The CFAs for the depression, anxiety and anger item banks had adequate fit, meeting criteria for scaling-corrected estimates of root mean square error of approximation (0.06, 0.06, and 0.05, respectively), comparative fit index (0.98, 0.97 and 0.95 respectively), and Tucker–Lewis index (0.98, 0.96, and 0.95, respectively). Across all three item banks, no modification indices had a standardized estimated parameter change ≥0.2, indicating that no items had local dependence.

Items showed invariance by age, gender and education. Although there were significant differences in fit between configural and scalar models, all items had a fully standardized expected parameter change <0.07 for age (18-55 years vs. 56+ years), <0.03 for gender (males vs. females), and <0.08 for education level (post–high school education vs. high school or lower), indicating negligible noninvariance.

The IRT parameters for the items in each bank are provided in Table 2, based on data weighted to the Australian population. Each item showed high discrimination, with estimates ranging from 1.29 to 5.34, and nonoverlapping difficulty parameters (overlapping difficulty parameters would indicate redundant response choices). The parameters were consistent with those reported in the original calibration of the item banks (Pilkonis et al., 2011), with higher discrimination parameters for the depression bank, followed by the anxiety and anger banks. However, in formal comparisons for invariance with the U.S. calibration sample, some items showed substantial noninvariance. Table 2 presents items ordered identically to the U.S. ordering, that is, by order of discrimination in the U.S. sample (Pilkonis et al., 2011). In the depression bank, items related to hopelessness (e.g., I felt helpless, I felt that I wanted to give up on everything) discriminated relatively better than depression, sadness, and unhappiness items in the Australian sample compared with the United States, with significant noninvariance identified between the two samples on some of these items as noted in the Table. Some items reflecting isolation or loneliness also demonstrated substantive noninvariance. Anxiety items were largely consistent with the U.S. calibration, with no items demonstrating noninvariance. In the anger item bank, a number of items showed noninvariance between Australia and the United States, although these items did not fit into distinct categories. In general, estimates of discrimination and thresholds were of the same magnitude in this sample as in the original calibration sample.

Table 2.

IRT Estimates Based on Weighted Data for the Three Item Banks.

	Slope (discrimination)	Location Threshold 1	Location Threshold 2	Location Threshold 3	Location Threshold 4
PROMIS Depression Item Bank
I felt hopeless	4.52	0.11	0.60	1.30	2.05
I felt depressed^a	3.55	−0.42	0.20	1.06	1.97
I felt worthless^a	4.72	0.16	0.65	1.22	1.90
I felt helpless^a	4.39	0.06	0.59	1.24	1.94
I felt like a failure^a	4.38	0.05	0.56	1.11	1.78
I felt that I had nothing to look forward to	4.66	0.05	0.57	1.15	1.81
I felt that nothing could cheer me up	4.99	0.23	0.72	1.30	2.11
I felt unhappy	3.57	−0.57	0.18	1.02	1.90
I felt sad	3.21	−0.64	0.15	1.01	1.92
I felt that I wanted to give up on everything^a	5.34	0.27	0.73	1.26	1.90
I felt that my life was empty	4.12	−0.01	0.53	1.15	1.92
I felt discouraged about the future	3.85	−0.33	0.24	1.01	1.81
I felt I had no reason for living	4.24	0.44	0.86	1.43	2.05
I found that things in my life were overwhelming	3.34	−0.06	0.50	1.18	1.95
I felt disappointed in myself	2.95	−0.42	0.26	1.03	1.70
I felt that I was not needed	2.99	0.07	0.59	1.24	2.03
I felt that nothing was interesting	3.28	0.15	0.70	1.41	2.24
I withdrew from other people^a	2.81	−0.15	0.44	1.11	2.08
I felt that I was to blame for things	2.77	0.20	0.76	1.42	2.06
I felt emotionally exhausted^a	2.71	−0.43	0.12	0.87	1.64
I had trouble making decisions	2.17	−0.46	0.49	1.58	2.60
I felt lonely^a	2.40	−0.52	0.14	0.99	1.71
I had trouble feeling close to people^a	2.87	−0.31	0.28	1.02	1.88
I felt upset for no reason	2.63	0.07	0.73	1.44	2.52
I felt pessimistic	3.38	−0.18	0.45	1.13	2.00
I felt ignored by people	2.58	−0.03	0.61	1.41	2.28
I felt that I was not as good as other people^a	2.66	0.04	0.51	1.26	1.90
I felt guilty	2.30	0.13	0.76	1.50	2.27
PROMIS Anxiety Item Bank
I found it hard to focus on anything other than my anxiety	3.30	0.14	0.89	1.78	2.68
My worries overwhelmed me	3.36	0.04	0.72	1.57	2.53
I felt uneasy	3.43	−0.50	0.37	1.37	2.44
I felt fearful	3.88	0.04	0.78	1.58	2.52
I felt like I needed help for my anxiety	3.63	0.28	0.70	1.52	2.24
I felt frightened	3.63	0.39	1.05	1.88	2.66
I felt nervous	3.55	−0.15	0.58	1.51	2.53
I felt anxious	3.86	−0.43	0.34	1.29	2.32
I felt tense	3.39	−0.54	0.28	1.20	2.30
It scared me when I felt nervous	3.21	0.57	1.20	1.96	2.56
I had difficulty calming down	3.06	0.02	0.85	1.90	2.88
Many situations made me worry	3.25	−0.13	0.60	1.45	2.40
I felt worried	3.13	−0.59	0.26	1.34	2.37
I had sudden feelings of panic	3.26	0.39	1.04	1.85	2.95
I felt something awful would happen	2.67	0.45	1.18	1.91	2.91
I was concerned about my mental health	2.46	0.09	0.62	1.42	2.30
I felt upset	2.81	−0.57	0.26	1.43	2.60
I felt terrified	3.08	0.87	1.55	2.24	3.08
I felt indecisive	2.07	−0.34	0.48	1.43	2.44
I had trouble relaxing	2.75	−0.54	0.20	1.06	2.05
I had trouble paying attention	2.01	−0.60	0.43	1.63	2.85
I felt fidgety	1.93	−0.10	0.84	1.90	3.00
I avoided public places or activities	2.08	0.02	0.65	1.39	2.39
I was anxious if my normal routine was disturbed	2.19	0.22	0.94	1.74	2.71
I was easily startled	1.84	0.23	0.95	1.71	2.47
I worried about other people’s reactions to me	2.08	−0.07	0.66	1.53	2.45
I had difficulty sleeping	1.34	−1.07	−0.08	1.09	2.09
I had a racing or pounding heart	1.68	0.18	1.05	2.13	3.24
I had twitching or trembling muscles	1.59	0.54	1.35	2.26	3.54
PROMIS Anger Item Bank
I was grouchy	2.22	−1.04	0.19	1.66	3.23
I stayed angry for hours	3.25	0.40	1.17	1.97	3.42
I felt angry	3.02	−0.54	0.45	1.64	3.08
I felt like I was ready to explode	3.30	0.37	1.15	1.87	3.14
I felt like I needed help for my anger	3.22	0.92	1.52	2.15	2.97
I felt angrier than I thought I should	3.32	0.50	1.06	1.98	3.18
I felt annoyed	2.28	−1.24	0.08	1.55	3.36
I made myself angry about something just by thinking about it	2.63	0.09	0.84	1.84	3.27
When I was angry, I sulked^a	2.18	0.52	1.30	2.37	3.46
I felt like breaking things	2.31	1.07	1.73	2.47	3.39
I was irritated more than people knew	2.44	−0.57	0.30	1.49	2.61
I felt like yelling at someone	2.66	0.02	0.72	1.68	2.75
Just being around people irritated me^a	2.19	0.13	0.89	1.92	3.00
I was stubborn with others	2.23	0.07	0.97	2.13	3.12
Even after I expressed my anger, I had trouble forgetting about it	3.35	0.25	0.83	1.63	2.60
I felt bitter about things	2.18	0.12	0.79	1.85	2.69
I felt resentful when I didn’t get my way	2.31	0.34	1.34	2.61	3.33
I held grudges toward others	2.14	0.38	1.28	2.19	3.19
I had a bad temper	2.99	0.38	1.23	1.97	2.90
I had trouble controlling my temper	3.41	0.70	1.52	2.15	3.05
I felt that people were trying to anger me	1.96	0.94	1.61	2.90	3.90
I felt guilty about my anger	2.24	0.39	0.93	1.90	2.88
I tried to get even when I was angry with someone	2.27	1.04	1.92	2.94	3.62
I was angry when something blocked my plans^a	2.22	0.56	1.39	2.58	3.87
I felt envious of others	1.29	0.28	1.33	2.65	4.28
When I was frustrated, I let it show^a	1.78	−0.06	1.00	2.37	3.69
I disagreed with people^a	1.42	−0.88	0.39	2.60	4.75
I was angry when I was delayed	2.06	0.15	1.09	2.15	3.14
When I was mad at someone, I gave them the silent treatment^a	1.57	0.22	1.10	2.21	3.19

Indicates items with substantial noninvariance (SEPC > 0.2) between the Australian and U.S. calibration samples.

Despite these similarities, there were also significant differences in the compositions of the two samples. The U.S. sample had underrepresentation of younger adults (19% aged 35 or younger in U.S. sample that completed one or more of the PROMIS emotional distress scales vs. 32% in the weighted Australian sample) and overrepresentation of older adults (46% vs. 33% in the U.S. vs. Australian samples aged 56 or older; χ² = 286.7, df = 5, p < .001). The U.S. sample also had slight overrepresentation of females (55% vs. 51% in the U.S. and Australian samples; χ² = 15.5, df = 1, p < .001), and underrepresentation of people with no high school qualification (2% vs. 11% in the U.S. vs. AU samples; χ² = 305.1, df = 3, p < .001). Based on T scores, scored on the basis of IRT estimates from the U.S. calibration sample with M = 50 and SD =10 in the U.S. general population, the weighted Australian sample had comparable scores on the depression (M = 51.0, SD = 10.8), anxiety (M = 48.6, SD = 10.2), and anger (M = 46.6, SD = 9.5) short form scales, although due to cross-national noninvariance, the T scores may not reflect the underlying population distributions in Australia.

Precision of the short forms, full item banks, and legacy scales across the spectrum of symptom severity was assessed using information curves. Figures 1 and 2 indicate that the full item banks had good precision across the spectrum from approximately two SD below the population mean up to four SD above the mean. This is to be expected due to the larger number of items. However, comparing the seven-item short forms and the legacy scales of 7 to 9 items (PHQ-9, GAD-7) it is evident that the short forms had greater overall precision across the spectrum, particularly in the upper tail where risk of disorder is highest.

Figure 1.

Information curves for the full PROMIS depression item bank (28 items), short form PROMIS depression item bank (7 items), and PHQ-9 (9 items).

Figure 2.

Information curves for the full PROMIS anxiety item bank (28 items), short form PROMIS anxiety item bank (7 items), and GAD-7 (7 items).

Accuracy in Screening for Clinical Criteria

Total raw scores (summed responses of all items in the bank) for the depression and anxiety PROMIS item banks (original 7-day versions) were compared to existing screeners for MDD, GAD, and PD against clinical criteria. Table 3 presents the AUC and sensitivity and specificity at prescribed cut points for the PROMIS item banks, short forms, and legacy scales (PHQ-9, GAD-7), compared with DSM-5 caseness for depression, GAD, and panic disorder. The AUC for the PROMIS depression bank in assessing MDD was significantly poorer than the AUC for the PHQ-9 (χ²₁ = 8.2, p =.004). However, there was no significant difference in the performance of the PROMIS anxiety bank in assessing clinical GAD relative to the GAD-7 (χ²₁ = 2.3, p = .133). In assessing PD, the performance of the PROMIS anxiety bank was not significantly different from the PADIS (χ²₁ = 1.07, p = .301) or the GAD-7 (χ²₁ = 3.79, p = .052) in identifying PD. A raw (sum) score ≥87 on the PROMIS depression item bank had 89% sensitivity and 83% specificity for MDD. A raw score ≥66 on PROMIS anxiety had 83% sensitivity and 78% specificity for GAD, while a raw score ≥74 on the same item bank had 82% sensitivity and 76% specificity for PD.

Table 3.

Performance of Item Banks, Short Forms, and Legacy Scales Against DSM-5 Clinical Criteria.

	AUC	Cut point	Sensitivity	Specificity
DSM-5 depression caseness
PROMIS depression item bank	0.925	≥87	0.894	0.825
PROMIS depression short form	0.919	≥25	0.894	0.807
PHQ-9	0.946	≥15	0.894	0.866
DSM-5 GAD caseness
PROMIS anxiety item bank	0.882	≥66	0.830	0.784
PROMIS anxiety short form	0.877	≥18	0.821	0.772
GAD-7	0.895	≥8	0.817	0.826
DSM-5 panic disorder caseness
PROMIS anxiety item bank	0.841	≥74	0.817	0.756
PROMIS anxiety short form	0.832	≥19	0.833	0.721
GAD-7	0.813	≥8	0.783	0.724
PADIS	0.885	≥4	0.855	0.778

Note. DSM-5 = Diagnostic and Statistical Manual of Mental Disorders, 5th edition; AUC = area under the curve; PROMIS® = Patient-Reported Outcomes Measurement Information System; PHQ = Patient Health Questionnaire; GAD = Generalized Anxiety Disorder scale.

Performance of existing PROMIS short forms was also assessed as they are of comparable length to the legacy scales. Although the 8-item short form PROMIS depression measure was accurate in identifying MDD caseness, it remained significantly less accurate than the PHQ-9 (χ²₁ = 10.5, p = .001). The 7-item short form PROMIS anxiety measure performed well and not significantly less accurately than the GAD-7 in detecting GAD (χ²₁ = 3.27, p = .071). Furthermore, the short form PROMIS anxiety measure was accurate in detecting PD, with operating characteristics not significantly different to the PADIS (χ²₁ = 1.8, p = .180) or the GAD-7 (χ²₁ = 1.75, p = .185) in detecting PD.

Time Frame Comparison

The correlational analysis demonstrated strong associations between 7- and 30-day versions of the PROMIS items, with correlations ranging from 0.79 to 0.90 for depression items, 0.84 to 0.89 for anxiety items, and 0.71 to 0.82 for anger items. Each 7-day item had a stronger correlation with the identical 30-day item than with any other item, with the single exception of the depression item “I felt fearful,” which had a correlation of 0.80 with both “I felt fearful” and “I felt frightened” using the 30-day timeframe.

Raw scores on the 30-day versions of the item banks were consistently higher than on the 7-day versions, indicating systematically greater endorsement of all 30-day items. Depression 30-day raw scores (M = 68.3, SD = 31.4) were significantly greater than 7-day raw scores (M = 63.5, SD = 31.9, t = 22.8, df = 2151, p < .001), as were anxiety raw scores (M = 62.8, SD = 25.4 vs. M = 59.1, SD = 26.0, t = 16.2, df = 1168, p < .001) and anger raw scores (M = 57.4, SD = 21.7 vs. M = 50.6, SD = 21.6, t = 24.3, df = 1095, p < .001).

CFAs indicated that both timeframe versions of each item bank performed similarly, with item loadings within ±0.05 for each item, and comparable proportion of variance explained by a single factor (70% vs. 73% for depression 30- and 7-day; 61% for both anxiety versions; and 56% for both anger versions).

Discussion

The findings indicate that the PROMIS emotional distress item banks provided robust measures of depression, anxiety and anger in an Australian population sample. The measures demonstrated evidence of unidimensionality, local independence, and invariance across age, gender, and education. These findings provide support for the validity of the PROMIS item banks in measuring emotional problems in the general population, providing confidence that the item banks are appropriate for use outside the United States and in samples that are diverse in terms of age, gender, and education. Some items were shown to exhibit substantive cross-national variations. In particular, hopelessness-related items tended to discriminate better in the Australian sample than more general depression/sadness items, relative to the original U.S. calibration sample, while items reflecting loneliness or isolation also performed differently in Australia and the United States. Similarly, some of the anger items showed noninvariance, although there was no consistent category of anger items that displayed noninvariance between countries. The observed differences may be due to cultural differences in the understanding of terms or in the expression of emotional states (Lange, Thalbourne, Houran, & Lester, 2002). Alternatively, the differences may reflect variation in the sampling methodology, with the original PROMIS calibration study using online market research panels supplemented with clinical subgroups (Pilkonis et al., 2011). The compositions of the sample reflected the different sampling approaches, with differences in age, gender and education, which may have been the source of the cross-national noninvariance. Notwithstanding these differences, the overall consistency of the item banks with the original calibration, and performance relative to legacy scales, provides confidence that they deliver a robust assessment of the constructs of depression, anxiety, and anger in the general population.

The item banks also performed well as indicators for clinical caseness, with high sensitivity and specificity in detecting MDD, GAD, and PD. The legacy scales (particularly PHQ-9) tended to have significantly stronger operating characteristics in detecting clinical caseness than the PROMIS item banks or short forms. However, items from the legacy scales were designed specifically to assess DSM-IV criteria, so it is unsurprising that they performed incrementally better than the PROMIS measures in detecting disorder. Nevertheless, information curves indicated that the PROMIS item banks and short forms tended to provide more precise measurement across the spectrum of symptom severity. The PROMIS measures may therefore have greater precision in capturing severity, while providing an accurate indicator of clinical caseness and need for clinical intervention. Further exploration of item banks specifically for panic disorder and other anxiety disorders may be warranted (Batterham, Sunderland, Carragher, & Calear, 2016).

The comparison of time frames indicated that corresponding items are rated comparably regardless of the time frame, with the only clear difference that longer time frames result in higher levels of item endorsement. These findings are consistent with previous research in the mental health domain (e.g., Watson, 1988) and with other PROMIS item banks (Lai, Cook, Stone, Beaumont, & Cella, 2009), and suggest that either of these time frames is likely to provide an accurate estimate of symptoms across the severity spectrum. Consequently, the choice of the most appropriate time frame may be based on pragmatic considerations. These considerations may include the period over which assessment is conducted (particularly for repeated assessments), the period where change may feasibly be expected to occur, and consistency with clinical definitions of psychopathology. Nevertheless, researchers and clinicians should be aware that symptoms that occur infrequently may not be endorsed as commonly when using a briefer time frame relative to a longer time frame.

This study provided unique data on the psychometric properties of the PROMIS emotional distress item banks in a large non–U.S. population-based adult sample. It also demonstrated that the item banks can be used to identify whether an individual is likely to meet criteria for MDD, GAD, or PD, and that the time frame used to assess symptoms has minimal impact on the properties of the measures. However, there were some limitations that could not be addressed in the study. The sample consisted of self-selected Facebook users, who were not representative of the general population in terms of gender or psychopathology. Traditional population recruitment tends to suffer from similar problems with representativeness of samples (Batterham, 2014). Although the weighting scheme largely accounted for overrepresentation of females and people with mental health problems, further testing in other population-based samples would provide additional confidence in the validity of the measures. In addition, the standard used to assess clinical criteria was a self-report checklist, due to the practical and financial constraints involved in conducting a large-scale population-based assessment. The checklist systematically assessed all criteria for MDD, GAD, and PD based on DSM-5, similar to other tools such as the MINI (Zbozinek et al., 2012). Nevertheless, self-report of DSM-5 criteria may not be consistent with clinician-administered measures, so clinician diagnosis would provide a more rigorous standard for diagnostic comparison. Further investigation of the reliability of the measures over time may provide additional support for their use.

The present study provided further evidence that the PROMIS emotional health item banks provide robust measures of depression, anxiety, and anger in the general population. These measures also provide a satisfactory indication of whether an individual is likely to meet clinical criteria and therefore may serve dual purposes of both assessing emotional health on a continuum and assessing risk of clinical states. Finally, the time frame comparison provided evidence that changing the time frame of a mental health measure does not typically affect the psychometric properties of the measure, although symptom endorsement will typically be more common over longer periods. This finding suggests that the time frame selected in assessing mental health outcomes should be based on pragmatic considerations.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: PJB, MS, and ALC are supported by National Health and Medical Research Council (NHMRC) Fellowships 1083311, 1052327, and 1013199, respectively. The study was funded by NHMRC Project Grant 1043952.

Supplemental Material

Supplemental material for this article is available online.

References

American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorders (4th ed.). Washington, DC: Author.

American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Washington, DC: Author.

Australian Bureau of Statistics. (2016). 3101.0 Australian demographic statistics. Retrieved from http://www.abs.gov.au/ausstats/abs@.nsf/mf/3101.0

Batterham

P. J.

(2014). Recruitment of mental health survey participants using Internet advertising: content, characteristics and cost effectiveness. International Journal of Methods in Psychiatric Research, 23, 184-191. doi:10.1002/mpr.1421

Batterham

P. J.

Brewer

J. L.

Tjhin

Sunderland

Carragher

Calear

A. L.

(2015). Systematic item selection process applied to developing item pools for assessing multiple mental health problems. Journal of Clinical Epidemiology, 68, 913-919. doi:10.1016/j.jclinepi.2015.03.022

Batterham

P. J.

Mackinnon

A. J.

Christensen

(2015). The Panic Disorder Screener (PADIS): Development of an accurate and brief population screening tool. Psychiatry Research, 228(1), 72-76. doi:10.1016/j.psychres.2015.04.016

Batterham

P. J.

Sunderland

Carragher

Calear

A. L.

(2016). Development and community-based validation of eight item banks to assess mental health. Psychiatry Research, 243, 453-462. doi:10.1016/j.psychres.2016.07.011

Batterham

P. J.

Sunderland

Carragher

Calear

A. L.

Mackinnon

A. J.

Slade

(2016). The Distress Questionnaire-5: Population screener for psychological distress was more accurate than the K6/K10. Journal of Clinical Epidemiology, 71, 35-42. doi:10.1016/j.jclinepi.2015.10.005

Brown

T. A.

(2015). Confirmatory factor analysis for applied research. New York, NY: Guilford Press.

10.

Cella

Yount

Rothrock

Gershon

Cook

Reeve

, . . . PROMIS Cooperative Group. (2007). The Patient-Reported Outcomes Measurement Information System (PROMIS): Progress of an NIH Roadmap cooperative group during its first two years. Medical Care, 45(5 Suppl. 1), S3-S11. doi:10.1097/01.mlr.0000258615.42478.55

11.

Choi

S. W.

Reise

S. P.

Pilkonis

P. A.

Hays

R. D.

Cella

(2010). Efficiency of static and computer adaptive short forms compared to full-length measures of depressive symptoms. Quality of Life Research, 19, 125-136. doi:10.1007/s11136-009-9560-5

12.

First

M. B.

Pincus

H. A.

Levine

J. B.

Williams

J. B.

Ustun

Peele

(2004). Clinical utility as a criterion for revising psychiatric diagnoses. American Journal of Psychiatry, 161, 946-954. doi:10.1176/appi.ajp.161.6.946

13.

L. T.

Bentler

P. M.

(1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological methods, 3, 424-453.

14.

L. T.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55. doi:10.1080/10705519909540118

15.

Lai

J. S.

Cook

Stone

Beaumont

Cella

(2009). Classical test theory and item response theory/Rasch model to assess differences between patient-reported fatigue using 7-day and 4-week recall periods. Journal of Clinical Epidemiology, 62, 991-997. doi:10.1016/j.jclinepi.2008.10.007

16.

Lange

Thalbourne

M. A.

Houran

Lester

(2002). Depressive response sets due to gender and culture-based differential item functioning. Personality and Individual Differences, 33, 937-954. doi:10.1016/S0191-8869%2801%2900203-3

17.

Luhmann

Hawkley

L. C.

Eid

Cacioppo

J. T.

(2012). Time frames and the distinction between affective and cognitive well-being. Journal of Research in Personality, 46, 431-441. doi:10.1016/j.jrp.2012.04.004 Personality Psychology [3100]

18.

Muthén

L. K.

Muthén

B. O.

(2010). Mplus user’s guide (6th ed.). Los Angeles, CA: Muthén & Muthén.

19.

Muthén

L. K.

Muthén

B. O.

(2013). Mplus (Version 7.2) [Computer software]. Los Angeles, CA: Muthén & Muthén

20.

Pilkonis

P. A.

Choi

S. W.

Reise

S. P.

Stover

A. M.

Riley

W. T.

Cella

, PROMIS Cooperative Group. (2011). Item banks for measuring emotional distress from the Patient-Reported Outcomes Measurement Information System (PROMIS): Depression, anxiety, and anger. Assessment, 18, 263-283. doi:10.1177/1073191111411667

21.

Ready

R. E.

Robinson

M. D.

Weinberger

(2006). Age differences in the organization of emotion knowledge: Effects involving valence and time frame. Psychology and Aging, 21, 726-736. doi:10.1037/0882-7974.21.4.72617201493

22.

Slade

Johnston

Oakley Browne

M. A.

Andrews

Whiteford

(2009). 2007 National Survey of Mental Health and Wellbeing: Methods and key findings. Australian and New Zealand Journal of Psychiatry, 43, 594-605. doi:10.1080/00048670902970882

23.

Spitzer

R. L.

Kroenke

Williams

J. B.

(1999). Validation and utility of a self-report version of PRIME-MD: the PHQ primary care study. Primary Care Evaluation of Mental Disorders. Patient Health Questionnaire. Journal of the American Medical Association, 282, 1737-1744.

24.

Spitzer

R. L.

Kroenke

Williams

J. B. W.

Lowe

(2006). A brief measure for assessing generalized anxiety disorder: The GAD-7. Archives of Internal Medicine, 166, 1092-1097.

25.

Takane

De Leeuw

(1987). On the relationship between item response theory and factor analysis of discretized variables. Psychometrika, 52, 393-408.

26.

Terry

P. C.

Stevens

M. J.

Lane

A. M.

(2005). Influence of response time frame on mood assessment. Anxiety, Stress & Coping: An International Journal, 18, 279-285. doi:10.1080/10615800500134688

27.

Vilagut

Forero

C. G.

Adroher

N. D.

Olariu

Cella

Alonso

, INSAyD Investigators. (2015). Testing the PROMIS(R) Depression measures for monitoring depression in a clinical sample outside the US. Journal of Psychiatric Res, 68, 140-150. doi:10.1016/j.jpsychires.2015.06.009

28.

Watson

(1988). The vicissitudes of mood measurement: Effects of varying descriptors, time frames, and response formats on measures of positive and negative affect. Journal of Personality and Social Psychology, 55, 128-141. doi:10.1037/0022-3514.55.1.1283418487

29.

Wells

J. E.

Horwood

L. J.

(2004). How accurate is recall of key symptoms of depression? A comparison of recall and longitudinal reports. Psychological Medicine, 34, 1001-1011.

30.

Whittaker

T. A.

(2012). Using the modification index and standardized expected parameter change for model modification. Journal of Experimental Education, 80(1), 26-44.

31.

Zbozinek

T. D.

Rose

R. D.

Wolitzky-Taylor

K. B.

Sherbourne

Sullivan

Stein

M. B.

. . . Craske

M. G.

(2012). Diagnostic overlap of generalized anxiety disorder and major depressive disorder in a primary care sample. Depression and Anxiety, 29, 1065-1071. doi:10.1002/da.22026

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.19 MB