Measuring Cumulative Stressfulness: Psychometric Properties of the COVID-19 Stressors Scale

Abstract

Scales assessing stressor exposure often fail to demonstrate adequate psychometric qualities, demonstrating low interitem reliability or complex factor structures, as would be expected, given that the majority of stressors are independent events. However, in large-scale mass crisis events, the stressors may be highly interrelated, indicating shared experience. Furthermore, few stressor exposure scales also measure appraised stressfulness of those stressors. Development of a psychometrically sound measure of both stressor exposure and appraisal advances the study of highly stressful events such as community-wide crises, especially in providing a useful measure of its cumulative stressfulness. The coronavirus disease 2019 (COVID-19) pandemic is an enduring, worldwide stressor with an indefinite timeline. The COVID-19 Stressor Scale is a 23-item measure of stressor exposure and appraisal related to the pandemic developed within the first weeks of widespread shelter-in-place practices in the Unites States. We present initial psychometric results of the COVID-19 Stressor Scale. Results of a principal components analysis indicate that the measure is unidimensional and has strong internal consistency. Evidence of convergent and discriminant validity were demonstrated. The COVID-19 Stressor Scale is a useful measure for studying the ongoing stressors associated with the pandemic and presents a model for measuring other massive, ongoing crises.

Keywords

COVID-19 measurement mental health psychological distress stress

The first documented cases of coronavirus disease 2019 (COVID-19; severe acute respiratory syndrome coronavirus 2 [SARS-Cov-2]) were disclosed to the public in the United States in January 2020. While it is possible that cases were present in the United States earlier, the American public gained awareness of the presence of COVID-19 in the country in early 2020. By March, the spread of the disease altered daily routines nationwide and was quickly classified as a global pandemic by the World Health Organization (2020). COVID-19 is a highly communicable but preventable novel coronavirus with a mortality rate that surpassed 103,500 deaths in the United States by June 1, 2020, exceeding rates seen during other recent flu and SARS epidemics (Centers for Disease Control and Prevention, 2020; Layne et al., 2020; Solomon, 2020; Stokes et al., 2020). Recent research identifies COVID-related stressors that include fears of infection, disruptions to work/learning and daily self-care routines, and lack of access to reliable information and resources (Brooks et al., 2020; Main et al., 2011; Qiu et al., 2020). The individual and family mental health consequences of crises spanning natural disasters, such as earthquakes or floods, and terrorist attacks experienced community wide are extensive and widespread (American Psychological Association, 2020; Baral & Bhagawati, 2019; Park et al., 2012). Sequelae of these community-wide events include a variety of mental health symptoms that may make both mental health–coping strategies and viral transmission prevention guidance difficult to implement—in particular, social distancing measures including quarantine may be crucial for slowing the spread of an infectious disease but are also associated with anxiety, loneliness, and isolation with lasting mental health impacts (Galea & Coffey, 2018; Solomon, 2020). Experts suggest that the COVID-19 pandemic is an instance of traumatic stress, likely worsening existing mental health difficulties for some and initiating new disorders in others (Horesh & Brown, 2020).

Substantial literature indicates that mass crises across a range of event durations, including earthquakes (Seto et al., 2019), hurricanes (Fussell & Lowe, 2014), nuclear disasters (Bolt et al., 2018), wildfires (Sprague et al., 2015), and acts of terror (Jose et al., 2019) have long-term sequelae, and it is likely that the COVID-19 pandemic’s impact on mental health will be similarly long-lasting. Mental health outcomes, including posttraumatic stress, anxiety, and difficulties coping, are common following exposure to mass crises (American Psychological Association, 2020; Bolt et al., 2018; Jose et al., 2019; Labarda et al., 2020; North & Pfefferbaum, 2013; Seto et al., 2019). Importantly, even a short-term event can have many long-term outcomes for mental health and coping behaviors, as noted in studies of short-duration natural hazards such as earthquakes (Baral & Bhagawati, 2019). In the case of the enduring pandemic, the pathogenic effects of stress over time may be amplified, because the stressful conditions of the community-wide crisis (e.g., the threat of infection and the impacts of social distancing/quarantine and related financial consequences) are sustained over weeks and months. For example, the COVID-19 pandemic has seen widespread surges in local infection rates that first peaked in April 2020, then rose again in June and July with over 150,000 deaths reported in the United States by August 1, 2020 (Johns Hopkins University Center for Systems Science and Engineering, 2020); concurrently, there are record high rates of unemployment now documented above 14% (U.S. Bureau of Labor, 2020).

An important determinant of mental health outcomes is total exposure (e.g., proximity to the threat in geographically restricted crises, pervasiveness of impacts, or number of negative consequences) to the stressor. However, relatively few studies have attempted to quantify these concurrent stressor exposure characteristics (e.g., Garfin et al., 2020) to examine more carefully the facets of a stressful experience; frequently exposure assessments rely on dichotomous counts of discrete events across a given timeframe (e.g., the commonly used Life Events Checklist; Gray et al., 2004). The appraised stressfulness of the event—that is, how people view the impacts of a mass crisis on their own life—is a separate and significant determinant of long-term mental health outcomes. Lazarus and Folkman’s (1984) theory of stress and coping posited that individuals’ perception, or appraisal, of a stressor determines the positive or negative impacts they may experience. Thus, it is not just the presence or accumulation of exposures to stressful events, but the context-dependent, evaluative judgments of the situation and its consequences that shape reactions to each stressor (Epel et al., 2018). For example, results from a study of the New Zealand Christchurch earthquake in 2016 reported that associations between earthquake exposure and major depression was largely explained by the experience of peritraumatic distress (i.e., symptoms of distress during exposure to a severe stressor such as horror, fear for safety, grief; Brunet et al., 2001) during the earthquakes (Bell et al., 2017). Reports from China from the early months of the COVID-19 pandemic suggest only moderate impacts on mental health (Qiu et al., 2020; Zhang & Ma, 2020), such that while fewer than one third of one national sample reported elevated depression, anxiety, or stress, more than half of respondents reported severe peritraumatic stress (Wang et al., 2020). Based on literature (e.g., Galea & Coffey, 2018; Galea et al., 2020; North & Pfefferbaum, 2013), which suggests that early experiences of maladjustment lead to subsequent long-term difficulties, any indication of early distress during the COVID-19 pandemic may foreshadow subsequent increases in psychopathology.

Because both the exposures to aspects of the pandemic and the perceived stressfulness of these exposures are likely important determinants of long-term mental health outcomes, it is critical to capture both elements and, ideally, to quantify them for measurement purposes.

Measurement Considerations

Studies of psychological and environmental stressors often focus on objectively stressful events that exceed available coping resources, such as bereavement or unemployment, and may focus on a single experience or the cumulative effects of several (e.g., life event scales that measure the stressfulness of a range of possible events). Event-specific approaches are useful as they assess the relative risk of events and their pathogenic sequelae while recognizing that responses to stress are necessarily influenced by personal and contextual factors through which events are interpreted (Cohen et al., 1983; Epel et al., 2018). An event-specific approach requires measuring individual appraisal of stressfulness (i.e., individuals’ perceptions of the degree to which the exposure is stressful) beyond simple counts of events and has proven a more reliable predictor of mental health outcomes than a cumulative event total (Cohen et al., 1983; Epel et al., 2018). A persistent criticism of global stressor scales is that event items may not be associated with one another (i.e., if your partner dies in a car accident, that does not make you more likely to have cancer). Often, life event scales are used to categorize participants (e.g., car accident group, bereaved group, wildfire group) and for this type of analysis internal consistency is not necessary. However, similar stressor events are likely to have elements of consistency. For example, crises related to diseases may be hypothesized to be similar to one another, but dissimilar from natural hazards such as wildfires or earthquakes. Unfortunately, measures developed are often specific to a single event (e.g., Galea et al., 2007), and psychometric evaluation of measures is rare. With regard to the COVID-19 pandemic, the proliferation of ad hoc measures lacking sufficient scale development is concerning. The authors attempt to remedy this lack of unsound measures by psychometrically evaluating one measure. The present work is an important step in the development of a measure of crisis stressfulness.

Few psychometrically sound measures related to mass crises that quantify the exposures to the event—and the perceived stressfulness of these exposures—exist (one commonly used example of subjective distress from unspecified traumatic events is the Impact of Event Scale; Weiss & Marmar, 1996). The COVID-19 Stressors Scale (Park et al., 2020) is a disaster-specific measure of stressor exposure and appraisal developed to quantify stressfulness of the COVID-19 pandemic. The measure was developed to include a set of items for each of the dominant themes identified in studies of previous SARS outbreaks and the early months of the COVID-19 pandemic in China (financial stressors or fear of infection, for example; Brooks et al., 2020; Main et al., 2011; Qiu et al., 2020). The current study aimed to assess the psychometric properties of the COVID-19 Stressor Scale and to provide preliminary evidence regarding its utility.

Current Study

Based on the above evidence, the present study examined the psychometric properties of a novel stress exposure scale developed during the first few weeks into the widespread U.S. quarantine and shelter-in-place orders in response to the spread of COVID-19. We aimed to demonstrate the scale’s preliminary reliability and validity from a national sample of adult participants who completed a survey about the pandemic between April 27 and 28, 2020.

Method

Participants

Parents 18 years or older living in the United States who are English-speaking were solicited to participate in an anonymous survey through Amazon’s Mechanical Turk (MTurk). MTurk is an online worker platform in which “employers,” in this case, the researchers, create an online job posting, or a Human Intelligence Task, or HIT (Sheehan, 2018). The HIT in this study sought workers who met the basic qualifications to pass the inclusion criteria for the research (i.e., were over 18 and reported a child under 18 residing in their home) to complete a 30-minute survey. MTurk has increased in popularity as a sampling mechanism in behavioral and public health fields, in part because MTurk workers are considered fairly representative of the characteristics of larger populations (Bartneck et al., 2015; Sheehan & Pittman, 2016), and research has found this method of recruitment to be replicable and valid (Mortensen & Hughes, 2018).

The protocol for the study was reviewed and approved by the University of Connecticut Institutional Review Board (No. X20 0075). The researchers posted the HIT on MTurk April 27, 2020, and solicited the sample via the platform. Interested candidates reviewed a brief explanation of the study protocol and followed a link to the assent page for the survey. Participants were given a $3 incentive on completion of the survey. Guidelines for online survey data management recommend eliminating cases in which substandard completion of the survey is suspected, by either pattern of response or time to completion (Kees et al., 2017; Sheehan, 2018). To determine appropriate response time for inclusion, average response time in seconds was calculated, along with percentiles for response time. Any entry that fell below the 10th percentile or above the 90th percentile was examined by the first author. Two cases were deleted due to a response time outside of that considered acceptable (one in excess of 50 hours, and one less than 5 minutes; average time to completion was approximately 34 minutes). The final sample consisted of 437 respondents with an average age of 35.72 years (SD = 8.66, range = 18–72 years); 47.8% were female (n = 208). The racial and ethnic composition of the sample includes the following: 80.5% non-Latinx, 19.5% Latinx, 10.6% Black, 17.2% Asian/Asian American, 3.2% Native Hawaiian/other Pacific Islander, 9.4% American Indian/Alaskan Native, 71.7% White.

Data Sufficiency and Cleaning

While there is a lack of consensus regarding the acceptable sample size for a principal component analysis/exploratory factor analysis, some have offered suggestions. Some (e.g., MacCallum et al., 1999) have suggested that the overall n is important, while others (e.g., Velicer & Fava, 1998) have argued that the case-to-variable ratio is most important. We considered both sets of recommendations in evaluating sufficiency of the sample size. First, many recommend sample sizes in the range of 100 to 300. MacCallum et al. (1999) echo the classic work of Gorsuch (1983) and recommend a minimum n of 100. Hutcheson and Sofroniou (1999) recommend a sample of 150 to 300. In the case-to-variable category, Velicer and Fava (1998) recommend a ratio of 10:1, while others (Bryant & Yarnold, 1995) recommend a ratio of 5:1. Regardless of method of sufficiency evaluation our sample is sufficiently large for the intended analysis. The n of 437 for 23 items is both above the n > 300 threshold as well as the 10:1 ratio threshold. Missing data were examined and are assumed to be missing completely at random given the lack of interpretable pattern in the missingness.

Measures

COVID-19-Specific Stressors

Inspired by noteworthy themes from research during previous SARS outbreaks and the early months of the COVID-19 pandemic in China (Brooks et al., 2020; Main et al., 2011; Qiu et al., 2020), we used a disaster-specific measure of stressor exposure and appraisal (COVID-19 Stressors Scale; Park et al., in press). This measure assesses stressfulness with binary (yes/no) past-week exposure to 23 stressors conceptually grouped into (1) infection-related stressors, (2) daily activity stressors, and (3) financial/resource-related stressors. For each item participants indicate “yes,” a subsequent question assessed the event’s stressfulness using a Likert-type rating scale from 1 to 5 (“not at all stressful” to “extremely stressful”). Thus, the scale yields count of stressor exposures (Park et al., 2020) and subsequent stress appraisal score (Park et al., 2020) for each of the conceptual domains. Said another way, the COVID-19 Stressors Scale includes 23 paired items—the first item in the pair is a binary response event item, phrased as “have you experienced/been . . . ” If respondents selected yes, branch logic took the respondent to a stress appraisal question phrased as “how stressful was . . . ” rated on a 5-point scale as described above. For the present study, a composite variable was computed in the following fashion: binary ratings were coded as 0/1 and multiplied by appraisal scores (1–5). Thus, the resulting variable had a range from 0 (indicating the respondent did not experience the event and thus reported no stress) to 5 (indicating that the respondent had the experience and that it was appraised as highly stressful).

Perceived Stress Scale

The Perceived Stress Scale (PSS; Cohen et al., 1983) was used as a measure of convergent validity. The measure assesses levels of global perceived stress reported by respondents. Responses are measured on a 5-point Likert-type scale (“never” to “very often”). Sum scores are created, such that higher scores indicate more stress. In previous studies (Cohen et al., 1983), and in this study, the scale demonstrated good internal consistency reliability (current study α = .81).

Generalized Anxiety Disorder Scale

The Generalized Anxiety Disorder Scale-7 is a seven-item measure of the severity of anxiety symptoms (Spitzer et al., 2006). This measure was used as an indicator of convergent validity. Respondents answer on a 0 to 4 scale from “Not at all” to “Nearly every day” to rate the frequency of their anxiety symptoms over the past 2 weeks. Scores range from 0 to 21 with higher scores indicating greater levels of anxiety. The scale has good reported internal consistency reliability in past examinations (α = .90; Spitzer et al., 2006) and in the present study, α = .93.

Results

First, to ensure that transformation of scores on the measure was consistent with both binary event response items and stressfulness appraisal items, correlations were examined among sum scores on the binary event response items, appraisal items, and transformed items. Correlations were all significant and positive, as expected. Binary event scores were significantly correlated with appraisal, r = .651, p < .001, and transformed scores, r = .673, p < .001. As expected due to the transformation employed, appraisal scores were highly correlated with transformed scores (r = .996, p < .001). These data suggest that the data were maintained as scores were transformed.

Distribution of Scores on the COVID-19 Stressors Scale

A preliminary examination of the distribution of scores on the COVID-19 Stressor Scale was conducted to better understand the pattern of responses. Scores on each item ranged from 0 to 5. For all but one item, modal scores were 0. Item 8, “Uncertainty about how long quarantine and/or social distancing requirements will last,” had a modal value of 3, indicating this was an item more commonly reported as stressful by respondents. Distributional data regarding means and modes for each item are presented in Table 1.

Table 1.

Means and Modes of the COVID-19 Stressors Scale Items.

Items	M	SD	Mode
Infection-related stressors
1. Risk of becoming infected	2.29	1.80	0
2. Self-monitoring of symptoms	1.75	1.50	0
3. Risk of loved ones becoming infected	2.50	1.88	0
4. Risk of unintentionally infecting other people	1.66	1.88	0
5. Read about or heard others talking about the severity and contagiousness of COVID-19	2.17	1.77	0
6. Stigma, shame, discrimination, or social exile related to quarantine or working in a high-risk area (e.g., others shunning you because you work in health care)	0.81	1.59	0
7. Stigma, shame or discrimination related to being in a certain age-group (e.g., negative statements about Millennials or Generation Z)	0.67	1.46	0
8. Uncertainty about how long quarantine and/or social distancing requirements will last	2.48	1.73	3
Daily routine–related stressors
9. Changes to daily personal care routines (e.g., cooking, cleaning, exercise/relaxation, hobbies)	1.60	1.75	0
10. Changes to daily work routines (e.g., unable to earn money, attend full- or part-time work schedule)	1.80	1.95	0
11. Changes to daily education routines (e.g., online instruction)	1.63	1.82	0
12. Changes to social routines (e.g., spending free time with friends/loved ones)	1.87	1.79	0
13. Changed responsibilities to care for dependents (e.g., childcare, eldercare)	1.74	1.82	0
14. Cancellation of planned or scheduled celebrations, entertainment, vacations or trips (e.g., graduations, birthdays, concerts)	1.67	1.78	0
15. Cancellation of meaningful personal or religious rituals (e.g., funerals, religious services)	1.01	1.67	0
16. Inability to travel (e.g., cancellation of vacations, weekend trips)	1.44	1.74	0
17. Increased contact with close others or loved ones (e.g., increased conflict, co-worrying)	1.30	1.76	0
18. Pressure to “make the most of” COVID-19 or “find a silver lining” while quarantining (e.g., social media fitness challenges, encouragement to increase productivity)	1.07	1.61	0
Resource-related stressors
19. Loss of current job security or income (e.g., inability to earn money)	1.51	1.98	0
20. Loss of current job training opportunities or education benchmarks (e.g., certification, apprenticeship, internship or degree completion)	0.89	1.67	0
21. Potential changes to the national or global economy (e.g., future job prospects, loss of investments)	2.04	1.82	0
22. Difficulty accessing important resources for daily life (e.g., health care, food, clothes, water, housing, medical supplies or prescriptions)	1.75	1.85	0
23. Inadequate access to reliable information about COVID-19 (including your personal risk of illness)	1.06	1.68	0

Note. COVID-19 = coronavirus disease 2019.

Exploratory Factor Analysis

A principal components analysis (PCA) for categorical variables was conducted using principal-axis normalization to explore the factor structure of the scale. Principal components factor extraction techniques are recommended in exploratory and descriptive research (Tinsley & Tinsley, 1987). Items were ordinal in nature. That is, though the positions on the scale are monotonic, their ranges are not well-defined and are not expected to represent numerically uniform increments. Thus, categorical PCA is required to accommodate the ordinal nature of the data. In categorical PCA, ordinal variables are monotonically transformed into their interval versions in order to maximize the variance explained by the selected number of principal components extracted from the interval data. The resulting factor structure was expected to include intercorrelated factors as all the items on the scale are meant to measure some aspect of COVID-19-related stressors. It was expected that individual items and, hence, their underlying factors, are likely to correlate with one another; therefore, the oblique Oblimin with Kaiser Normalization rotation method was used.

Two factors have eigenvalues greater than 1. One factor had a rotated eigenvalue of over 21, while the second factor was at roughly 2.0. Given that the rule of eigenvalue greater than one typically results in the identification of too many factors, the scree plot was interpreted to aid in factor retention decision making. Using the scree test, one factor was retained. It is also worth noting that the single factor solution was more interpretable. The second factor identified with an eigenvalue greater than one included one main item (Item 2, related to self-monitoring of symptoms), and an assortment of poor loading items which primarily loaded negatively. Its extraction may be an artifact of the measurement properties of the scale. Taken together, the evidence led to the retention of one factor. The component loadings are presented in Table 2. Factor 1 accounted for 21.76% of the variance explained.

Table 2.

Factor Loadings for the COVID-19 Stressors Scale.

Item	Loading	Item–total correlation	α if item removed
8: Uncertainty about time lines . . .	.86	.56	.94
1: Concern about infection . . .	.82	.62	.94
14: Cancellation of celebration . . .	.81	.68	.94
11: Change to education routine . . .	.80	.55	.94
16: Inability to travel . . .	.80	.61	.94
5: Read or heard about others . . .	.79	.46	.94
12: Changes to social routines . . .	.79	.55	.94
21: Changes to economy . . .	.78	.49	.94
18: Pressure to find silver lining . . .	.77	.60	.94
22: Difficulty accessing resources . . .	.75	.62	.94
13: Changes to dependent care . . .	.74	.70	.94
3: Risk of infection to loved ones . . .	.73	.59	.94
4: Risk of unintentionally infecting others . . .	.72	.49	.94
7: Age-group stigma, shame . . .	.72	.69	.94
9: Change to personal care routines . . .	.72	.74	.94
6: Workplace stigma, shame . . .	.71	.70	.94
17: Increased contact with others . . .	.70	.74	.94
20: Loss of job training or education . . .	.69	.63	.94
10: Changes to work routines . . .	.68	.67	.94
23: Inadequate access to information . . .	.68	.62	.94
15: Cancellation of religious rituals . . .	.67	.58	.94
19: Loss of job security or income . . .	.63	.59	.94
2: Self-monitoring of symptoms . . .	.61	.68	.94

Note. COVID-19 = coronavirus disease 2019.

Reliability and Validity

To further examine the properties of the COVID-19 Stressors Scale and to offer supporting information, additional testing was undertaken. First, to determine if the items on the scale were consistent with one another, a coefficient alpha was estimated. The coefficient alpha was good, indicating the measure is highly internally consistent, α = .96. We show the item–total correlations in Table 2. Alpha ratings for scale-if-item-deleted were similarly high, ranging from .933 to .936. The majority of scores were clustered around .934, suggesting high levels of internal consistency.

Next, scores on the COVID-19 Stressors Scale were correlated with other measures of interest to determine convergent and discriminant validity. It was anticipated that scores on a measure of disaster-related stress would be highly correlated with other measures of stress and anxiety, including the PSS and GAD. Correlations were in the expected directions. Table 3 presents a correlation table. Scores on the COVID-19 Stressors Scale were significantly positively correlated with scores on the GAD (r = .54, p < .001) and PSS (r = .46, p < .001). These correlations were all in the expected directions and strengths.

Table 3.

Bivariate Correlations With the COVID-19 Stressors Scale.

Scales	1	2	3	4
1. COVID-Stressor Scale Composite Score	—
2. Generalized Anxiety (GAD)	.54**	—
3. Depression (MDI)	.51**	.88**	—
4. Perceived Stress Scale	.46**	.58**	.72**	—

Note. COVID-19 = coronavirus disease-19; GAD = Generalized Anxiety Disorder Scale; MDI = Major Depression Inventory.

p < .05. ^**p < .001.

Discussion

We sought to examine the psychometric properties of the COVID-19 Stressors Scale. Establishing the dimensionality and convergent/discriminant validity of the scale is the first step in establishing the scale’s utility for research purposes. Preliminary results indicate that the COVID-19 Stressors Scale is possibly a unidimensional scale and that the scale is internally consistent. Furthermore, evidence from a test of convergent validity indicates that the measure is associated with measures of constructs conceptually similar to stress appraisal, including anxiety. Other work (Hynes et al., 2020) provides information regarding profiles of unique patterns of stressors are interpretable, and pending work by these authors confirms the factor structure of the instrument through confirmatory factor analysis.

The implications of the present work relate to the broader literature regarding stress exposure and appraisal assessment. Evidence from the present scale evaluation suggests that it is possible to develop a psychometrically valid scale (under classical test theory) that sits at the intersection of event and global approaches to stress exposure and appraisal measurement, where multiple stressor domains (resource-related, health-related, or routine-related) can be measured in an internally consistent fashion during a community-wide, sustained crisis such as the COIVD-19 pandemic. The unidimensional nature of the COVID-19 Stressors Scale suggests that the pandemic itself can be understood as an event with pervasive impacts across multiple stressor domains. Furthermore, the high internal consistency suggests that stress appraisals across different stress exposures are strongly interrelated.

This approach to stressor measurement advances our capability to quantify the range of exposures and the relative appraised stressfulness across these different exposures. Furthermore, the COVID-19 Stressors Scale assesses, in a single metric, the cumulative stressfulness across many different types of exposures. Such an approach is rarely taken in mass crisis (or any other) context; instead, a global appraisal score is typically used (e.g., Cohen et al., 1983) or, more rarely, a measure of cumulative exposures (e.g., Garfin et al., 2014). Both may be important indices, but perhaps more powerful will be the cumulative stressfulness of these multiple exposures. Future work is needed to determine the relative utility of these different ways of quantifying stress following community-wide crises with long-term consequences.

While results from the present study are mathematically sound, there are limitations of the sample itself. The sample was collected using Amazon’s MTurk, an online task platform. It is possible that MTurk recruitment may have limited the sample’s generalizability as all respondents were MTurk workers, with some unifying characteristics (access to a computer with internet access and an Amazon account, specifically). Furthermore, the sample was not representative of the whole of the United States. Future studies examining the psychometric properties of the COVID-19 Stressors Scale should consider larger, and more diverse, samples. Future research is required to better understand the psychometric properties of the COVID-19 Stressors Scale and its utility in a research setting. A confirmatory factor analysis testing the factor structure in a new sample will provide important information about the replicable structure of the measure and inform future measure refinement, and is currently being conducted by the present research team. As well, research to determine the predictive validity of the measure would provide data about its utility in research contexts.

There are also several limitations of the scale itself that must be noted. Future work with this measure should expand efforts to capture the experiences of vulnerable groups in particular (e.g., BIPOC [Black, Indigenous, People of Color] and other underrepresented groups) and attend to the potential for any internalized stigma they may experience anchored in the context of a given crisis. For example, given the likely origins of COVID-19 in late 2019 and the subsequent stigma created by publicized acts of racism aimed at the Asian population over the course of 2020, further work should be conducted specifically with Asian and Asian American samples regarding stigma related to COVID-19. Furthermore, other minorities, especially those who are African Americans, Latinx, and Native American, were more severely affected by the disease, suffering higher mortality rates (Johns Hopkins University Center for Systems Science and Engineering, 2020); they suffered potential discrimination stemming from their risk status and consequently potential stress resulting from their increased vulnerability to the disease. Items that relate to instances of racism, microaggressions, or other negative beliefs regarding one’s race (or, alternatively, concurrent but separate measurement of racial and/or ethnic stigma) and to enhanced health risk vulnerability might have been useful in this instance. Another concern relates to the reading level of the scale. A check of the reading level of the scale using Microsoft Word standard software indicated that the scale was developed at a reading level of higher than 10th grade, and a reading ease of only 22.3. This suggests that some of the words and phrases used in the items may be difficult for the average adult reader to understand. Future revisions of the scale should address difficulty in reading level. In its present version, we advise that the scale only be used with individuals who have graduated from high school/earned a GED, as the reading level would be most appropriate for that population. In addition, the original study from which these data were drawn was not intended to be a measurement study. Thus, we did not collect optimal measures for establishing convergent or discriminant validity. Future studies examining the psychometric properties of this measure or similar instruments would be advised to collect measures more suitable for validity verification. Finally, all individuals likely experience stressors in different ways and that stress is likely to manifest in different ways. The items of this scale may not capture all the ways in which stress manifests, so the addition of other items may be warranted as the scale continues to be developed.

Despite these limitations and the need for future efforts to confirm the measurement properties of the COVID-19 Stressors Scale, the present study provides information about the scale. These results provide a preliminary indication that the COVID-19 Stressors Scale may be unidimensional, and is an internally consistent measure of stressors exposure and stressfulness appraisal. With a growing body of research seeking to evaluating the variation in impacts from COVID-19 on mental health and related adverse consequences across diverse regions and populations (Russell et al., 2020), it is important that researchers employ common metrics so that valid comparisons can be made and effective interventions can be developed. The scale discussed in this report has demonstrated reliability and possible preliminary potential utility for exploring mental health issues in the ongoing COVID-19 pandemic. Practitioners and scholars working to respond to COVID-19 with agility to best assess then respond to public distress may strengthen their efforts by including psychometrically tested scales like this one in their research and service provision designs.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Rachel R. Tambling

References

American Psychological Association. (2020). Disasters. https://www.apa.org/topics/disasters

Baral

I. A.

Bhagawati

K. C.

(2019). Post traumatic stress disorder and coping strategies among adult survivors of earthquake, Nepal. BMC Psychiatry, 19(1), Article 118. https://doi.org/10.1186/s12888-019-2090-y

Bartneck

Duenser

Moltchanova

Zawieska

Voracek

(2015). Comparing the similarity of responses received from studies in Amazon’s Mechanical Turk to studies conducted online and with direct recruitment. PLOS One, 10(4), e0121595. https://doi.org/10.1371/journal.pone.0121595

Bell

C. J.

Boden

J. M.

Horwood

L. J.

Mulder

R. T.

(2017). The role of peri-traumatic stress and disruption distress in predicting symptoms of major depression following exposure to a natural disaster. Australian & New Zealand Journal of Psychiatry, 51(7), 711–718. https://doi.org/10.1177/0004867417691852

Bolt

M. A.

Helming

L. M.

Tintle

N. L.

(2018). The associations between self-reported exposure to the Chernobyl nuclear disaster zone and mental health disorders in Ukraine. Frontiers in Psychiatry, 9, 32. https://doi.org/10.3389/fpsyt.2018.00032

Brooks

S. K.

Webster

R. K.

Smith

L. E.

Woodland

Wessely

Greenberg

Rubin

G. J.

(2020). The psychological impact of quarantine and how to reduce it: Rapid review of the evidence. Lancet, 395(10227), 912–920. https://doi.org/10.1016/S0140-6736(20)30460-8

Brunet

Weiss

D. S.

Metzler

T. J.

Best

S. R.

Neylan

T. C.

Rogers

Fagan

Marmar

C. R.

(2001). The Peritraumatic Distress Inventory: A proposed measure of PTSD criterion A2. American Journal of Psychiatry, 158(9), 1480–1485. https://doi.org/10.1176/appi.ajp.158.9.1480

Bryant

F. B.

Yarnold

P. R.

(1995). Principal-components analysis and exploratory and confirmatory factor analysis. In Grimm

L. G.

Yarnold

P. R.

(Eds.), Reading and understanding multivariate statistics (pp. 99–136). American Psychological Association.

Centers for Disease Control and Prevention. (2020). COVID-19 forecasts. https://www.cdc.gov/coronavirus/2019-ncov/covid-data/forecasting-us.html

10.

Cohen

Janicki-Deverts

(2012). Who’s stressed? Distributions of psychological stress in the United States in probability samples from 1983, 2006 and 2009. Journal of Applied Social Psychology, 42(6), 1320–1334. https://doi.org/10.1111/j.1559-1816.2012.00900.x

11.

Cohen

Kamarck

Mermelstein

(1983). A global measure of perceived stress. Journal of Health and Social Behavior, 24(4), 385–396. https://doi.org/10.2307/2136404

12.

Epel

E. S.

Corsswell

A. D.

Mayer

S. E.

Prather

A. A.

Slavich

G. M.

Puterman

Mendes

W. B.

(2018). More than a feeling: A unified view of stress measurement for population science. Frontiers of Neuroendocrinology, 49, 147–169. https://doi.org/10.1016/j.yfrne.2018.03.001

13.

Fussell

Lowe

S. R.

(2014). The impact of housing displacement on the mental health of low-income parents after Hurricane Katrina. Social Science & Medicine, 113, 137–144. https://doi.org/10.1016/j.socscimed.2014.05.025

14.

Galea

Brewin

C. R.

Gruber

Jones

R. T.

King

D. W.

King

L. A.

McNally

R. C.

Ursano

R. J.

Petukhova

Kessler

R. C.

(2007). Exposure to hurricane-related stressors and mental illness after Hurricane Katrina. Archives of General Psychiatry, 64(12), 1427–1434. https://doi.org/10.1001/archpsyc.64.12.1427

15.

Galea

Coffey

S. F.

(2018). Social support moderates effects of natural disaster exposure on depression and posttraumatic stress disorder symptoms: Effects for displaced and nondisplaced residents. Journal of Traumatic Stress, 31(2), 223–233. https://doi.org/10.1002/jts.22270

16.

Galea

Merchant

R. M.

Lurie

(2020). The mental health consequences of COVID-19 and physical distancing the need for prevention and early intervention. JAMA Internal Medicine, 180(6), 817–818. https://doi.org/10.1001/jamainternmed.2020.1562

17.

Garfin

D. R.

Silver

R. C.

Holman

E. A

. (2020). The novel coronavirus (COVID-2019) outbreak: Amplification of public health consequences by media exposure. Health Psychology, 39(5), 355–357. https://dx-doi-org.web.bisu.edu.cn/10.1037/hea0000875

18.

Garfin

D. R.

Silver

R. C.

Ugalde

F. J.

Linn

Inostroza

. (2014). Exposure to rapid succession disasters: A study of residents at the epicenter of the Chilean Bío Bío earthquake. Journal of Abnormal Psychology, 123(3), 545–556. https://doi.org/10.1037/a0037374

19.

Gorsuch

R. L.

(1983). Factor analysis (2nd ed.). Lawrence Erlbaum.

20.

Gray

Litz

Hsu

Lombardo

(2004). Psychometric properties of the Life Events Checklist. Assessment, 11(4), 330–341. https://doi.org/10.1177/1073191104269954

21.

Horesh

Brown

A. D.

(2020). Traumatic stress in the age of COVID-19: A call to close critical gaps and adapt to new realities. Psychological Trauma, 12(4), 331–335. http://doi.org/10.1037/tra0000592

22.

Hutcheson

Sofroniou

(1999). The multivariate social scientist: Introductory statistics using generalized linear models. Sage.

23.

Hynes

Tambling

R. R.

Russell

Park

Feindrich

. (2020). Measuring COVID-19-related stress using the COVID-19 Stressors Scale: A latent profile analysis [Manuscript submitted for publication]. Department of Behavioral Sciences, Purdue University Northwest.

24.

Johns Hopkins University Center for Systems Science and Engineering. (2020). COVID-19 dashboard. https://www.arcgis.com/apps/opsdashboard/index.html#/bda7594740fd40299423467b48e9ecf6

25.

Jose

Holman

E. A.

Silver

R. C.

(2019). Community organizations and mental health after the 2013 Boston Marathon bombings. Social Science & Medicine, 222, 367–376. https://doi.org/10.1016/j.socscimed.2018.08.019

26.

Kees

Berry

Burton

Sheehan

(2017). An analysis of data quality: Professional panels, student subject pools, and Amazon’s Mechanical Turk. Journal of Advertising, 46(1), 141–155. https://doi.org/10.1080/00913367.2016.1269304

27.

Labarda

C. E.

Jopson

Q. D. Q.

Hui

V. K.

Chan

C. S.

(2020). Long-term displacement associated with health and stress among survivors of typhoon Haiyan. Psychological Trauma: Theory, Research, Practice, and Policy, 12(7), 765–773. https://doi.org/10.1037/tra0000573

28.

Layne

S. P.

Hyman

J. M.

Morens

D. M.

Taubenberger

J. K.

(2020). New coronavirus outbreak: Framing questions for pandemic prevention. Science Translational Medicine, 12(534), Article eabb1469. https://doi.org/10.1126/scitranslmed.abb1469

29.

Lazarus

R. S.

Folkman

(1984). Stress, appraisal & coping. Springer.

30.

MacCallum

Widaman

Zhang

Hong

(1999). Sample size in factor analysis. Psychological Methods, 4(1), 84–99. https://doi.org/10.1037/1082-989X.4.1.84

31.

Main

Zhou

Luecken

L. J.

Liu

(2011). Relations of SARS-related stressors and coping to Chinese college students’ psychological adjustment during the 2003 Beijing SARS epidemic. Journal of Counseling Psychology, 58(3), 410–423. https://doi.org/10.1037/a0023632

32.

Mortensen

Hughes

T. L.

(2018). Comparing Amazon’s Mechanical Turk platform to conventional data collection methods in the health and medical research literature. Journal of General Internal Medicine, 33(4), 533–538. https://doi.org/10.1007/s11606017-4246-0

33.

North

C. S.

Pfefferbaum

(2013). Mental health response to community disasters: A systematic review. JAMA Journal of the American Medical Association, 310(5), 507–518. https://doi.org/10.1001/jama.2013.107799

34.

Park

C. L.

Finkelstein-Fox

Russell

B. S.

Fendrich

Hutchison

Becker

. (in press). Americans’ psychological resilience early in the covid-19 pandemic: Protective resources and coping strategies. Psychological Trauma: Theory, Research, Practice, and Policy.

35.

Park

C. L.

Riley

Snyder

(2012). Meaning making coping, making sense, and posttraumatic growth following the 9/11 terrorist attacks. Journal of Positive Psychology, 7(3), 198–207. https://doi.org/10.1080/17439760.2012.671347

36.

Park

C. L.

Russell

B. S.

Fendrich

Finkelstein-Fox

Hutchison

Becker

(2020). Americans’ COVID-19 stress, coping, and adherence to CDC guidelines. Journal of General Internal Medicine, 35(8), 2296–2303. https://doi.org/10.1007/s11606-020-05898-9

37.

Qiu

Shen

Zhao

Wang

Xie

(2020). A nationwide survey of psychological distress among Chinese people in the COVID-19 epidemic: Implications and policy recommendations. General Psychiatry, 33(2), e100213. https://doi.org/10.1136/gpsych-2020100213

38.

Russell

B. S.

Hutchison

Tambling

Tomkunas

A. J.

Horton

A. L

. (2020). Initial challenges of caregiving during COVID-19: Caregiver burden, mental health, and the parent–child relationship. Child Psychiatry & Human Development, 51, 671–682. https://doi.org/10.1007/s10578-020-01037-x

39.

Seto

Imamura

Suppasri

(2019). Challenge to build the science of human survival from disaster starting from analysis for the 2011 Tohoku Tsunami. Journal of Disaster Research, 14(9), 1323–1328. https://doi.org/10.20965/jdr.2019.p1323

40.

Sheehan

K. B.

(2018). Crowdsourcing research: Data collection with Amazon’s Mechanical Turk. Communication Monographs, 85(1), 140–156. https://doi.org/10.1080/03637751.2017.1342043

41.

Sheehan

K. B.

Pittman

(2016). Amazon’s Mechanical Turk for academics: The HIT handbook for social science research. Melvin & Leigh.

42.

Solomon

H. V.

(2020). COVID-19 checklist: Mask, gloves, and video chatting with grandpa. Psychiatry Research, 288, Article e112986. https://doi.org/10.1016/j.psychres.2020.112986

43.

Spitzer

R. L.

Kroenke

Williams

J. B. W.

Löwe

(2006). A brief measure for assessing generalized anxiety disorder the GAD-7. Archives of Internal Medicine, 166(10), 1092–1097. https://doi.org/10.1001/archinte.166.10.1092

44.

Sprague

C. M.

Kia-Keating

Felix

Afifi

Reyes

Afifi

(2015). Youth psychosocial adjustment following wildfire: The role of family resilience, emotional support, and concrete support. Child Youth Care Forum, 44(3), 433–450. https://doi.org/10.1007/s10566-014-9285-7

45.

Stokes

E. K.

Zambrano

L. D.

Anderson

K. N.

Marder

E. P.

Raz

K. M.

Felix

S. E. B.

Tie

Fullerton

K. E.

(2020). Coronavirus disease 2019 case surveillance—United States, January 22–May 30, 2020. MMWR Morbidity and Mortality Weekly Report, 69(36), 759–765. http://doi.org/10.15585/mmwr.mm6924e2externalicon

46.

Tinsley

H. E. A.

Tinsley

D. J.

(1987). Uses of factor analysis in counseling psychology research. Journal of Counseling Psychology, 34(4), 414–424. https://doi.org/10.1037/0022-0167.34.4.414

47.

U.S. Bureau of Labor. (2020). Unemployment rate. https://data.bls.gov/timeseries/LNS14000000

48.

Velicer

W. F.

Fava

J. L.

(1998). Affects of variable and subject sampling on factor pattern recovery. Psychological Methods, 3(2), 231–251. https://doi.org/10.1037/1082-989X.3.2.231

49.

Wang

Pan

Wan

Tan

C. S.

R. C.

(2020). Immediate psychological responses and associated factors during the initial stage of the 2019 coronavirus disease (COVID-19) epidemic among the general population in China. International Journal of Environmental Research and Public Health, 17(5), 1–25. http://doi.org/10.3390/ijerph17051729

50.

Weiss

D. S.

Marmar

C. R.

(1996). The Impact of Event Scale–Revised. In Wilson

Keane

T. M.

(Eds.), Assessing psychological trauma and PTSD (pp. 399-411). Guilford.

51.

World Health Organization. (2020). Coronavirus disease pandemic. https://www.who.int/emergencies/diseases/novel-coronavirus-2019

52.

Zhang

Z. F.

(2020). Impact of the COVID-19 pandemic on mental health and quality of life among local residents in Liaoning Province, China: A cross-sectional study. International Journal of Environmental Research and Public Health, 17(7), 2381. http://doi.org/10.3390/ijerph17072381