Mental Health Screening Before and After the 2020 COVID-19 Closures of US College Campuses: A Cross-Sectional Comparison Using Natural Language Processing

Study Design

We used data from students participating in the ISP for this cross-sectional study. The Johns Hopkins Medicine Institutional Review Board acknowledged this secondary analysis of deidentified data (under application 00326442) as exempt from human studies research review and deemed that consent from participants was not required. We conducted the analysis in accordance with the ethical standards outlined in the 1964 Declaration of Helsinki.

Data Sources and Sample

We obtained the ISP data from the AFSP. We decided to select the cutoff for the pre– and post–COVID-19 closure periods as Sunday, March 15, 2020, the date by which more than one-half of US postsecondary institutions had transitioned to virtual learning.^20,21 The transition happened quickly. As of March 8, only 2 institutions had closed. By March 18, >85% of institutions had transitioned to virtual learning. ²⁰ To control for differences in seasonal variation, we established the time frame for the analysis as 1 full year before and after March 15, 2020. We included students at institutions who were using the ISP continuously between March 16, 2019, and March 15, 2021. We defined continuous use as at least 1 student at their institution completing the questionnaire in periods that correspond to months when a school would be in session (eSection 1 in the Supplement).

Outcome Measures

We examined the following utilization outcomes: (1) the number of participants per month per institution, (2) the proportion of participants who participated in a dialogue, and (3) the proportion of participants who sought a referral to care. We counted dialogue participation as yes (vs no) when the student had sent a message to the mental health counselor after completing the survey, and we calculated the proportion by using the denominator of ISP participants during the same pre– or post–COVID-19 closure period.

We used NLP methods to analyze the content of the dialogue entries and to determine whether a student sought a referral to care. We calculated proportions by using the denominator of all ISP participants during the same pre– or post–COVID-19 closure period. Before analysis, one author (C.A.) deidentified the text data by using established NLP methods for deidentifying personal health information. ²² To mitigate the possibility of bias and strengthen reliability, we implemented a rigorous 4-step process to determine the outcome of seeking a referral to care: (1) we established a standardized definition for seeking a referral to care, (2) we manually coded a test set of dialogues using the standardized definition, (3) we trained and validated the NLP models on the manually coded test set, and (4) we applied the NLP models to the rest of the dataset.

For step 1, we drafted a definition for seeking a referral to care and applied the definition to a random sample of 10 dialogues by having 4 to 6 authors code each dialogue as yes or no for seeking a referral to care. We discussed disagreements in coding and revised the definition of seeking a referral to care through a consensus-building process. We executed this process on sets of 10 randomly sampled dialogues and assessed interrater agreement, as measured by Randolph κ. Randolph κ is an alternative to Cohen κ when there are >2 coders. ²³ We obtained a Randolph κ of 0.93 for seeking a referral to care after we coded 70 unique dialogues.

For step 2, we randomly sampled a test set of 200 dialogues and assigned random combinations of 3 authors from a 6-author coding team to code each dialogue. The majority vote among the 3 coders determined whether a dialogue was coded yes or no for seeking a referral to care. We determined that the interrater agreement for seeking a referral to care was still satisfactory in this 200-dialogue test set (0.78).

For step 3, an author with expertise in NLP (C.A.) tested various NLP model settings and prompts for GPT-4o (OpenAI) on 50 of the 200 dialogues to identify the best-performing model. The author then applied the highest-performing settings and prompt to the full 200-dialogue test and checked against the manual coding. We determined that GPT-4o achieved an accuracy score of 90.2% and a Cohen κ of 0.80 on this 200-dialogue test set.

After we achieved these robust performance scores, for step 4, we allowed GPT-4o to assign yes or no for seeking a referral to care to the rest of the dialogues. Likely for safety reasons, GPT-4o refused to evaluate 281 dialogues (many of which contained language about self-harm). Two authors (S.L. and A.M.W.) coded those dialogues (achieving 89% agreement). A third coder (J.A.) coded the remaining 31 dialogues for which the 2 authors had disagreed. We considered students who did not enter any dialogue text as not having sought a referral. Additional details about the NLP methods are included in eSection 3 of the Supplement, such as the codebook definition for seeking a referral, the model settings tested, and the final model settings and prompt.

Predictor

The predictor was a binary variable representing the pre– and post–COVID-19 closure periods for the 1 year before and after the end of the day on Sunday, March 15, 2020, with the former serving as the reference group.

Covariates

We collected sociodemographic measures from self-reported student data. Each institution selects which sociodemographic variables to collect, leading to variability in the availability of certain variables. Adopting a threshold used in prior work with ISP data, we included only sociodemographic variables that had no more than 30% missing data. ²⁴ Retained sociodemographic variables included age (5.0% missing data), gender (2.6% missing data), and race and ethnicity (19.2% missing data). The survey obtained age as a numerical entry. An open-ended question assessed gender with categorical responses in a “select all that apply” format and coded as female, male, transgender, genderqueer (which included people who identified as gender nonconforming), and other (which included people who selected self-describe, nonbinary, gender agnostic, agender, third gender, other, or ≥1 response). Similarly, an open-ended question queried race and ethnicity with categorical responses that included White (some institutions used the descriptors Caucasian/European/White), American Indian or Alaska Native, African American/Black (some institutions used African/Afro-Caribbean), Hispanic/Latin(x), multiethnic, Asian/Pacific Islander, Arab/Middle Eastern, other, and prefer not to answer.

Analysis

We used RStudio version 2023.12.1+402 (Posit PBC) for our analyses. We calculated descriptive statistics for sociodemographic characteristics of participants in the year before and after March 15, 2020, as well as the total number of users, mean monthly users, and proportion of users participating in the dialogue and seeking a referral. We calculated the mean monthly participation per institution as the total number of participants completing a questionnaire per institution (determined by the survey time stamp) in each of the pre– and post–COVID-19 closure periods divided by 12 (to get a monthly average). Of note, to calculate mean monthly participation, we combined participation in the 2 half-months on either end of the pre–COVID-19 year (March 16-31, 2019, and March 1-15, 2020) to create 12 total months, and we did the same for the following year. We used 2-sample t tests to test for differences in the mean monthly users before and after March 15, 2020, and Pearson χ² tests to determine differences in proportions.

We developed 4 pairs of regression models: an unadjusted model and an adjusted model for each outcome. For each model, we used the closure period (pre– vs post–COVID-19) as the predictor and 95% CI for its coefficient to test the hypothesis that utilization was different before and after March 15, 2020. For the first pair, we used Poisson regression to model the outcome of participation count per institution in each calendar month. The adjusted model included institution, calendar month, the proportion of female respondents at an institution in that month, and the proportion of White respondents at an institution in that month. We chose female and White respondents because they were the largest categories of gender and race and ethnicity. For the second and third pairs, we used logistic regression to model the outcomes of engaging in a dialogue and seeking a referral, respectively. Both adjusted models included each participant’s institution, gender, and race and ethnicity, as well as the month of participation. For the fourth model pair, we used Poisson regression to execute an interrupted time-series analysis of the participation count per institution in each month. ²⁵ The interrupted time-series equations are shown here.

( 1 ) log ( e x p e c t e d p a r t i c i p a n t s ) = β 0 + β 1 * m o n t h n u m b e r + β 2 * p e r i o d + β 3 * m o n t h n u m b e r * p e r i o d + β 4 * m o n t h n a m e + log ( e x p o s u r e )

( 2 ) log ( e x p e c t e d p a r t i c i p a n t s ) = β 0 + β 1 * m o n t h n u m b e r + β 2 * p e r i o d + β 3 * m o n t h n u m b e r * p e r i o d + log ( e x p o s u r e )

The monthnumber (0-24 in the 2-year time series) and the interaction term between monthnumber and period (pre- vs postclosure) distinguished the time-series models from the first pair of Poisson models described previously. Of the 2 interrupted time-series regressions, the first included a term (monthname) to adjust for monthly variation in participation. All Poisson regressions used the number of days in each calendar month as the exposure, and 95% CIs were generated by robust SEs given overdispersion in the data.

We conducted 8 post hoc sensitivity analyses (eSections 6-9 in the Supplement). The first 3 analyses assessed whether the findings were driven by particular groups of institutions by excluding 3 groups of schools from the analysis (low-utilization schools, schools with inconsistent participation, and schools in the top and bottom deciles of utilization). The next 3 analyses examined how handling the safety-flagged dialogues in 3 ways affected our results. For the seventh, given the rollout of vaccines in December 2020, ²⁶ we investigated how restricting the comparison periods to March 16 to December 31 of 2019 and 2020 affected the results. Eighth and last, we checked for trends in ISP utilization across gender and race subgroups.

Limitations and Strengths

This study was limited by several factors. First, differences could not be assessed in the implementation of the program before and after the campus closures (eg, the number of students invited and how they were contacted), which might have influenced utilization metrics. The lack of difference in overall utilization before and after March 15, 2020, might not truly indicate equivalent rates of participation. For example, if institutions considerably ramped up email invitations to participate in the ISP after March 15, 2020, but utilization did not change, this would mean that the proportion of students participating actually decreased. Second, our “seeks referral” variable was based on student-reported interest in connecting to mental health care services, but we lacked data to determine whether students actually connected to such services. Third, limited and variable assessment of students’ socioeconomic characteristics gave us little ability to disentangle factors, such as financial means, that might have influenced differences in utilization among racial groups. Fourth, it is not clear how many students were invited to participate in the ISP, whether invitation procedures differed among institutions, and whether recruitment after the campus closures would have differentially affected students who stayed near campus versus those who returned home.

Strengths of this study included the use of a large dataset from multiple institutions, as well as NLP methods with robust performance metrics to analyze a large amount of text from several thousand participants and infer referral-seeking behaviors.