Abstract
Young people may search for, or be exposed to, self-harm- and suicide-related content on social media, and they sometimes use these platforms to communicate about their own experiences of self-harm and suicide. Preliminary evidence demonstrates exposure to self-harm and suicide-related content online has both protective and harmful effects. However, research in this area has been hindered by the lack of validated measures of young people’s behavior, confidence, and safety when communicating online about self-harm and suicide. This study seeks to address this by reporting on the development and validation of the #chatsafe Online Safety Scale in a sample of 535 adolescents and young adults aged 16 to 25 years (M = 21.81, SD = 2.41). Exploratory factor analysis and confirmatory factor analysis were performed to explore the factor structure of the scale and validate the measurement models. Multigroup analyses were performed to assess gender invariance, and Cronbach’s alpha was calculated to assess the internal consistency of each subscale. Results indicate that the scale has good factor structure, performs well across genders, and demonstrates good internal consistency. This study provides preliminary validation of the #chatsafe Online Safety Scale and suggests it can be used in studies exploring young people’s online experiences and behaviors related to self-harm and suicide, providing a stronger and comparable evidence base to guide policy and practice.
Background
Young people may encounter or seek self-harm and suicide-related content on social media, and they sometimes use these platforms to communicate about their own experiences of self-harm and suicide.1–4 Research shows such content can have both protective and harmful effects. 5 Communicating about self-harm and suicide online can allow young people to seek and provide support in an environment where they feel comfortable; 3 however, exposure to certain content may trigger self-harm urges and behaviors.4–6
Given the mixed impacts, further research is needed to understand young people’s experiences with self-harm and suicide-related content and communication online. Progress has been limited by the lack of validated measures of young people’s use, confidence, and safety when communicating online about self-harm and suicide. A reliable and valid tool will facilitate the design of rigorous studies, enable behaviors to be uniformly measured, and strengthen the evidence base. This evidence will inform the development of policies and strategies to create safer online environments for young people.
This study reports on the development and validation of a measure of young people’s behavior, confidence, and safety when communicating online about self-harm and suicide.
Methods
Item generation
Initial items were developed to assess young people’s behavior, confidence, and safety when communicating online about self-harm and suicide and were designed to align with the #chatsafe guidelines.7,8 These guidelines were developed using a six-stage Delphi consensus method: 7 (1) systematic search of peer-reviewed and grey literature, (2) roundtables with key stakeholders (social media companies, policymakers, young people), (3) extraction and review of statements from peer-reviewed articles (n = 149), grey literature (n = 52), and roundtables (n = 6), (4) formation of expert panels (suicide prevention professionals, young people), (5) data collection and analysis, and (6) guideline development. The item structure (Likert scale items and vignettes) was adapted from the National Survey for Community Health. 9 Items were reviewed for suitability and readability by one senior academic and six young people.
Items were arranged into a single scale comprising two sections. The first assesses online behavior across four domains: general usage, seeing self-harm and suicide content online (exposure to content), responding to self-harm and suicide content online, and posting self-harm and suicide content online. The second section includes three vignettes measuring confidence and safety when sharing (vignette one), responding (vignette two), and communicating (vignette three) online about self-harm and suicide.
Participant recruitment
Participants were recruited through social media and Prolific, an online recruitment platform. 10 Eligible participants (1) were aged between 16 and 25 years and (2) had used social media to communicate about self-harm or suicide and/or seen self-harm- or suicide-related information online. In total, 963 participants responded to the advertisement, 641 provided consent and clicked the REDCap survey link, and 535 completed the #chatsafe Online Safety Scale and were included in the analysis.
Data analysis
Data were analyzed using R (version 4.3.3) and followed a four-stage approach. First, items were reviewed in consultation with online safety content experts (two academics with 6 years’ combined experience and exceptional knowledge of the #chatsafe guidelines) and through inspection of network graphs. Second, exploratory factor analysis (EFA) was used to refine the scale, with oblimin rotation applied to account for likely correlations between factors. Confirmatory factor analysis (CFA) was then conducted to validate the measurement models (subscales and vignettes). Finally, multigroup CFA was performed to test measurement invariance across genders, given evidence suggests females and males may use social media differently. 11 Three levels of invariance were tested: configural (factor loadings and intercepts are estimated freely), metric (factor loadings are held constant), and scalar (factor loadings and intercepts are held constant). Cutoff values for comparative fit index (ΔCFI; ≤0.01), root mean squared error of approximation (ΔRMSEA; ≤0.015), and standardized root mean squared residual (ΔSRMR; ≤0.03 at the metric level and 0.015 at the scalar level) were used. The internal consistency of each subscale was calculated.
Findings
The final sample consisted of 535 Australian young people aged between 16 and 25 years (M = 21.81, SD = 2.40). Participant demographics are displayed in Table 1.
Participant Demographics
Social media use items
Stage 1. Initial review
Visual inspection and expert consultation resulted in no items being removed.
Stage 2. Exploratory factor analysis
The KMO statistic (0.87) indicated suitability for factor analysis. Parallel analysis suggested up to seven factors, Velicer’s MAP suggested five, and content exploration supported four. EFA was conducted for four-, five-, and six-factor solutions. The four-factor solution resulted in no cross-loadings and was theoretically plausible. Three were removed due to low loadings (see Table 2).
Item Generation and Refinement
Stage 3. Confirmatory factor analysis
CFA on the four-factor model indicated poor fit (RMSEA = 0.12, CFI = 0.78). Modification indices were calculated, and network graphs were created to identify sources of misfit. This resulted in the removal of three items. Fit statistics indicated an acceptable model fit (RMSEA = 0.10, CFI = 0.85). Table 3. displays the factor loadings and the internal consistency for each subscale. Figure 1. displays the network model of all items.
Factor Loadings for General Use Items (Section 1) and Vignette Items (Section 2)
Supplementary Appendix A2 includes a list of complete items and item abbreviations.
Behavior items. Note. G1 = Search mental health, G2 = Share mental health, G3 = Search suicide, G4 = Share suicide, G5 = Search self-harm, G6 = Share self-harm, S1 = Saw suicide information, S2 = Saw self-harm information, S3 = Saw suicidal thoughts behaviors, S4 = Saw self-harm thoughts behaviors, R1 = Responded suicide information, R2 = Responded self-harm information, R3 = Responded celebrity suicide, R4 = Responded suicidal thoughts behaviors, C1 = Created suicide information, C2 = Created self-harm information, C3 = Created celebrity suicide, C4 = Created personal suicidal thoughts behaviors, C5 = Created other suicidal thoughts behaviors.
Stage 4. Gender invariance testing
Fit statistics for the configural, metric, and scalar models provide support for gender invariance (see Table 4).
Fit Statistics for the Configural, Metric and Scalar Models
For vignette 1, partial scalar invariance was supported by freely estimating the following items: “To advise Jamie to use sensitive language so as not to stigmatise suicide”, “To include helpline information in his post”, “To include a content or trigger warning before his post”.
For vignette 2, partial scalar invariance was supported by freely estimating the following items: “Ignore the post because it is none of my business”, “Ask Jay directly if she is thinking of ending her life”, “Ask Jay directly if she is thinking of ending her life”.
Vignette 1. Safety sharing
Stage 1. Initial review
Visual inspection and expert consultation resulted in the removal of seven items.
Stage 2. Exploratory factor analysis
The KMO statistic (0.80) indicated suitability for factor analysis. The optimal number of factors was suggested to be between four (parallel analysis) and one (Velicer’s minimum average partial). EFA was conducted for one-, two-, and three- factor solutions. The one-factor solution produced the most meaningful model and was an acceptable fit (RMSEA = 0.08, CFI = 0.91). Two items did not load and were subsequently removed (see Table 2).
Stage 3. Confirmatory factor analysis
CFA was performed on the one-factor model. Fit statistics indicated the model was a good fit for the data (RMSEA = 0.08, CFI = 0.96). Table 3. displays the factor loadings and the internal consistency for this subscale. Supplementary Appendix A3 displays the network model of all items.
Stage 4. Gender invariance testing
Fit statistics for the configural and metric models supported gender invariance. However, scalar invariance was not supported, prompting the exploration of partial scalar invariance. Comparison of configural modes by gender guided the selection of intercepts to be freely estimated. Partial scalar invariance was supported by freely estimating the following items: “To advise Jamie to use sensitive language so as not to stigmatize suicide,” “To include helpline information in his post,” “To include a content or trigger warning before his post” (see Table 4).
Vignette 2. Safety responding
Stage 1. Initial review
Visual inspection and consultation with online safety experts resulted in the removal of two items.
Stage 2. Exploratory factor analysis
The KMO statistic (0.79) indicated suitability for factor analysis. The optimal number of factors was suggested to be between five (parallel analysis) and two (Velicer’s minimum average partial) and content exploration suggested one. EFA was conducted for one-, two-, and three-factor solutions. Fit statistics indicated tall solutions were acceptable; however, several items cross-loaded. Examination of modification indices and factor loadings helped identify potential sources of misfit, leading to the removal of six items. EFA of the revised items was conducted. The two-factor solution demonstrated an appropriate fit (RMSEA = 0.08, CFI = 0.91) and produced two meaningful factors. One item did not load and was subsequently removed (see Table 2).
Stage 3. Confirmatory factor analysis
CFA was performed on the two-factor model. Factor loadings suggested the removal of one item (<0.03). Fit statistics indicated the subsequent model was a good fit for the data (RMSEA = 0.08, CFI = 0.93). Table 3 displays the factor loadings and internal consistency for each subscale. Supplementary Appendix A3 displays the network model of all items.
Stage 4. Gender invariance testing
Fit statistics for the configural and metric models supported gender invariance. However, scalar invariance was not supported, prompting exploration of partial scalar invariance. Partial scalar invariance was supported by freely estimating the following items: “Ignore the post because it is none of my business,” “Ask Jay directly if she is thinking of ending her life” (see Table 4).
Vignette 3. Safety communicating
Stage 1. Initial review
Visual inspection and expert consultation led to removal of two items.
Stage 2. Exploratory factor analysis
The KMO statistic (0.74) indicated suitability for factor analysis. The optimal number of factors was suggested to be between five (parallel analysis) and one (Velicer’s minimum average partial). EFA was conducted for one-, two-, and three- factor solutions. Fit statistics indicated all factor solutions were acceptable: however, several items cross-loaded. Examination of modification indices and factor loadings identified potential sources of misfit, resulting in the removal of six items. EFA of the revised items was conducted. The one factor solution demonstrated an acceptable fit (RMSEA 0.05, CFI = 0.96). Two items did not load and were subsequently removed (see Table 2).
Stage 3. Confirmatory factor analysis
CFA was performed on the one factor model. Fit statistics indicated the model was a good fit for the data (RMSEA = 0.07, CFI = 0.97). Table 3. displays the factor loadings and the internal consistency for the subscale.
Stage 4. Gender invariance testing
Fit statistics for the configural, metric, and scalar models supported gender invariance (see Table 4).
#Chatsafe online safety scale
The #chatsafe Online Safety Scale measures young people’s social media use, confidence and safety when communicating online about suicide and self-harm. The refined scale includes two sections. Section one assesses online behavior across four domains: general usage (n = 6), exposed to content (n = 4), responded to content (n = 4) and created content (n = 5). Items are rated on a 4-point scale from 1 = Often to 4 = Never, with additional items for respondents who have encountered or posted self-harm or suicide-related content online (see Supplementary Appendix A1). Section two includes three vignettes measuring confidence and safety when sharing (vignette one), responding (vignette two), and communicating (vignette three) online about self-harm and suicide. Respondents are provided with a series of items that reflect potential advice that may be given in response to each vignette and asked to indicate their likelihood of providing this advice on 5-point scale from 1 = Very likely to 5 = Very unlikely, with lower scores indicating greater adherence to the #chatsafe guidelines.
Discussion
This study provides preliminary validation of the #chatsafe Online Safety Scale for use with young people aged 16 to 25 years. EFA and CFA supported the scales structure, and multigroup CFA indicated that the measure performs well across genders. Cronbach’s alpha demonstrates good internal consistency for each subscale.
This is the first study to develop and validate a measure of young people’s social media use, confidence, and safety when communicating online about suicide and self-harm, and in doing so addresses a significant gap in evidence. This measure will support future research in this area and help advance the evidence base pertaining to young people’s online safety when it comes to self-harm and suicide-related content, as well as their experiences communicating about their own self-harm and suicidal behaviors online.
However, the study is not without limitations. Firstly, it does not assess concurrent validity. As there are no existing measures of young people’s behavior, confidence, and safety when communicating online about suicide and self-harm, we were unable to determine how the #chatsafe Online Safety Scale correlates with established measures. Secondly, the sample includes Australian young people aged 16 to 25 years, the majority of whom speak English at home and have some level of university education, and therefore findings may not be representative of all young people, particularly those from culturally or linguistically diverse backgrounds. Future validation with more diverse samples is recommended. Thirdly, this scale does not distinguish between intentional and unintentional exposure.
Conclusion
This study provided preliminary validation for the #chatsafe Online Safety Scale, the first measure of its kind to assess young people’s online behavior, confidence, and safety when communicating about suicide and self-harm. The scale demonstrates sound factor structure, good internal consistency, and measurement invariance across genders. It can be used in future studies exploring young people’s online experiences and behaviors related to self-harm and suicide, providing a stronger and comparable evidence base to guide policy and practice.
Footnotes
Authors’ Contributions
B.K. conceptualization, methodology, formal analysis, data curation, writing—original draft; J.R. conceptualization, methodology, writing—review and editing, supervision; C.G. formal analysis, data curation, writing—review and editing; C.C. conceptualization, writing—review and editing; S.M. conceptualization, methodology, writing—review and editing; A.S. methodology, writing—review and editing; L.L.S. conceptualization, methodology, writing—original draft, supervision.
Ethical Approval and Consent to Participate
This study was approved by the University of Melbourne Human Research Ethics Committee (HREC24238). Informed consent was obtained from all participants in this study.
Author Disclosure Statement
No competing financial interests exist.
Funding Statement
No funding was received for this article.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
