Abstract
Objective:
Self-report questionnaires are common for measuring posttraumatic stress disorder (PTSD). The experience of life threat—Criterion A—serves a gatekeeper function for diagnosing PTSD, and evidence suggests false positives are common on questionnaires. It remains unknown how common they are and whether extra instructions can reduce them.
Methods:
The present study assessed 42 youths, 10–17 years of age, from a clinic setting. Youths and parents completed regular PTSD questionnaires and then enhanced versions with more detailed instructions and examples of Criterion A and non-Criterion A events. Parents completed a semistructured interview as the verification of true versus false positives.
Results:
In the full sample, parents endorsed 41 and children endorsed 45 false positive events. The mean was significantly greater than zero for both parents and children. Parents endorsed 59 and children endorsed 138 false positive symptoms. When false positive events were endorsed, this was significantly associated with more false positive symptoms for both parents and children. An enhanced questionnaire failed to reduce false positive events for the full sample.
Discussion:
The common occurrence of false positives suggests caution is warranted when interpreting estimates from questionnaire-based research about the prevalence of PTSD. While this attempt to eliminate false positives was not fully successful, there may be other useful enhancements to consider in future research.
Introduction
Self-report measures, also known as questionnaires, are commonly used in posttraumatic stress disorder (PTSD) research. Accurate data about PTSD requires a two-step algorithm of identifying a qualifying life event followed by the symptoms that developed from that event. There have been no known studies that used interviews concurrent with questionnaires to determine how often both false positive events and false positive symptoms are reported. This research is meant to fill this gap by comparing questionnaire responses to interview responses.
The importance of this issue is that large-scale surveys addressing important epidemiological and public health issues often rely on questionnaires because interviews are prohibitive due to cost and time. For example, the Centers for Disease Control conducted an online survey of 26,174 public health workers to ascertain their mental health status during the COVID-19 pandemic (Centers for Disease Control and Prevention, 2021). They found 36.8% endorsed pandemic-related PTSD, which is more than ten times the past-year general population rate of 3.5% (Kessler et al., 2005). Another group of researchers screened 94 healthcare workers about COVID stress and found the frequency of PTSD was 25.4%. However, when the same sample was interviewed, the frequency of PTSD was only 7.9% (Scott et al., 2023).
Only two other known studies calculated false positive diagnoses. In a study of 118 Cambodian community adults, 47% were identified with PTSD on a questionnaire, but only 20.3% with interviews (Silove et al., 2007). In contrast, in a study of 98 crime victims in Brazil, all of whom were recruited because they had a Criterion A event, 48% were identified with PTSD on a questionnaire, and nearly the same proportion, 51%, with interview (Pupo et al., 2011). Neither study attempted to calculate false positive events or symptoms, and neither identified cases from questionnaires based on the diagnostic algorithm; cases were identified by surpassing a cutoff score of symptom severity. In addition, the Cambodian participants were not from a clinical (i.e., help-seeking) population, thus limiting the generalizability of findings.
The aims of the current study were threefold. First, this study will document for the first time the extent of false positives for trauma events generated with questionnaires and verified by interviews. A clinic population was chosen because that setting would have the most direct application for individuals seeking help. Hypothesis 1 is that false positives will be significantly greater than zero on questionnaires for both children and parents.
Second, this study will document for the first time the relation between false positive events and false positive symptoms. Hypothesis 2 is that those with false positive events will endorse more false positive symptoms compared to those with no false positive events.
Third, the questionnaire will be administered a second time with expanded instructions to educate respondents on the difference between traumatic versus stressful events. Hypothesis 3 is that respondents will endorse fewer false positive events and false positive symptoms on the enhanced questionnaire. Each of these questions will be explored separately for children and caregivers as respondents.
Methods
Participants
Consecutive new intakes were recruited at a private outpatient child and adolescent mental health clinic, Child Counseling Associates, located in Metairie, Louisiana. Inclusion criteria: Ten- through 17-year-old patients presenting to the clinic for outpatient care. Exclusion criteria: psychosis, autism, individuals with active suicidal plans, blindness, deafness, and non-English speaking families. The routine clinical intake procedure at the clinic screened out youths with these issues. From September 1, 2020 to October 20, 2022, there were 81 new intakes in our age range. Four were referred out of the clinic after the intake session. Seven dropped out after one or two sessions and were not approached. Two did not fill out intake measures. Five were not approached about the study because parents did not attend sessions after the intake and were too difficult to reach. One was not approached by mistake. Of the remaining 62 who were eligible, 10 declined when approached, seven verbally agreed and then failed to sign consent forms, two signed consent forms and then failed to complete measures, one could not complete all measures, and the remaining 42 signed consents and completed all measures.
Procedure
All new patients at this clinic completed a set of questionnaires at their first visit as part of routine procedure. Caregivers completed a parent-report version while ten- to 17-year-old children completed a child-report version. At their second or third visit, their clinicians asked them if they would be willing to talk with a research assistant about the study. If they agreed, a research assistant would meet with them in a separate office to obtain written informed consent from both caregivers and children. Completion of enhanced questionnaires and parent interviews happened at their next visit. If time was not permitting or one party was not present, these sometimes happened after two or three more visits. The study was approved by the Tulane Committee on Use of Human Subjects.
Questionnaire
Regular PTSD
The Child PTSD Checklist (CPC) is a self-administered questionnaire (Sims et al., 2024). The first part is Criterion A events, which includes a conventional definition of life-threatening traumatic events: “To count an event, your child must have experienced one of these: (1) felt like he/she might die, or (2) he/she had a serious injury or felt like he/she might get a serious injury, or (3) he/she saw (1) or (2) happen to another person, or saw someone die.” There were 10 types of events (see Table 1), and an eleventh option for Other. The second part includes 21 items mapped directly onto the 20 PTSD symptoms (two items are used to measure the dissociation symptom) from the Diagnostic and Statistical Manual, Fifth Edition (American Psychiatric Association, 2013). Each item was rated on a 5-point (0–4) Likert scale. The range of possible scores was 0–84. There was a parent/caregiver version (CPC-P) and a child version for children 10 years of age and older (CPC-C).
Sample Characteristics
N, 42.
N = 40, two fathers were deceased.
N = 41.
Enhanced PTSD
The enhanced version was identical to the CPC except for the addition of four types of added instructions: (1) A definition of stressful events was added to the introductory definition of Criterion A traumatic events to demonstrate the difference; (2) examples of traumatic events and stressful events were added to each of the 10 traumatic events to illustrate the difference; (3) an instruction to stop the questionnaire after the menu of trauma events if all the events that were endorsed happened before three years of age; and (4) an instruction to stop the questionnaire after the menu of trauma events if no trauma events were endorsed. See Supplementary Appendix for the exact wordings. The rationale for excluding events that happened before three years of age was based on the literature that the existing case reports of children under three years with suspected PTSD does not contain convincing cases of PTSD (Scheeringa, 2009).
Semi-structured interview
The Kiddie Schedule of Affective Disorders and Schizophrenia Present and Lifetime Version for School-Age Children (K-SADS) is a semi-structured interview about children’s Diagnostic and Statistical Manual-5 psychiatric disorders. We used only the PTSD module, which has shown good test-retest reliability (Cohen’s kappa = 0.60), and construct validity (Kaufman et al., 1997). The PTSD module of the Screen Interview was used to assess for traumatic events. Four items were added—fire, animal attack, burn, and near drowning—to be consistent with the CPC. To be consistent with the CPC, only events that happened at three years of age or older were counted.
The K-SADS probe question was read verbatim and the respondent was allowed to respond. For trauma events, a simple “yes” was not accepted. If caregivers endorsed events, they were asked to describe them. Respondents had to convince interviewers that events met the life-threat criterion. Further, respondents had to describe children’s reactions to the events to further document the experience of life threat. Did children appear panicked? Did children say they were frightened? If caregivers did not witness children’s reactions firsthand, they could describe how witnesses or children themselves described their reactions to them. For PTSD symptoms, a similar interview process of iteration and clarifying probes was used. As with trauma events, a simple “yes” answer was never accepted. Respondents had to provide examples of the symptoms to convince interviewers that symptoms existed. Interviewers probed about symptoms in relation to every trauma event that was endorsed.
Two research assistants were trained to conduct the interviews by the principal investigator (PI). Both had undergraduate degrees and two to three years of experience as full-time assistants in clinical research with child and adolescent psychiatric populations. They were not clinicians. Each interview was recorded and the PI reviewed every recording jointly with each assistant to ensure fidelity to PTSD definitions and prevent drift. One assistant conducted the first 15 interviews, and the other assistant conducted the remaining 27 interviews.
Data analysis
Parent interview was used as the criterion for determining true positives because they would have more reliable memory of past events compared to children and adolescents. Hypothesis 1—that false positives will be significantly greater than zero on regular questionnaires—were tested with Wilcoxon one-sample signed-rank tests. Children and parent reports were tested separately. Age, sex, and race were examined as potential covariates. Pearson correlations were used when both variables were continuous, and point biserial correlations were used when one variable was dichotomous.
Hypothesis 2—that false positives events will positively associate with false positive symptoms—was tested with Wilcoxon rank-sum tests. False positive symptoms were calculated as the sum of symptoms on regular questionnaires that were not endorsed on K-SADS interviews.
Hypothesis 3—that an enhanced questionnaire will decrease the number of false positive events and false positive symptoms—were tested with one-sample t-tests or Wilcoxon signed-rank tests.
Sample size estimation for hypothesis 1
There were no known prior studies on this topic on which to base a calculation for an expected effect size. If respondents endorsed one fewer event on the enhanced questionnaire compared to the regular questionnaire this is likely to be clinically significant. In our experience with this measure at this clinic, the mean number of trauma events endorsed has been 1.8 (SD = 1.7). If the mean number of trauma events endorsed is one less with the enhanced questionnaire (M = 0.8), the sample size needed for 80% power is 27 subjects.
Results
Table 1 shows the demographics of the 42 participants. The age of children ranged from 10 to 17 years, and the median was 14 years. All parent respondents were the biological mothers. Table 2 shows the mean number of trauma events, plus number and severity of PTSD symptoms for each questionnaire. Number of false positive endorsements of events as traumas.
Descriptive Statistics for Each Measure
PTSD, posttraumatic stress disorder.
Twenty-seven parents and 23 youths endorsed at least one false positive event. False positives on regular questionnaires were significantly larger than zero for both parents and children (Table 3). Table 4 shows the most common type of event for false positives was natural disasters, and the second most common was medical procedures. The frequencies of false positives for parent-report and child-report were not significantly correlated with age, sex, or race of the children.
Number of False Positives with Regular Questionnaires
False Positives, True Positives, False Negatives, and True Negatives for Regular Questionnaires
There were no true positives on the K-SADS.
FN, false negative; FP, false positive; TN, true negative; TP, true positive.
Relation of false positive events with false positive symptoms.
Parents endorsed 59 and children endorsed 138 false positive symptoms. Eighteen parents and 21 youths endorsed at least one false positive symptom. Parents who endorsed at least one false positive event endorsed significantly more false positive symptoms on regular questionnaires compared to parents who endorsed zero false positive events (see Table 5).
Relation of False Positive Events to False Positive Symptoms
SD, standard deviation; FP, false positive.
Also, children who endorsed at least one false positive event endorsed significantly more false positive symptoms compared to children who endorsed zero false positive events. Can enhanced instructions on a questionnaire reduce false positive events and symptoms?
The duration between regular and enhanced questionnaires for parents was M = 57.4 days, SD = 30.8, median = 49, range 21–139, and for children was M = 58.2 days, SD = 32.1, median = 46.5, range 21–139. No participants experienced new true positive traumatic events between administration of the two questionnaires.
For both parents and children, contrary to the hypothesis, the number of false positive events on enhanced questionnaires did not decrease compared to regular questionnaires (Table 6).
False Positives of Regular Versus Enhanced Questionnaires
FP, false positive; SD, standard deviation.
Inspection of the data revealed that the false positives reported on regular questionnaires often did disappear with the enhanced questionnaire, but new false positives appeared on the enhanced questionnaire which produced no overall mean change. Specifically, of the 41 false positive events endorsed by parents on the regular questionnaire, 25 were not endorsed on the enhanced questionnaire, but 19 new false positives appeared on the enhanced questionnaire that were not endorsed on the regular questionnaire. Of the 45 false positive events endorsed by children on the regular questionnaire, 22 were not endorsed on the enhanced questionnaire, but 21 new false positives appeared on the enhanced questionnaire that were not endorsed on the regular questionnaire.
The number and severity of false positive symptoms were not significantly different for regular compared to enhanced questionnaires for both parents and children as respondents (Table 6).
Discussion
This study documented for the first time the frequency of false positives for identifying trauma events on questionnaires when verified with interviews. For both parents and children, the presence of false positive events was associated with false positive symptoms. We failed, however, to show that a questionnaire enhanced with more detailed definitions, instructions, and examples could fully eliminate false positives. These findings have three main implications for interpreting previous research and implications for future research.
First, epidemiological research that relies on questionnaire data often may appear to inflate prevalence rates for PTSD. For adults, questionnaire-based past-year prevalence of PTSD adults was 4.7% in one major study (Kilpatrick et al., 2013). Interview-based past-year in adults was lower, 3.5%, in one study (Kessler et al., 2005) but higher in another, 7.3% (Norris, 1992). For youths, questionnaire-based lifetime prevalence of PTSD in adolescents was 14.6% in one study (Elklit and Petersen, 2008). Interview-based lifetime prevalence was lower in two studies: 9.2% (Breslau et al., 1991) and 4.7% (McLaughlin et al., 2013). Our findings suggest these discrepancies may be due to false positives.
Second, these findings may shed light on how some questionnaire-based studies of targeted cohorts have produced surprisingly high prevalence rates of PTSD. The COVID studies in the introduction section provide one example, but there are many other studies showing high rates of PTSD in individuals who never faced life-threat, such as divorce of parents (Joseph et al., 2000), living near an oil spill (Mong et al., 2012), and farmers who culled diseased livestock (Olff et al., 2005). In combination with our findings, these studies suggest that questionnaires are open to bias, and may be inappropriate for cohorts following uncommon stressful events.
Third, the finding that new false positives were not substantially eliminated with the enhanced questionnaire was disappointing. This was the first study to test enhanced instructions for PTSD, and our results must be viewed with caution. If this finding is replicated, it suggests that questionnaires should never be used alone for PTSD, at least for trauma events.
While the enhanced questionnaire did not meet the study expectations to eliminate all false positive events, the strategy of enhancing existing questionnaires may deserve further study. It did eliminate nearly half of the false positive events in both adults and children, and it decreased false positives without adding new false positives in a subgroup of participants. One possible enhancement is that all DSM editions of PTSD have emphasized in the text that events that cause PTSD are sudden and unexpected (Marx et al., 2024), but that emphasis has never made it into the criteria or questionnaires. There may be other useful types of enhancements to consider (Rubin et al., 2024).
Limitations
We were not able to address a possible order effect for the administration of questionnaires. The appearance of new false positives on the enhanced questionnaire, and the lack of a significant difference in symptom severity between measures, however, suggests no order effect was present in this study. Results may have been different if we used child interviews in combination with parent interviews to verify false positives. Some have argued that the definition of trauma events needs to be expanded to nonlife-threat stressful events, but that is beyond the scope of this study that examined how well respondents are willing or able to follow questionnaire instructions. False positives may have been erroneously inflated if children or parents endorsed events on regular questionnaires but parents failed to acknowledge them during interviews because of shame, guilt, or embarrassment. This would most likely occur with events of physical abuse, sexual abuse, and domestic violence. The possibility of erroneous false positives in this sample, however, seems rare because there were a total of only seven false positives for those events, and parents appeared to freely discuss all but one of those events during interviews. These findings were from a help-seeking, clinic population and may not be generalizable to nonclinic populations. The sample size was small, and the findings ought to be considered preliminary that need replication.
Conclusion
The overall aim was to discover the extent to which false positives are generated on questionnaires for trauma events and symptoms. This study clarified that false positives are endorsed for events and symptoms in approximately half of a child and adolescent help-seeking population, but they are not distributed evenly; a minority of the sample produces most of the false positives. Our preliminary evidence did not support the premise that enhanced instructions could reduce false positives in every respondent, although it appeared to work for a subset. Future studies that use questionnaires for PTSD ought to provide substantially more caution in the interpretations of estimates derived from this method. The benefits of saving time and money to assess large samples may come with a sizeable cost of providing a degree of misleading information.
Clinical Significance
Misdiagnoses in clinic settings can give patients incorrect impressions of the cause of their problems and lead to misguided treatment planning. Assessment of PTSD with standardized measures can help identify accurate diagnoses. Standardized measures are used in some clinics for all new intakes and done in other clinics on an as-needed basis after clinical interviewing has detected past traumas. In both situations, these findings suggest that Criterion A events need to be reviewed by clinicians prior to interpreting symptom scores to prevent false positive diagnoses.
Footnotes
Acknowledgment
The author thanks research assistants Catalina Pacheco and Jenna Sims, and clinicians Ruth Arnberger, LCSW and Marti Tidwell, LCSW for assistance with this research.
Disclosures
The author receives royalties from Guilford Press and Central Recovery Press.
Supplementary Material
Supplementary Appendix
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
