The Assessment of Situational Judgement Questionnaire: A Novel Instrument to Detect Susceptibility to Financial Scamming

Abstract

Background:

Existing measures of scam susceptibility lack ecological validity and situational variability. Evidence suggests that all adults may be susceptible to scams, though a comprehensive fraud victimization theory remains to be explored.

Objective:

To identify cognitive and sociodemographic variables that differentiate individuals with high scam susceptibility from those less susceptible. This article describes the development and feasibility of the Assessment of Situational Judgment questionnaire (ASJ), a brief tool designed to detect scam susceptibility.

Methods:

The 17-item ASJ was developed using a combination of existing scams reported by the Florida Division of Consumer Services and legitimate scenarios. Participants were presented with scam and legitimate scenarios and queried regarding their willingness to engage. Response options were offered with instructions on a 7-point Likert scale (extremely unlikely to extremely likely). Pilot data from a development sample provided the foundation for the final version of the ASJ.

Results:

The final version of the ASJ was administered to 183 online participants. The Scam factor (8 items) explained 50.6% of the variance. The Legit factor (9 items) reported on a 7-point Likert scale explaining 10.6% of the variance. A Scam to Legit ratio provides a proxy for overall scam susceptibility. Cut-off scores of 24 on the Scam factor, 47 on the Legit factor, and 0.62 on the ratio optimize measures of scam susceptibility.

Conclusions:

The ASJ is a brief, ecologically valid measure of scam susceptibility. There is a need for a sensitive and specific tool to detect scam susceptibility in clinical, community, and financial settings.

Keywords

Alzheimer’s disease deception decision making scam susceptibility vulnerable populations

INTRODUCTION

Financial exploitation is an ever-evolving public health threat that is increasing at an alarming rate and has been particularly well-documented in older adults. As one of the most common forms of elder abuse [1], financial exploitation has cost older adults in the United States up to $3 billion in losses [2]. The complexity and maliciousness of scams are ever changing [3] and even the most common forms of scams have perniciously morphed into scenarios that are increasingly challenging to detect, even in individuals without impaired cognition. The evasion of scams requires vigilance and intact higher order cognitive processes that may change as a function of age and with development of cognitive impairment. Interestingly, there is also data indicating Gen Z individuals (age 20 or younger) with increased technological savvy, have had a 156% surge in cyber fraud compared to a 112% growth amongst individuals aged 60 and over, making these two groups the fastest growing scam cohorts between 2017 and 2020 [4], although likely due to differing reasons. For example, younger individuals lacking wisdom gained through life experience, and older adults having a higher risk of subtle to not-so-subtle cognitive impairments.

Despite the severity of this pressing public health concern, research in this area is predominantly reliant on survey-based methods after-the-fact. There are limited tools to measure scam susceptibility that incorporate ecological validity and situational variability. For example, one existing measure to assess scam susceptibility is a face valid 5-item self-report questionnaire in which participants rate their agreement to statements such as “If something is too good to be true, it usually is” using a 7-point Likert scale [5 –7]. This measure is heavily relied upon in the literature with successful prediction of scam susceptibility. Similarly, Nolte et al. [8] have developed a tool to measure susceptibility to COVID scams using actual scams and one legitimate item. While these tools can measure aspects of scamming susceptibility, they are limited in the scope of situations evaluated underscoring the need for an ecologically valid and situationally varied instrument that is easily implemented in the community and clinical practice across age cohorts. Given this need, our goal was to develop a measure of situational decision making across scenarios involving both deception and scams as well as legitimate situations that be easily administered both online and in person.

To this end, we developed the Assessment of Situational Judgment questionnaire (ASJ). The ASJ is novel in that it includes situations designed to reflect actual scams in the state of Florida as identified by the Florida Division of Consumer Services as well as benign legitimate situations requiring discretional judgment of the participant. Our aim was to distinguish factors that contribute to greater susceptibility to scams versus more precautious judgement. We identified 8 scam scenarios and 9 legitimate scenarios via an initial pilot study and tested in a development sample. An initial factor analysis indicated that two of the legitimate items required modification as they were frequently perceived as scams and were then adjusted for implementation in an independent test sample. Responses on the ASJ items were then evaluated in the new sample in conjunction with assessments of experience and understanding of financial scams and the patient-version of the AD8, a brief, easily self-administered measure of cognitive function [9 –11]. Here, we describe the development and validation of the ASJ in relationship to cognitive functioning and demographic profiles. We hypothesized that individuals with higher AD8 scores, supporting the presence of cognitive impairment, would perform worse on the ASJ and that individuals with better social connectiveness would perform better.

METHODS

Data collection for the ASJ was approved by the University of Miami Institutional Review Board.

Instrument development

The ASJ (Fig. 1) was developed using a combination of existing scams reported by the Florida Division of Consumer Services and legitimate scenarios. For Scam items, participants were asked to rate their willingness to engage in actual scam scenarios such as, “An IRS representative calls and indicates that you have an outstanding debt to the IRS and if a payment is not received immediately, you could be arrested. The caller ID indicates ‘IRS.’ The caller instructs you to purchase a prepaid debit card to settle the debt. Would you buy the prepaid card?” For Legitimate items, participants were asked to rate their willingness to engage in potential scenarios across a variety of domains (e.g., online, in person, via telephone) such as, “You cannot find a taxi and are in a rush to get to the airport. Your travel companion recommends that you take an Uber, but you do not have the app. Would you download the app and enter your credit card information?” Response options were offered with instructions on a 7-point Likert scale (extremely unlikely to extremely likely; Extremely unlikely = 1, unlikely = 2, slightly unlikely = 3, either = 4, slightly likely = 5, likely = 6, and extremely likely = 7). Likert scale responses were averaged to obtain Scam and Legitimate scores with higher scores indicating greater likelihood to engage in the scenario. A series of 20 questions were generated and tested internally and 17 questions (8 Scam, 9 Legitimate) were selected based on internal feedback from the study team regarding readability, feasibility, and participants’ ability to comprehend the question. This study was deemed exempt and approved by the University of Miami Institutional Review Board.

Fig. 1

Assessment of situation judgment.

Development sample

An initial development sample was recruited via Amazon’s Mechanical Turk (MTurk; http://www.MTurk.com), an online marketplace and crowdsourcing platform commonly used in psychology, political science, and decision-making research that provides data at least as reliable as those obtained via traditional methods [12]. MTurk participants have been shown to be more diverse than many existing Internet samples [12] and this diversity has remained demographically stable despite COVID-19 [13]. Moreover, it has been demonstrated that MTurk data is just as if not more representative of the broader population as compared to data collected via traditional methods [14, 15]. Participants received compensation of $3 for their time (average: 10 min and 51 s) [12]. Only participants with IP addresses in the United States were included. Using MTurk, a link to the ASJ was provided to Survey Monkey (http://www.surveymonkey.com), another online crowdsourcing platform commonly used by psychologists, political scientists, and decision-making researchers. Before completing the survey, participants were introduced to the purpose of the research and were provided with the opportunity to either provide or withhold their consent.

The development sample consisted of 204 participants (59.3% female), average age 57.6 years (range 22–91 years). Seven additional participants registered but did not complete data collection. The educational attainment of the sample included 8.8% completed high school, 20% completed some college, 22% completed college, and 28.9% completed some post-graduate education. Thirty-six percent of the participants reported living alone and participants reported using a computer for an average of 6.8±4.2 h/day. A factor analysis using Principal Axis with Varimax rotation [16] was used to study the structure of the ASJ and revealed a 3-factor solution. Scam items all strongly loaded into 1 factor (Scam), had an Eigenvalue of 7.7, and explained 51.4% the variance. Legitimate items loaded across two factors (Legit 1, Legit 2). When the entire model was forced into a 2-factor solution, the Legit factor had an Eigenvalue of 2.0, explaining 13.7% of the variance. However, two of the Legitimate questions were commonly perceived as scams and were subsequently reworked in the final version of the ASJ (Fig. 1). Performance across the development sample revealed that older participants were on average better able to discriminate scam items than younger participants suggesting that (a) there is no apparent increase of financial scamming due solely to age and (b) to better understand why older adults become victims of financial scamming, an assessment of cognitive functioning needed to be included.

Test sample

For the test sample, in addition to the revised ASJ, the self-report AD8 was included to screen for cognitive impairment [9]. The AD8 is a brief, sensitive measure that validly and reliably differentiates between nondemented and demented individuals and has been used as a general screening device to detect cognitive change regardless of etiology. The AD8 has been validated both as an informant interview and a self-report interview against gold standard assessments including the clinical dementia rating scale and neuropsychological testing [9 –11]. In the present study, the self-report version was used. Eight questions were asked of the participant: whether they experience: 1) Problems with judgment; 2) Reduced interest in hobbies/activities; 3) Repeating questions, stories, or statements; 4) Trouble learning how to use a tool, appliance, or gadget; 5) Forgetting to correct month or year; 6) Difficulty handling complicated financial affairs; 7) Difficulty remembering appointments; and 8) Consistent problems with thinking and/or memory. The following instructions were given to the participant: “Remember, ‘Yes, a change’ indicates that you think there has been a change in the last several years caused by cognitive (thinking and memory) problems.” Items endorsed as “Yes, a change” are summed to yield the total AD8 score. An individual with an AD8 score > 2 is likely to have cognitive impairment. Questions regarding prior experience with financial scamming were also asked of the participant covering topics including recognition of the problem as a national concern, knowledge of types of scams, awareness of friends and family members who were scammed, as well as whether the participant was scammed, and if so, how they would respond. Both questionnaires were administered in English.

Statistical analysis

Statistical analyses were conducted using IBM SPSS v28 (Armonk, NY) [17]. Descriptive statistics were used to summarize overall sample characteristics. Student t-tests or One-way analysis of variance (ANOVA) with Tukey-Kramer post-hoc tests were used for continuous data and Chi-square analyses were used for categorical data. To assess ASJ item variability, the item frequency distributions, range, and standard deviations were calculated. Item and subscale scores were examined for floor and ceiling effects. A factor analysis was used to determine the individual factor loading of each of the Scam and Legit items for the test sample. The association between ASJ Scam and Legit factor scores with participant characteristics (age, education, and AD8 score) was determined using Pearson product–moment correlation coefficients. Internal consistency was examined as the proportion of the variability in the responses that is the result of differences in the respondents, reported as the Cronbach alpha reliability coefficient. Coefficients greater than 0.7 are good measures of internal consistency [18]. In addition to the two factor scores, a scam to legit ratio was created as a proxy for overall scam susceptibility with a possible range from of scores from 0.13 to 6.22. We hypothesized that lower ratio scores would indicate greater ability to detect both scams and legitimate scenarios. Higher ratio scores would indicate both a greater likelihood to fall for a scam as well as less ability to differentiate between scam and legitimate scenarios.

Group validity was assessed by examining the ASJ Scam and Legit scores and the ratio of Scam to Legit scores by sample characteristics (sociodemographic and lifestyle characteristics, AD8 scores, computer, and social media usage, as well as prior exposure and perceived vulnerability to scams). The ASJ Scam and Legit scores and Scam to Legit ratio were compared by cognitive status based on AD8 scores (0–1 versus > 2). Multiple comparisons were addressed using the Bonferroni correction.

Finally, to provide cut-off scores for the ASJ that could be used in clinical practice and future research, we conducted a receiver operator characteristics curve analysis. As there is no known Gold Standard for situational judgment, we used the AD8 scores (0-1 versus 2 or greater) as an indicator of impaired judgment. We report the area under the curve (AUC) and 95% confidence intervals (95% CI). Cut-off scores were determined using the Youden Index which represents the maximal potential effectiveness of a test to classify disease. A Youden Index of 0.5 or greater supports that the test meets empirical benchmarks for diagnostic purposes.

RESULTS

Test sample demographics

One hundred and eighty-three participants completed the ASJ. Restrictions were set so that only participants with IP addresses in the United States over age 55 with a Human Intelligence Task (HIT) approval percentage over 75% could participate. The task was hidden from the public and only viewable to those that met the criteria. Overall sample characteristics are displayed in Table 1. The participants’ mean age was 63±5.3 years (range 53 to 79 years), 61.7% were female, and 93.4% were non-Hispanic white. Mean education was 15.9±2.2 years (range 12 to 20 years), mean daily time spent on the computer was 6.7±2.7 (range 1 to 12 hours), 94.5% used social media, and 31.7% lived alone. The participants’ cognitive status indicated that 39.3% endorsed cognitive impairment (mean AD8 score was 1.8±2.4; range 0 to 8). Mean Scam scores were 23.1±16.8 (possible range 8 to 56; range 8 to 55) and mean Legitimate scores were 44.5±8.9 (possible range 9–63; range 20 to 61). The mean ratio of Scam to Legitimate scores was 0.49±0.30 (possible range 0.13 to 6.62; range 0.13 to 1.04). The participants’ responses regarding scam experience and knowledge indicated that 50.6% had prior exposure to scams, 63.5% indicated they were able to recognize an attempted scam, 44.4% reported being moderately familiar with the national problem of scamming (refer to Table 1 for other levels of familiarity with scam types), 35.4% indicated they had friends that were scammed, and 86% reported that they were able to recognize a scam. When asked which population segment was most vulnerable to scamming, 40.4% of participants indicated that older adults were most susceptible (refer to Table 1 for other categories). Regarding perceived vulnerability to scams 34.8% of participants reported they were vulnerable, 15.2% indicated they were a scamming victim, 80.9% would tell a family member if they were scammed, 85.4% would report to authorities if they were scammed, and the majority (63.4%) of participants reported that they believed online scams would be the most effective scamming modality.

Table 1

Sample characteristics (n = 183)

Variable	Mean (SD) or %	Range
Age, y	63.0 (5.3)	53–79
Sex, % Female	61.7
Race, % Non-Hispanic White	93.4
Education, y	15.9 (2.2)	12–20
Lives alone, % Yes	31.7
Daily Time Spent on Computer, hours	6.7 (2.7)	1–12
Social Media Account, % Yes	94.5
AD8 Scores	1.8 (2.4)	0–8
% Subjective Impairment	39.3
Scam Score	23.1 (16.8)	8–55
Legit Score	44.5 (8.9)	20–61
Scam/Legit Ratio	0.49 (0.30)	0.13–1.04
Prior Exposure to Scamming Scenarios, % Yes	50.6
Recognized Attempted Scam, % Yes	63.5
Familiar with National Problem of Scamming, %
Not Familiar	2.2
Slightly Familiar	17.4
Somewhat Familiar	16.9
Moderately Familiar	44.4
Extremely Familiar	19.1
Friends Who Were Scammed, % Yes	35.4
Able to Recognize Scam Attempt, % Yes	86.0
Group Most Vulnerable to Scamming, %
Young Adults	11.2
Mid-Life Adults	19.4
Older Adults	40.4
Adults with Cognitive Impairment	29.2
Perceived Self-Vulnerability, % Yes	34.8
Scamming Victim, % Yes	15.2
Would Tell Family if Scammed, % Yes	80.9
Would Report to Authorities if Scammed, % Yes	85.4
Most Effective Scamming Modality, %
In-person	5.6
Online	63.4
Phone	26.4
Mail	4.5
Mean (SD) or %

Performance of ASJ on the test sample

Table 2 demonstrates the properties of the ASJ Scam and Legit factors with loading scores, Eigenvalues, inter-item correlations, item-factor correlations, and the correlations between the ASJ items and AD8 scores. The Scam factor had an Eigenvalue of 8.6 contributing 50.6% of the variance of the ASJ. Individual items for the Scam factor were strongly correlated with each other suggesting they all capture similar aspects of susceptibility to being financially scammed. Scam items had strong factor loadings and strong item-factor correlations. Scam items had moderate correlation with AD8 scores supporting an association between scam susceptibility and cognitive functioning. The Legit factor had an Eigenvalue of 1.8 contributing 10.6% of the variance of the ASJ. Individual items for the Legit factor were weakly-to-moderately correlated with each other suggesting they each capture different aspects of legitimate decision-making scenarios. Legit items had moderate factor loadings and item-factor correlations. Legit items had weak-to-moderate correlation with AD8 scores suggesting that decisions for legitimate scenarios were less associated with cognitivefunctioning.

Table 2

ASJ Item Distributions, Factor Loading, Inter-Item and Item-Factor Correlation, and Convergence with AD8 Scores

	Mean (SD)	Inter-Item Correlations									Factor	Item-	Item-AD8
Scam Scenarios		S1	S2	S3	S4	S5	S6	S7	S8		Loading	Factor R	Score R
Disaster relief (S1)	3.17 (2.11)	1									0.896	0.917	0.510
IRS threat (S2)	2.70 (2.22)	0.86	1								0.937	0.966	0.567
Credit card (S3)	2.92 (2.22)	0.85	0.91	1							0.923	0.952	0.503
Work from home (S4)	3.09 (2.25)	0.81	0.87	0.85	1						0.914	0.924	0.489
Warrant (S5)	2.82 (2.22)	0.89	0.93	0.92	0.89	1					0.947	0.971	0.532
Remote access home computer (S6)	2.78 (2.19)	0.85	0.92	0.91	0.86	0.91	1				0.925	0.956	0.557
Internet romance (S7)	2.91 (2.26)	0.88	0.94	0.91	0.89	0.93	0.91	1			0.943	0.967	0.567
Unclaimed money (S8)	2.75 (2.19)	0.85	0.93	0.90	0.87	0.91	0.92	0.92	1		0.936	0.959	0.540
Eigenvalue											8.6
% Variance Explained											50.6
Legit Scenarios		L1	L2	L3	L4	L5	L6	L7	L8	L9
Uber (L1)	5.20 (1.67)	1									0.379	0.468	0.127
Phone company (L2)	4.33 (2.03)	0.17	1								0.380	0.631	0.342
Academic survey (L3)	5.26 (1.77)	0.07	0.34	1							0.609	0.589	0.239
Air BnB (L4)	4.42 (1.82)	0.42	0.27	0.15	1						0.396	0.577	0.200
EBay (L5)	5.25 (1.42)	0.30	0.02	0.15	0.29	1.00					0.605	0.464	0.160
Work IT (L6)	4.75 (1.90)	0.08	0.38	0.35	0.18	0.10	1				0.506	0.625	0.239
Apple store (L7)	5.72 (1.23)	0.18	0.19	0.37	0.16	0.32	0.15	1.00			0.710	0.471	0.111
Bank fraud (L8)	4.41 (2.06)	0.15	0.38	0.36	0.26	0.21	0.43	0.08	1		0.469	0.667	0.268
Program install (L9)	5.18 (1.61)	0.09	0.29	0.19	0.19	0.22	0.36	0.23	0.34	1	0.527	0.572	0.267
Eigenvalue											1.8
% Variance Explained											10.6

Reliability and scale score features of the ASJ

The internal consistency of the ASJ was assessed with Cronbach alpha (Table 3). The internal consistency was good to excellent (0.98 for the Scam factor, and 0.73 for the Legit factor). The ASJ Scam and Legit factors and the ratio of Scam to Legit factors covered most of range of possible scores and the mean, median and standard deviation demonstrated a sufficient dispersion of scores for assessing the full caregiving experience with low percentage of missing data. There were minimal floor and ceiling effects for the Legit factor and ratio. While there was no ceiling effect for the Scam factor, there was a 20% floor effect. The Scam and Legit factors were moderately correlated with each other supporting that they tend capture distinct aspects of decision making and susceptibility to scamming (r = –0.59). The Scam/Legit ratio was correlated with both factors but was more strongly correlated with the Scam factor (r = 0.97 versus 0.41).

Table 3

ASJ Scale Features: Internal Consistency, Score Distributions, and Inter-Scale Correlations

ASJ Properties			Reliability	Score Features and Distribution						Inter-Scale Correlation Spearman ρ
Domain	Items	Possible Range	Cronbach alpha (95% CI)	Range	Mean	Median	SD	% Floor	% Ceiling	Scam	Legit	Ratio
Scam	8	8–56	0.985 (0.982–0.988)	8–55	23.1	13.0	16.8	20.2	0	1
Legit	9	9–63	0.735 (0.673–0.789)	20–61	44.5	45.0	8.9	0	0	0.595	1
Ratio	n/a	0.13–6.22	n/a	0.13–1.04	0.49	0.33	0.30	0.5	0	0.970	0.408	1

Note: % Floor is the percentage with lowest possible score. % Ceiling is the percentage with highest possible score. n/a, not applicable.

There was low-to-moderate negative correlation between age and the Scam factor, Legit Factor and Scam/Legit Ratio low-to-moderate positive correlation between the total AD8 score and the Scam factor, Legit Factor and Scam/Legit Ratio (Table 4). Education was negatively correlated with the Scam factor and Scam/Legit ratio, but not with Legit factor.

Table 4

Correlation between ASJ Components and Age, Education, and AD8 Scores

	Age	Education	AD8 Total Score
Scam Factor	–0.437 (<0.001)	0.267 (<0.001)	0.592 (<0.001)
Legit Factor	–0.240 (0.001)	0.082 (0.273)	0.416 (<0.001)
Scam/Legit Ratio	–0.435 (<0.001)	0.287 (<0.001)	0.547 (<0.001)
Pearson R (p)

The Scam to Legit ratio increased with increasing AD8 scores (Fig. 2) supporting our hypothesis that with increased cognitive concerns, individuals were more likely to fall for scams and had more difficulty differentiating between scam and legitimate scenarios.

Fig. 2

Relationship of ASJ Scam to Legit Ratio to AD8 Scores. This figure demonstrates the relationship of the ASJ Scam to Legit Ratio to AD8 Scores in a sample of 183 individuals. As AD8 scores increase, there is a corresponding increase in the ASJ Scam to Legit Ratio with a noticeable increase between AD8 scores of 1 and 2 (marked by vertical arrow) that represents the cut-off point for cognitive impairment. This supports the hypothesis that with increased cognitive concerns, individuals were more likely to fall for scams and had more difficulty differentiating between scam and legitimate scenarios.

ASJ scores by sociodemographic characteristics

ASJ Scam and Legit scores and Scam/Legit ratios by sample sociodemographic and lifestyle characteristics, AD8 scores, computer, and social media usage, as well as prior exposure and perceived vulnerability to scams is shown in Table 5. After correction for multiple comparisons, there was no difference in Scam or Legit factor scores between men and women, however women had a slightly higher Scam/Legit ratios (p = 0.013), indicating a modestly greater likelihood to fall for scams. There was no difference in Scam, Legit or ratio scores between non-Hispanic Whites and other racial/ethnic groups, although this should be interpreted with caution given the sample was predominantly non-Hispanic White. Age differences in the test sample replicated findings from the development sample with the Scam and Legit factor scores and the Scam/Legit ratio decreasing with age. There were no differences between Scam or Legit factor scores by education, however, the Scam/Legit ratio increased with higher education (p = 0.004). There were no differences in Scam, Legit or ratio scores based on whether the participant lived alone. Legit factor scores were lower in participants who did not use social media (p < 0.001) and Scam factor scores, and Scam/Legit ratios were higher in those individuals with the highest level of computer use (ps < 0.001). This is consistent with the participants reporting the most effective scamming modality to be ones that occur online. Participants who reported no prior experience with financial scamming had higher Scam factor scores and Scam/Legit ratios than those with a prior experience. Conversely, participants who reported the highest perceived vulnerability to being scammed also had higher Scam factor and Scam/Legit ratios than those with lower perceived vulnerability (ps < 0.001). Finally, individuals with self-reported cognitive impairment as per higher AD8 scores had higher Scam and Legit factor scores and Scam/Legit ratios than participants with lower AD8 scores (ps < 0.001). This was further evaluated by examining participants’ sociodemographic characteristics and item level responses by AD8 categorization with non-impaired individuals scoring 0–1 and impaired individuals scoring 2 or greater (Table 6). Participants with an AD8 score > 2 had significantly higher scores for every Scam factor item than participants with AD8 scores of 0–1. Individuals with AD8 scores > 2 also had significantly higher Legit factor scores except for three items suggesting a greater predilection to engage in most scenarios without differentiation. This was also reflected in the higher Scam to Legit ratio.

Table 5

ASJ Scores by Sociodemographic and Participant Characteristics

	Sex			Race			Living Alone
Domain	Male	Female	p	White	Non-White	p	Yes	No	p
Scam	19.7 (15.9)	25.3 (17.1)	0.029	22.9 (16.6)	26.3 (19.9)	0.525	19.4 (15.4)	24.9 (17.2)	0.040
Legit	44.2 (8.9)	44.8 (8.9)	0.656	44.5 (8.9)	44.7 (9.8)	0.939	42.7 (10.1)	45.4 (8.1)	0.065
Ratio	0.42 (0.28)	0.54 (0.31)	0.013	0.49 (0.30)	0.53 (0.32)	0.713	0.43 (0.26)	0.52 (0.32)	0.047
	Age, y				Education, y
Domain	50–59	60–69	70–79	p	≤12 y	13–16	>16	p
Scam	30.9 (17.2)	21.9 (16.2)	11.5 (9.1)	<0.001	16.7 (14.9)	21.6 (16.3)	28.2 (17.3)	0.013
Legit	47.1 (6.8)	44.1 (9.5)	40.6 (8.8)	0.009	45.7 (10.2)	43.9 (8.8)	45.4 (8.6)	0.509
Ratio	0.63 (0.31)	0.48 (0.29)	0.28 (0.15)	<0.001	0.35 (0.26)	0.47 (0.29)	0.59 (0.31)	0.004
	Social Media			Computer time/day, h
Domain	Yes	No	p	4 or less	5–8	More than 8	p
Scam	23.7 (17.0)	12.7 (7.1)	0.055	13.9 (10.2)	28.0 (17.1)	21.2 (17.7)	<0.001
Legit	45.0 (8.7)	34.6 (6.5)	<0.001	42.2 (7.9)	45.3 (8.9)	44.5 (8.9)	0.128
Ratio	0.50 (0.31)	0.37 (0.19)	0.218	0.33 (0.21)	0.59 (0.31)	0.44 (0.30)	<0.001
	AD8 scores			Prior Experience			Perceived Vulnerability
Domain	0–1	2–8	p	Yes	No	p	Yes	No	p
Scam	15.5 (12.5)	34.6 (15.9)	<0.001	16.6 (14.2)	30.1 (16.7)	<0.001	31.3 (17.6)	18.9 (14.8)	<0.001
Legit	41.7 (8.9)	48.8 (6.8)	<0.001	43.7 (8.6)	45.3 (9.1)	0.245	47.0 (8.0)	43.1 (9.0)	0.005
Ratio	0.36 (0.23)	0.69 (0.29)	<0.001	0.36 (0.25)	0.63 (0.29)	<0.001	0.64 (0.31)	0.42 (0.27)	<0.001

Mean (SD). Bold p-values signify significance after correction for multiple comparisons using Bonferroni correction (corrected p < 0.017).

Table 6

ASJ Scores by Cognitive Status

	AD8 Score 0–1	AD8 Score 2+	p
Age, y	64.1 (5.2)	61.4 (5.0)	<0.001
Sex, % Female	63.1	59.7	0.650
Education, y	15.7 (2.2)	16.2 (2.2)	0.140
Race, % Non-Hispanic White	95.5	90.3	0.164
Live Alone, %	38.7	20.8	0.011
AD8 Score	0.2 (0.4)	4.4 (2.1)	<0.001
Scam Score	15.5 (121.5)	34.6 (15.9)	<0.001
Disaster relief (S1)	2.3 (1.7)	4.5 (1.9)	<0.001
IRS threat (S2)	1.7 (1.5)	4.2 (2.2)	<0.001
Credit card (S3)	2.0 (1.8)	4.3 (2.1)	<0.001
Work from home (S4)	2.2 (1.7)	4.4 (2.3)	<0.001
Warrant (S5)	1.9 (1.7)	4.3 (2.1)	<0.001
Remote access home computer (S6)	1.8 (1.6)	4.3 (2.1)	<0.001
Internet romance (S7)	1.9 (1.6)	4.5 (2.2)	<0.001
Unclaimed money (S8)	1.8 (1.6)	4.2 (2.1)	<0.001
Legit Score	41.7 (8.9)	48.8 (6.8)	<0.001
Uber (L1)	5.0 (1.8)	5.5 (1.4)	0.089
Phone company (L2)	3.8 (2.1)	5.2 (1.5)	<0.001
Academic survey (L3)	4.9 (5.8)	5.8 (1.3)	0.001
Air BnB (L4)	4.1 (1.9)	4.9 (1.5)	0.007
EBay (L5)	5.1 (1.8)	5.5 (1.1)	0.031
Work IT (L6)	4.4 (2.0)	5.3 (1.5)	0.001
Apple store (L7)	5.6 (1.4)	5.9 (0.9)	0.139
Bank fraud (L8)	3.9 (2.2)	5.1 (1.7)	<0.001
Program install (L9)	4.8 (1.8)	5.7 (1.2)	<0.001
Scam/Legit Ratio	0.36 (0.23)	0.69 (0.29)	<0.001

Mean (SD) or %. Bold p-values signify significance after correction for multiple comparisons using Bonferroni correction (corrected p < 0.0.002).

Classification and discrimination of ASJ scores

Finally, to provide cut-off scores for the ASJ that could be used in clinical practice and future research, we conducted a receiver operator characteristics curve analyses with a Youden Index to determine optimal cut-off scores (Table 7). In the absence of any existing Gold Standard for situational judgment, we used the AD8 scores as an indicator of impaired judgment. The Scam factor had an AUC of 0.790 with an optimal cut-off score of 24, the Legit factor had an AUC of 0.730 with an optimal cut-off score of 47, and the Scam/Legit ratio had an AUC of 0.767 with an optimal cut-off score of 0.62. Youden Index scores support that the Scam factor and the Scam/Legit ratio provide evidence of ASJ ability to detect susceptibility to scamming.

Table 7

Receiver Operator Characteristics Curve Analyses for ASJ Cut-Off Scores

Variable	AUC	95% CI	p	Cut-Off	Youden Index
Scam	0.790	0.719–0.861	<0.001	24	0.588
Legit	0.730	0.657–0.804	<0.001	47	0.403
Scam/Legit Ratio	0.767	0.690–0.844	<0.001	0.62	0.579

AUC, Area under the curve; 95% CI, 95% confidence interval.

DISCUSSION

The ASJ is a novel brief assessment that captures the potential vulnerability of older adults to be a victim of financial scamming. The ASJ provides an easy way to assess recognition of scam and legitimate scenarios and to determine the extent to which community-dwelling older adults can differentiate between scamming scenarios and legitimate requests for information in the context of decision making. The ASJ exhibits excellent data quality and psychometric properties while maintaining the brevity and simple format to facilitate its use in clinical practice, clinical research, and non-medical situations. Optimal cut-off scores were provided for use in clinical practice and future research in this area.

We found no differences in situational judgment between men and women, racial and ethnic groups, or due to educational attainment. Perhaps opposite to what might be expected, increasing age was associated with better ASJ scores. Although older adults are often the targets of financial scamming, is likely that the presence of cognitive impairment, rather than the victim’s age itself has more to do with susceptibility. High computer use was associated with worse scores while social media use may offer some protection. The most common platform for financial scamming is online, suggesting that monitored older adults with cognitive impairment may be at the highest risk while using their computers. Conversely, engagement in a social media community could provide “tips” to avoid fraudulent schemes, even if some scenarios may have been legitimate. If an individual was previously scammed, they were more likely to recognize fraudulent schemes, while individuals who perceived themselves to be at risk were more likely to be scammed based on their ASJ response pattern.

By assessing cognitive functioning, collecting demographic data, frequency of computer and social media usage, and information regarding knowledge and perceived vulnerability to scams, the ASJ allowed us to disentangle factors associated with scam susceptibility. Our data support the notion that with age, people may gain wisdom and become more cautious towards scams. This is in line with the complex findings in the literature regarding the association between age and scam susceptibility. While some investigators found that older age is associated with greater fraud [19, 20] and victimization [6, 21], there are also findings that indicate that tech savvy members of Gen Z and middle aged adults are more likely to fall for scams [22 –24], consistent with our findings amongst those aged 50–59 as compared to the 60–69 and 70–79 year old cohorts. The variability in fraud and scamming rates in relation to age may also be due to differences in reporting rates, socioeconomic status, and exposure to media regarding scams [25].

It will be important for future research in scam susceptibility to incorporate brief, sensitive, and specific measures of cognition and dementia, such as the AD8. Our data indicate that higher likelihood of cognitive impairment, as discerned by an AD8 score > 2, was associated with a greater willingness to engage in nearly all the scenarios presented, both Scam and Legit. Future studies should leverage easily administered measures of risk-taking as well as neuropsychological measures of executive function and decision making in conjunction with the ASJ. A recent meta-analysis, Shao, et al. [25] underscored the need for a “comprehensive fraud victimization theory, improving research methods, extending existing research, exploring physiological mechanisms of elderly fraud, and strengthening prevention and intervention efforts.” Tools like the ASJ in concert with cognitive, psychosocial, and emotional measures that can be easily implemented will be important building blocks in the construction of a fraud victimization theory and in designing effective interventions.

In addition to the Scam and Legit factors, we explored the use of a Scam to Legit ratio to detect subtle vulnerability suggesting those with high ratios may have less ability to differentiate between scam and legitimate scenarios. We had hypothesized that lower ratio scores would indicate lower risk of falling for scams while higher ratio scores would indicate both a greater likelihood to fall for a scam as well as less ability to differentiate between scam and legitimate scenarios. Our findings supported that the Scam to Legit ratio could be useful to explore the effectiveness of an intervention to enable an older adult to discriminate between a scam and legitimate scenario. The optimal cut-off scores of 24 for the Scam factor and 0.62 for the Scam to Legit ratio allow easy determination of susceptibility to scamming.

There are several limitations in this study. Though our data did not show differences in Scam and Legit scores between non-Hispanic Whites and other racial/ethnic groups, tools such as the ASJ should continue to focus on cultural sensitivity particularly amongst minority groups, for whom the data suggest greater vulnerability with regards to fraud and scams [26, 27]. Further research in more diverse samples is needed. Given the online methods of data collection, we were limited to using a self-reported measure of cognition, the AD8, to establish the presence of cognitive impairment. While the AD8 has been validated against gold standard measures and biomarkers, future research would benefit from direct comparison of the ASJ against neuropsychological testing, particularly in the areas of executive function and attention. The ASJ is a self-report scale so there may be a bias to report less susceptibility, however the ASJ alternates Scam and Legit scenarios to limit this possibility. As this is cross-sectional study, the longitudinal properties and predictive power of the ASJ still needs to be elucidated. Strengths of this study include incorporation of a cognitive measure to help disentangle deficits due to aging versus cognitive impairment. The brevity of the ASJ and its use online and pen and paper make it easy to use in a variety of clinical and non-clinical situations.

There are several potential implications of our findings. First, the Legit factor was less influenced by participant characteristics suggesting that future interventions could be more focused on increasing recognition of scamming scenarios. Second, the Scam/Legit ratio demonstrated that it could be used as a measure to capture change due to interventions in individuals with high susceptibility to scamming. The goal of developing the ASJ was to create a tool that requires decision making across both benign and deceptive situations to separate people with a high probability of becoming a victim of a financial scam. While some tools have been developed to assess beliefs about deception and scams [5 –7] and to examine choices made for COVID-19 related questions between different age cohorts [8], to date, there are few tools to obtain a quantifiable estimate of the older adult to differentiate between scam and legitimate scenarios. Efficient screening tools in this field require ecological validity, accessibility, reliability, sensitivity, and specificity. Therefore, our objective was to create a sensitive and specific as well as ecologically valid tool that is situationally varied that could be easily implemented in clinical care and clinical research settings across age cohorts to detect vulnerability to fraud and scams. Such a sensitive and specific tool could also be of great value to caregivers determining need for financial conservatory, the financial services industry, and insurance companies.

Footnotes

ACKNOWLEDGMENTS

The authors have no acknowledgments to report.

FUNDING

This study was supported by grants from the Alzheimer’s Association, and the Evelyn F. McKnight Brain Foundation through the American Brain Foundation and American Academy of Neurology to SJG, from Florida Department of Health and McKnight Brain Research Institute to BEL, and from the National Institute on Aging (R01 AG071514) to JEG. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

CONFLICT OF INTEREST

SJG and BEL are co-creators of the Assessment for Situational Judgment.

JEG is an Editorial Board Member of this journal but was not involved in the peer-review process nor had access to any information regarding its peer-review.

The authors take full responsibility for the data, the analyses and interpretation, and the conduct of the research; have full access to all of the data; and have the right to publish all data.

DATA AVAILABILITY

The dataset for this project is available to all interested parties. Please contact JEG at jeg200@miami.edu.

References

Amstadter

, Zajac

, Strachan

, Hernandez

, Kilpatrick

, Acierno

(2011) Prevalence and correlates of elder mistreatment in South Carolina: The South Carolina elder mistreatment study. J Interpers Violence 26, 2947–2972.

Roberto

, Teaster

(2011) The Met Life Study of Elder Financial Abuse: Crimes of occasion, desperation, and predation against America’s elders. Met Life Mature Market Institute, New York.

Federal Bureau of Investigation. Do Yourself a Favor: Be Crime Smart. https://www.fbi.gov/how-we-can-help-you/safety-resources/scams-and-safety

Orcutt

(2021) State of Internet Scams 2021, Social Catfish. https://socialcatfish.com/scamfish/state-of-internet-scams-2021/

Han

, Boyle

, James

, Yu

, Bennett

(2016) Mild cognitive impairment and susceptibility to scams in old age. J Alzheimers Dis 49, 845–851.

James

, Boyle

, Bennett

(2014) Correlates of susceptibility to scams in older adults without dementia. J Elder Abuse Negl 26, 107–122.

Boyle

, Yu

, Wilson

, Gamble

, Buchman

, Bennett

(2012) Poor decision making is a consequence of cognitive decline among older persons without Alzheimer’s disease or mild cognitive impairment. PloS One 7, e43647.

Nolte

, Hanoch

, Wood

, Hengerer

(2021) Susceptibility to COVID-19 scams: The roles of age, individual difference measures, and scam-related perceptions. Front Psychol 12, 789883.

Galvin

, Roe

, Powlishta

, Coats

, Muich

, Grant

, Miller

, Storandt

, Morris

(2005) The AD8: A brief informant interview to detect dementia. Neurology 65, 559–564.

10.

Galvin

, Roe

, Xiong

, Morris

(2006) Validity and reliability of the AD8 informant interview in dementia. Neurology 67, 1942–1948.

11.

Galvin

, Roe

, Coats

, Morris

(2007) Patient’s rating of cognitive ability: Using the AD8, a brief informant interview, as a self-rating tool to detect dementia. Arch Neurol 64, 725–730.

12.

Buhrmester

, Kwang

, Gosling

(2016) Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspect Psychol Sci 6, 3–5.

13.

Moss

, Rosenzweig

, Robinson

, Litman

(2020) Demographic stability on Mechanical Turk despite COVID-19. Trends Cogn Sci 24, 678–680.

14.

Stewart

, Ungemach

, Harris

, Bartels

, Newell

, Paolacci

, Chandler

(2015) The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers. Judgm Decis Mak 10, 479–491.

15.

Bentley

, Peesapati

, Church

(2016) “I thought she would like to read it” Exploring Sharing Behaviors in the Context of Declining Mobile Web Use. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pp. 1893–1903.

16.

Bartholomew

(1984) The foundations of factor analysis. Biometrika 71, 221–232.

17.

SPSS IBM SPSS Statistics for Macintosh, Version 25.0.

18.

Colman

(2015) A dictionary of psychology, Oxford quick reference.

19.

Huang

, Lawitz

(2016) The New York State cost of financial exploitation study, Office of Children and Family Services.

20.

Burnes

, Henderson

Jr , Sheppard

, Zhao

, Pillemer

, Lachs

(2017) Prevalence of financial fraud and scams among older adults in the United States: A systematic review and meta-analysis. Am J Public Health 107, e13–e21.

21.

Anderson

(2003) Off the hook: Reducing participation in telemarketing fraud. AARP Foundation, Washington, DC.

22.

Anderson

(2013) Consumer fraud in the United States, 2011: The third FTC survey. Federal Trade Commission, Washington, DC.

23.

Flatley

(2016) Overview of fraud statistics: Year ending Mar 2016. Office of National Statistics.

24.

Mueller

, Wood

, Hanoch

, Huang

, Reed

(2020) Older and wiser: Age differences in susceptibility to investment fraud: The protective role of emotional intelligence. J Elder Abuse Negl 32, 152–172.

25.

Shao

, Zhang

, Ren

, Li

, Lin

(2019) Why are older adults victims of fraud? Current knowledge and prospects regarding older adults’ vulnerability to fraud. J Elder Abuse Negl 31, 225–243.

26.

Lichtenberg

, Ficker

, Rahman-Filipiak

(2016) Financial decision-making abilities and financial exploitation in older African Americans: Preliminary validity evidence for the Lichtenberg Financial Decision Rating Scale (LFDRS). J Elder Abuse Neglect 28, 14–33.

27.

Beach

, Schulz

, Castle

, Rosen

(2010) Financial exploitation and psychological mistreatment among older adults: Differences between African Americans and non-African Americans in a population-based survey. Gerontologist 50, 744–757.