The (Im)perfect Automation Schema: Who Is Trusted More,Automated or Human Decision Support?

Abstract

Objective

This study’s purpose was to better understand the dynamics of trust attitude and behavior in human-agent interaction.

Background

Whereas past research provided evidence for a perfect automation schema, more recent research has provided contradictory evidence.

Method

To disentangle these conflicting findings, we conducted an online experiment using a simulated medical X-ray task. We manipulated the framing of support agents (i.e., artificial intelligence (AI) versus expert versus novice) between-subjects and failure experience (i.e., perfect support, imperfect support, back-to-perfect support) within subjects. Trust attitude and behavior as well as perceived reliability served as dependent variables.

Results

Trust attitude and perceived reliability were higher for the human expert than for the AI than for the human novice. Moreover, the results showed the typical pattern of trust formation, dissolution, and restoration for trust attitude and behavior as well as perceived reliability. Forgiveness after failure experience did not differ between agents.

Conclusion

The results strongly imply the existence of an imperfect automation schema. This illustrates the need to consider agent expertise for human-agent interaction.

Application

When replacing human experts with AI as support agents, the challenge of lower trust attitude towards the novel agent might arise.

Keywords

human-automation interaction trust in automation expert systems expert-novice differences decision making

INTRODUCTION

Novel decision support systems based on artificial intelligence (AI) are making inroads to a plethora of application domains. When introducing such systems into any given work environment, the main goal is to improve overall safety and performance (e.g., Mosier & Manzey, 2020). As such, these systems become more and more sophisticated, and can sometimes even outperform human experts (e.g., Bejnordi et al., 2017; McKinney et al., 2020). Given the wide range of application domains of AI support (Islam et al., 2022), it is clear that AI support will in some cases replace support that was previously given by a human. Thus, it is important to understand human–AI trust in order to foster successful interaction dyads with respect to joint performance. Specifically, one important issue is the potential difference in trust towards AI compared to support by other humans.

Prior research has investigated to which degree trust attitude (Lee & See, 2004) differs towards automated support compared to human support (Madhavan & Wiegmann, 2007). One key finding from this earlier research is the so-called perfect automation schema, as brought forward by Dzindolet et al. (2002). The perfect automation schema proposes that without failure experience, humans have higher trust towards automated support compared to support by other humans. Moreover, Dzindolet et al. (2002) also suggest that after experiencing a failure by the respective support agent, trust decreases more for automation support than for human support (illustrated in Figure 1(a)). They argue that this difference is due to expectations of near-perfect performance when working together with automated systems, but that there is no such expectation when working together with another human due to the awareness of human fallibility.

Figure 1.

Key predictions from the perfect automation schema (a) and from recent results in line with an imperfect automation schema (b).

However, recent findings challenge this often referred-to effect. Specifically, in numerous experiments with relatively large sample sizes, Rieger et al. (2022), Roesler et al. (2022), and Appelganc et al. (2022) have consistently shown higher trust attitude towards humans than towards automated support systems, irrespective of failure experience. For instance, Rieger et al. (2022) conducted three experiments in different contexts (i.e., loan assignment, X-Ray evaluation, chemical plant) with the consistent finding of higher trust towards human support agents than towards decision support systems or AI. Moreover, in these studies, trust was reduced after failure experience, in line with theoretical frameworks of human-automation interaction (Hoff & Bashir, 2015). Crucially, though, this reduction in trust did not differ between automated and human support agents (Rieger et al., 2022). It seemed like participants were not more forgiving towards humans than towards technical support agents after failure experience in these studies, in contrast to the expectations from the perfect automation schema. Given these findings, Rieger et al. (2022) coined this pattern of results as an imperfect automation schema (illustrated in Figure 1(b)), given the fact that trust was consistently higher for human support agents.

To get a better idea about which subdimensions of trust are relevant for the above-mentioned differences, Roesler et al. (2022) further looked into the trust difference between humans and automation, using a multidimensional trust questionnaire. They found that the overall trust difference was mainly related to the performance dimension, that is, that participants perceived human support to be more reliable than automated support. Moreover, Appelganc et al. (2022) ruled out the possible explanation of differences in expectations towards what one perceives as “reliable” for the respective support agents. These combined findings by Rieger et al. (2022), Roesler et al. (2022), and Appelganc et al. (2022) therefore raise the question of why there was such a difference to the perfect automation schema as proposed by Dzindolet et al. (2002).

When taking a closer look at the experiments providing evidence for the perfect versus imperfect automation schema, it becomes clear that the framing of the human support notably differed between these two lines of research. Specifically, whereas Dzindolet et al. (2002) framed the human support to be the former participant (i.e., a novice), Rieger et al. (2022), Roesler et al. (2022), and Appelganc et al. (2022) framed the human support as an expert in the task (i.e., to be more comparable to the technical support conditions). Of course, in any professional setting, it seems more likely that a human support agent is an expert in the task. Although this rationale might have made sense to design the earlier studies by Rieger et al. (2022), it is of course also an inconsistency with the research by Dzindolet et al. (2002). Thus, the main goal of the present study was to investigate whether the discrepancy between the perfect and the imperfect automation schema comes from the framed expertise of the human support agent. Moreover, the inconsistent findings are not only theoretically puzzling but of high practical relevance, as the dynamics of trust attitude and especially behavior are crucial for practitioners seeking to implement AI agents effectively. If human expertise framing is the decisive point for the (im)perfect automation schema, practitioners can utilize this insight to make informed decisions on how to deploy AI support systems, ensuring that AI agents are appropriately introduced to the interaction dyad. So in the possible absence of the postulated advantage of trust in technologies (Dzindolet et al., 2002), practitioners might even need to implement trust-building measures when introducing AI technologies to jobs previously performed by another human.

In earlier research, the focus was mainly on questionnaire-based outcomes such as trust attitude. Therefore, another goal was to also look at trust behavior. However, given the proposed close link between trust attitude and behavior (Hoff & Bashir, 2015; Lee & See, 2004), one would expect the results to be extendable to a more behavioral outcome.

The Present Experiment

We conducted an experiment using the medical X-ray screening context that was also used in earlier research (Appelganc et al., 2022; Rieger et al., 2022; Roesler et al., 2022). We chose this task because novel AI systems can already compete with the performance of highly specialized human experts in this domain (Bejnordi et al., 2017; McKinney et al., 2020). Three groups of participants carried out the simulated X-ray task, with the support agent experimentally manipulated between the three groups to be framed as either an AI, a human expert, or as a human novice. The AI was framed to be a capable system based on deep learning, the human expert was framed to be a medical expert from a clinic, and the novice was framed to be a former participant (in line with Dzindolet et al., 2002). Moreover, we varied failure experience within-subjects to investigate whether the forgiveness towards different agents differed. Specifically, to understand the dynamics of trust, in the first block, participants always received correct recommendations. In the second block, the agents’ reliability dropped and the agent gave several imperfect recommendations. Then, in the third and final block, no more failures occurred and only correct recommendations were given by the agents. We used this design to be able to study trust prior to failure experience (i.e., trust formation), after failure experience (i.e., trust dissolution), and after having experienced a perfect agent again (i.e., trust restoration). Overall, this design allowed us to investigate overall effects of support agent on trust as well as potential differences in failure-induced trust changes between the agents. In order to consider both trust attitude and behavior, we assessed trust not only via subjective ratings but also by a behavioral measure (i.e., how strongly participants followed the support agents’ recommendations).

Our hypotheses were based on earlier research (Appelganc et al., 2022; Rieger et al., 2022; Roesler et al., 2022) as well as on research on the perfect automation schema (Dzindolet et al., 2002). Hypotheses were preregistered via the Open Science Framework (OSF, https://osf.io/z73j2). We had the same hypotheses for trust attitude and trust behavior, as trust attitude is commonly assumed to go hand in hand with behavior (Hoff & Bashir, 2015; Lee & See, 2004). Thus, we will introduce them jointly here. We hypothesized that trust would be higher for the human expert than for the AI than for the human novice support. Moreover, we hypothesized that trust would be higher for the initial, perfect block than for the subsequent block with failure experience, and that trust in the final, back-to-perfect block would be higher than the imperfect, but lower than in the initial perfect block. Further, we expected an interaction effect of agent and block, based on the idea that humans are more forgiving towards other humans (Dzindolet et al., 2002; Madhavan & Wiegmann, 2007). Specifically, we expected a stronger decrease after failure experience in the AI compared to the human expert condition, with only a slight decrease in the novice condition (as expectations are likely low in this condition anyway). Moreover, we expected a stronger increase in trust after the final, back-to-perfect block in the human conditions (i.e., expert and novice).

We also included perceived reliability as an additional dependent variable, with similar hypotheses (please refer to the OSF). Perceived reliability was mainly included to check whether trust attitude and behavior is also related to perceived reliability, as suggested by Roesler et al. (2022). In sum, the goal of the present study was to investigate whether the (im)perfect automation schema depends on the level of expertise of the human support in the task.

METHOD

This research complied with the tenets of the Declaration of Helsinki, and the experiment was approved by the local ethics committee at the Department of Psychology, Technische Universität Berlin, Germany. Informed consent was obtained from each participant. The experiment was preregistered (https://osf.io/z73j2), and the data and scripts are available under https://osf.io/75s2z/.

Participants

Overall, 110 participants (79 female, 28 male, 3 nonbinary) took part in the experiment and were considered for further analyses. Participants were recruited via the participant portal of the Department of Psychology at TU Berlin and online postings. They ranged in age from 18 to 40 (M = 26.16, SD = 4.36) and reported normal or corrected-to-normal vision as well as fluency in English. Target sample size of 108 was preregistered based on a power analysis to detect a small to medium effect size of .15 at the standard .05 alpha error probability for the within-between interaction effect of the two experimental factors.

In addition, 31 participants were also tested but excluded from further analyses. Out of those, 14 participants were excluded because they failed one or more attention checks. Moreover, 15 participants were excluded because they reported very good knowledge about AI, as this might have possibly undermined the believability of the experimental task. Finally, two participants were excluded because they provided the same initial rating in the task 20 times or more throughout the experiment.

Apparatus and Stimuli

The experiment ran on a JATOS (Lange et al., 2015) server and was built using jspsych (de Leeuw, 2015). Participants could individually run the experiment in their browser. The images used in the X-ray task contained simulated 1/f ³ noise, which resembles the power spectrum of mammograms (Burgess et al., 2001) and was used already in earlier research (e.g., Rieger et al., 2022; Rieger & Manzey, 2022; Sha et al., 2018).

Procedure

Participants gave their informed consent, and they were also briefed about the aim of the study. Subsequently, the experiment started with a general introduction to radiology and the task participants were about to perform. The task was the same task as in Rieger et al. (2022, Experiment 2). Specifically, the task was to evaluate which percentage of simulated X-rays were brighter than a given cutoff (brighter than grayscale value of 150). To allow participants to perform this task, they were first shown a grayscale continuum, with the critical cutoff threshold marked, along with an example image. Participants were instructed that parts brighter than the cutoff could potentially be malignant and that their task would be to estimate the percentage of potentially malignant tissue in X-ray samples. They were also instructed that when the percentage is lower than 15%, there is no reason for concern at all. Further, participants were instructed that in radiology, X-ray images are analyzed in dyadic teams, and that collaborative decision making has been found to improve the quality of health care assessments.

After the introduction to the task, participants were told that during the experiment, they would be supported by either an AI, an expert colleague working at a hypothetical clinic, or by the decisions of a former participant. In all groups, participants were again told that they would work in a dyadic team to improve their decision making. In the AI group, participants were instructed that they would be supported by an AI, which was described as follows:

The system is an innovative artificial intelligence based on deep neural networks. It continuously analyzes previous decisions and outcomes. The intelligent system decides by using learned parameters to increase the probability of making the correct assessment. The combination of technologies like machine learning, data analysis, and natural language processing constantly improves the artificial intelligences’ assessments.

In the expert group, participants were instructed that they would be supported by an expert from a hypothetical clinic and the expert was described as follows:

Your colleague has been working as a radiologist for many years, and has gained considerable experience. The colleague has worked on many previous cases and knows which areas to look at and rate the relevance of anomalies in a differentiated manner to increase the probability of making the correct assessment. Your colleagues’ decisions are based on knowledge, effort and experience.

In the novice group, participants were instructed that they would be supported by the decisions of a former participant and this was further described as follows:

The participants had to make their decisions alone without the help of another participant. The participant from which the decisions are presented to you, has no specific experience in radiology assessment except for participating in the study. The decisions by the participant are influenced by the given instructions about health care evaluations of X-Rays and the given information of the presented case. The same information has been and will be presented to you.

Subsequently, participants were shown four examples (two correct assessments and two incorrect assessments). This was done to illustrate the importance of both correctly estimating low percentages (to avoid unnecessary treatment) as well as high percentages (to avoid missing cancer). Then, participants were again reminded about the type of support they would later on receive. After this reminder, the first attention check was included, asking participants about the type of support agent they would be supported by. Afterward, they were again reminded about the task and the support agent and were told that they would be working on a total of 60 cases, with some questions asked in between.

Following the extensive introduction and framing, the experiment consisting of three blocks with 20 trials each started. Blocks 1 and 3 were with perfect support from the support agent, while in the second block, participants received six erroneous recommendations from their support agent. Three of the six erroneous recommendations were underestimations and three were overestimations of the true value. The wrong recommendations always deviated 15%–25% from the true value (M = 20.42, SD = 3.72). The respective 20 trials per block were counterbalanced across participants.

On each trial, participants were first shown a persona along with an X-ray image. For the personas, we used AI-generated images from Karras et al. (2020). Participants were then asked to fill in their manual assessment (i.e., without the aid from the support agent). The minimum time to fill this out was set to 1.5 s. After entering their own assessment, participants were shown the recommendation of their respective support agent. Recommendations were color-coded (i.e., green: ≤15% malignant, orange: >15% ≤ 50% malignant, red: >50% malignant) and shown below the own initial assessment (see Figure 2). After seeing the advice, participants could then enter their final assessment after 1.5 s, which could be the same as before or adjusted in response to the recommendation of the agent.

Figure 2.

Exemplary trial from the AI condition.

After each block of 20 trials, participants were asked about their trust in their respective support agent and their perceived reliability of their support agent. These questions were embedded within two distractor items and one attention check item. Note that after the first block, the initial attention check (i.e., which support agent), was also repeated within this block of questions.

After completing the questions subsequent to the third block, participants were debriefed (i.e., that no real AI/expert/novice supported them, that no medical information was shown, and that all cases were made up) and thanked for their participation.

Design and Dependent Variables

Support agent (i.e., AI vs. expert vs. novice) was varied between-subjects, and failure experience (i.e., perfect support, imperfect support, back-to-perfect support) was varied within subjects, resulting in a 3 × 3 mixed design. Subjective dependent variables were trust attitude (single item, from 0 “Not at all” to 100 “Completely”) and perceived reliability (single item, from 0% to 100%). The single item was preferred against a more complex multi-item assessment as earlier research (Rieger et al., 2022) showed a high correlation between the single-item trust and one of the most widely used multi-item questionnaires in the field (Jian et al., 2000). For trust behavior, we calculated the absolute difference between the own initial assessment and the final assessment after seeing the recommendation. If, after seeing the recommendation, participants changed their assessment away from the recommendation (e.g., initial assessment 20, recommendation 29, final decision 15), we used negative scores (in this example, final trust behavior value: −5). If participants over-corrected their initial assessment (e.g., initial assessment 20, recommendation 29, final decision 35), we subtracted the over-correction from the adaptation (in this example, final trust behavior value: 9–6 = 3). Thus, higher values represent higher behavioral trust. Trials in which participants’ initial assessments equaled the upcoming recommendation were excluded from the behavioral trust analysis.

RESULTS

We ran separate 3 (support agent) x 3 (block) mixed-ANOVAs for our dependent variables. The corresponding results are visualized in Figure 3(a)–(c). Whenever the assumption of sphericity was violated for the within-subjects factor, we report Greenhouse–Geisser corrected results. For post hoc pairwise comparisons, we used the Bonferroni–Holm correction as preregistered, and the p-values were adjusted accordingly.

Figure 3.

Means and standard errors for (a) trust attitude, (b) perceived reliability, (c) trust behavior as an adjustment to the recommendation, and (d) trust behavior as full agreement with the agent, as a function of support agent (human expert versus articial intelligence (AI) versus human novice) and block (i.e., block 1: perfect support, block 2: imperfect support, block 3: back-to-perfect support).

Trust Attitude

For trust attitude, there was a main effect of support agent, $F (2,107) = 11.803, p < 0.001, η_{p}^{2} = 0.181$ , with higher trust towards the human expert (69.1) compared to the AI (59.9, p = .041) and compared to the human novice (47.1, p < .001). The difference between the AI and the novice condition was also significant (p < .001). Moreover, there was also a significant main effect of block, $F (1.88,201.08) = 23.014, p < 0.001, η_{p}^{2} = 0.177$ , with higher trust after the first, perfect block (63.2) than after the second, imperfect block (54.3, p < .001) than after the third, back-to-perfect block (58.5, p = .002). The difference between the second and third block was significant, as well (p < .001), indicating a sort of trust restoration. The interaction was not significant, $F (3.76,201.08) = 0.478, p = 0.740, η_{p}^{2} = 0.009$ .

Perceived Reliability

The pattern of effects for perceived reliability directly corresponded to the effects of subjective trust. The main effect of support agent was again significant, $F (2,107) = 14.100, p < 0.001, η_{p}^{2} = 0.209$ . Again, all three pairwise comparisons were significant (p’s < .040), with higher perceived reliability for the human expert (69.2%) than for the AI (60.1%) than for the human novice (45.5%) Moreover, the main effect of block was also again significant, $F (1.78,189.97) = 21.903, p < 0.001, η_{p}^{2} = 0.170$ . Perceived reliability was highest after the first, perfect block (62.3%) and significantly higher than after the second, imperfect block (54.3%, p < .001) and higher than after the third, back-to-perfect block (58.2%, p < .001). The difference between the second and third block was also significant (p < .001). The interaction of support agent and block was again not significant, $F (3.55,189.97) = 1.472, p = 0.217, η_{p}^{2} = 0.027$ . Notably, the perceived reliability was significantly lower than the true overall (90%) reliability, t (109) = 15.428, p < .001.

Trust Behavior

We ran a parallel analysis for trust behavior. In this analysis, there was again a significant main effect of support agent, $F (2,107) = 4.697, p = 0.011, η_{p}^{2} = 0.081$ . Participants in the human expert (6.90) and AI (6.95) conditions did not differ in the way they corrected their own judgment towards the support agent’s advice (p = .950). However, the difference between human expert and human novice (4.72, p = .023) as well as between the AI and the human novice (p = .023) were significant. In addition, the main effect of block was significant, $F (1.83,195.80) = 10.253, p < 0.001, η_{p}^{2} = 0.087$ . Here, there was no difference between the first two blocks (block 1: 6.62, block 2: 6.45, p = .563) but trust behavior in the third block was significantly lower (5.49, p’s < .001). The interaction for trust behavior was not significant, $F (3.66,195.80) = 0.681, p = 0.593, η_{p}^{2} = 0.013$ . The decline in trust behavior over the experimental blocks seems to be in contrast to trust attitude; however, this might be related to practice effects in manual performance. To crosscheck this, we ran an additional analysis for the accuracy of the initial assessments. That is, if participants became better in their initial assessments, there might have been less room for them to show trust behavior in the final, perfect support block. This ANOVA indeed revealed a significant learning effect, $F (1.73,185.34) = 14.478, p < 0.001, η_{p}^{2} = 0.119$ , with significant improvements in assessing the X-rays from the first to the second and third blocks. Thus, the main effect of block for trust behavior needs to be taken with a grain of salt.

We included an additional behavioral trust measure (Figure 3(d)), which might be less prone to be affected by learning effects to further investigate and confirm the prior results. Specifically, we checked the percentage of trials where participants’ final decision equaled exactly the recommendation of the support agent, as this indicates full dependence on the recommendation. For this analysis, trials in which the initial assessment of the participant already perfectly matched the agent’s recommendation were excluded (3.30%). A perfect match between the participants’ final rating and the support agents’ recommendation was found in 12.46% of all remaining trials. However, these matches were not equally distributed across the different conditions as indicated by a main effect of support agent, $F (2,107) = 4.858, p = 0.010, η_{p}^{2} = 0.083 .$ . The percentage of perfect agreement was highest for the human expert (15.38%) followed by the AI (13.15%) and lowest for the human novice (3.73%). Both the agreement with the AI and human expert were significantly higher than the agreement with the human novice (p’s < .040), with no difference between the AI and human expert condition (p = .562). The main effect of block was also significant, $F (1.81,194.02) = 3.883, p = 0.026, η_{p}^{2} = 0.035$ . Here, the first, perfect (11.5%), and the final, back-to-perfect (11.6%) block did not differ (p = .977). In both of these blocks, perfect agreement rates were significantly higher (p_1vs.2 = .027, p_2vs.3 = .021) than in the second, imperfect (9.2%) block. Again, no significant interaction emerged, $F (3.63,194.02) = 1.039, p = 0.384, η_{p}^{2} = 0.019$ .

DISCUSSION

The introduction of AI support systems into various domains has the potential to improve safety and performance. Therefore, AI support is in some cases replacing human support. As trust towards a support agent is crucial for successful human-agent interaction, the main goal of the present study was to disentangle the inconsistencies in the literature concerning the findings supporting a perfect automation schema (Dzindolet et al., 2002) and findings supporting an imperfect automation schema (Rieger et al., 2022). To this end, in a simulated X-ray task, we experimentally varied the support agent to be either an AI, a human expert, or a human novice. At the same time, we also varied failure experience within-subjects (i.e., perfect recommendations in the first block, imperfect recommendations in the second block, and back-to-perfect recommendations in the third block).

The first key finding is that the degree of expertise of human support agents determines attitudes towards and perceptions of human support. In particular, trust was higher for human expert support than for human novice support. More crucially, trust in the AI condition was in between these two human conditions. For human experts, this is in line with findings supporting an imperfect automation schema in different task contexts (Rieger et al., 2022). For human novices, this is in line with findings supporting a perfect automation schema (Dzindolet et al., 2002). Consequently, human expertise seems to be a defining boundary condition for the existence of an (im)perfect automation schema. Given the fact that in most real-world applications, human experts are usually supported by other experts (e.g. two radiologists examining the same X-ray), the AI versus expert comparison seems to be the more relevant one. However, the strength of the (im)perfect automation schema may vary based on different influencing factors, such as task complexity, required user expertise, system reliability, and contextual variables.

The results for perceived reliability also match the trust attitude findings. This is in line with the findings by Roesler et al. (2022), who also reported higher trust towards human experts than towards AI on the performance subscale in their multidimensional trust questionnaire. Despite the consistency across subjective measures, there were no differences in trust behavior between the AI and expert conditions. Even though trust attitude has often been postulated to guide behavior (Hoff & Bashir, 2015; Lee & See, 2004), the present findings only partially support this link. Specifically, lower trust attitude only went hand-in-hand with lower trust behavior for the comparisons involving the novice condition. Perhaps, the lack of differences in trust behavior between the AI and expert conditions might be due to the fact that trust attitude and behavior are not perfectly correlated (e.g., Chavaillaz et al., 2019) and both attitude and behavior have other factors impacting the respective component (e.g., Hoesterey & Onnasch, 2022; Rice & Keller, 2009; Wiczorek & Meyer, 2019). This disconnection between trust attitude and behavior might be rooted in the performance-based components rather than attribute-based components of the framing, as both the AI and the human expert agents were framed as having expertise in the task (Langer et al., 2021; Rieger et al., 2022). However, as one reviewer suggested we also want to acknowledge the possibility of both Type I and Type II errors in the current results. Both types of errors may have contributed to the lack of association between trust and behavior, which should be therefore investigated more closely in future research.

The second key finding was that there were no differences in trust restoration between the support agents. We observed the typical pattern of trust formation, dissolution, and restoration (Lewis et al., 2018) for trust attitude and behavior as well as perceived reliability. However, this pattern did not differ between the different support agents. This result contributes to a growing body of research (Rieger et al., 2022; Roesler et al., 2022; Roesler & Onnasch, 2020) indicating that there are no differences in forgiveness towards different agents. The fact that the impact of failure experience did not differ between the support agents is again in contrast to the perfect automation schema. The perfect automation schema (Dzindolet et al., 2002) and other earlier research (Madhavan & Wiegmann, 2007) have argued that humans would be more forgiving towards mistakes by human support agents due to the awareness of human fallibility. Even though this assumption might be somewhat reasonable, we did not obtain any evidence in line with this assumption—neither for human experts nor for human novices. This finding may be explained by the idea that agent performance overshadows other features of the agent in this formation-dissolution-restoration process.

When integrating these two key findings, it seems like the predictions of the perfect automation schema (Dzindolet et al., 2002) are limited to situations prior to failure experience where an automated system is compared to a novice human. Moreover, we argue that having a human expert as a support agent is much more realistic in most settings—particularly in domains where AI is recently being introduced (e.g., banking (Bahrammirzaee, 2010; O’Neil, 2020), medicine (Bejnordi et al., 2017; McKinney et al., 2020), or process industry (Mao et al., 2019)). Thus, the imperfect automation schema seems to be more appropriate and makes the following predictions. First, the imperfect automation schema predicts lower trust (attitude and behavior) for technical systems such as AI than for human experts. Second, it predicts a decrease in trust after failure experience, with no differences in forgiveness towards the respective support agent. Third, it predicts trust restoration after receiving perfect support following agent failures, again irrespective of the support agent framing. In sum, given the apparent under-trust towards just-as-capable AI, the predictions of the imperfect automation schema challenge the approach of focusing on automation’s weaknesses instead of its high reliability to achieve adequate trust. Moreover, the substantial gap between perceived reliability (58.81%) and truly experienced reliability (90%) gives further reason to focus not on the weaknesses but on the strengths of a support agent. As this is a common issue in the interaction with highly reliable support (Parasuraman & Riley, 1997; Rieger et al., 2022), there is a need for research that explores ways to counteract the under-utilization of support agents. Moreover, the degree of reliability underestimation seems to be context-sensitive. Whereas earlier research (Rieger et al., 2022) already illustrated the context sensitivity of perceived reliability, the underlying reasons for this are still largely unexplored. One influencing factor seems to be the task difficulty, as perceived reliability is especially lower for tasks that are perceived more difficult by humans (e.g., the radiology task used in this study) compared to easier ones (e.g., loan assignment) (Appelganc et al., 2022). In terms of future research, it would be useful to extend the current findings by examining the influencing factors for the strength of the actual-perceived reliability gap.

Another aspect that should be addressed in future research regards the possible impact of the sequence of assessments on the finding of a perfect or imperfect automation schema. In the recent study, the participants’ assessments were always made before the recommendations of the support agents were presented. This marks another difference to the study of Dzindolet et al., 2002, where the sequence was reversed. Making own assessments prior to or after receiving recommendations from a support agent might have different impacts on information processing. While we are uncertain about how this difference could explain the disparities between our expert condition and the previous findings of Dzindolet et al., 2002, we cannot completely rule out the possibility that variations in the assessment sequence might have played a role in contributing to these differences in some manner. Therefore, further research is necessary to better understand the potential impact of the assessment sequence.

Limitations

The present study of course does not come without limitations. First, the experiment was conducted online and therefore has the typical limitations associated with conducting research online. However, we used strict, preregistered exclusion criteria to ensure data quality, and the effect sizes were relatively large. Second, it is of course impossible to know to which degree participants actually believed the support agent manipulation, a limitation that is shared, however, with a lot of earlier research in this field (Dijkstra, 1999; Dijkstra et al., 1998; Dzindolet et al., 2002, 2003; Madhavan & Wiegmann, 2007). Third, this study only included a first-party perspective (i.e., the perspective of the advice-taker, Langer & Landers, 2021), and earlier research (Rieger et al., 2022) has shown that changing to the second party perspective (i.e., the perspective of being assessed) might make a difference. However, it seems unrealistic to have one’s own X-ray case to be evaluated by a novice, and we, therefore, limited the present research to the first-party perspective.

CONCLUSION

In sum, the present study corroborates earlier findings (e.g., Rieger et al., 2022) that showed higher trust attitude towards human expert support agents than towards AI support agents. Failure experience reduced trust with a trust repair after subsequent failure-free interaction but with no differences in failure effects across the different support agents. Thus, the imperfect automation schema seems more appropriate to describe differences in trust towards human or technical support agents. As AI will ubiquitously be a changing force in the way we work together, human factors research will need to keep up to also consider AI-specific trust-related aspects such as explainability in order to foster human-technology symbiosis.

Author’s Note

The experiment was preregistered via the open science framework (OSF) at https://osf.io/z73j2. The data is obtainable via the OSF at https://osf.io/75s2z/.

KEY POINTS

Trust attitude towards and perceived reliability of the support agents were higher for human experts than for AI than for human novices.

Failure experience reduces trust, which can be restored after failure-free interaction, but not differently for the different support agents.

The findings are in line with an imperfect automation schema; however, the expertise of the human support seems to be the boundary condition for the (im)perfect automation schema.

By acknowledging the existence of an imperfect automation schema, practitioners can address potential challenges of lower trust towards AI agents and implement strategies to bridge the gap between human and AI support agents.

Supplemental Material

Supplemental Material - The (Im)perfect Automation Schema: Who is Trusted More, Automated or Human Decision Support?

Supplemental Material for The (Im)perfect Automation Schema: Who Is Trusted More, Automated or Human Decision Support? by Tobias Rieger, Luisa Kugler, Dietrich Manzey and Eileen Roesler in Human Factors

Footnotes

ORCID iDs

Tobias Rieger

Eileen Roesler

Supplemental Material

Supplemental material for this article is available online.

Tobias Rieger is a researcher and lecturer at the Department of Psychology and Ergonomics, TU Berlin, Germany. He earned a PhD in psychology at TU Berlin.

Luisa Kugler is a student at Humboldt-Universität zu Berlin. She earned an MSc in human factors at TU Berlin.

Dietrich Manzey is an emeritus university professor of work, engineering and organizational psychology in the Department of Psychology and Ergonomics, TU Berlin, Germany. He earned his PhD in experimental psychology at the University of Kiel, Germany, in 1988 and his habilitation in psychology at the University of Marburg in 1999.

Eileen Roesler is an assistant professor of psychology at George Mason University. She earned PhD in psychology at TU Berlin.

References

Appelganc

Rieger

Roesler

Manzey

(2022). How much reliability is enough? A context-specific view on human interaction with (artificial) agents from different perspectives. Journal of Cognitive Engineering and Decision Making, 16(4), 207–221. https://doi.org/10.1177/15553434221104615

Bahrammirzaee

(2010). A comparative survey of artificial intelligence applications in finance: Artificial neural networks, expert system and hybrid intelligent systems. Neural Computing and Applications, 19(8), 1165–1195. https://doi.org/10.1007/s00521-010-0362-z

Bejnordi

B. E.

Veta

Johannes van Diest

van Ginneken

Karssemeijer

Litjens

van der Laak

J. A. W. M.

Hermsen

Manson

Q. F.

Balkenhol

Geessink

Stathonikos

van Dijk

M. C.

Bult

Beca

Beck

A. H.

Wang

Khosla

Gargeya

Venâncio

(2017). Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA, 318(22), 2199–2210. https://doi.org/10.1001/jama.2017.14585

Burgess

A. E.

Jacobson

F. L.

Judy

P. F.

(2001). Human observer detection experiments with mammograms and power-law noise. Medical Physics, 28(4), 419–437. https://doi.org/10.1118/1.1355308

Chavaillaz

Schwaninger

Michel

Sauer

(2019). Expertise, automation and trust in x-ray screening of cabin baggage. Frontiers in Psychology, 10(256). https://doi.org/10.3389/fpsyg.2019.00256.

de Leeuw

J. R.

(2015). jsPsych: A JavaScript library for creating behavioral experiments in a web browser. Behavior Research Methods, 47(1), 1–12. https://doi.org/10.3758/s13428-014-0458-y

Dijkstra

J. J.

(1999). User agreement with incorrect expert system advice. Behaviour and Information Technology, 18(6), 399–411. https://doi.org/10.1080/014492999118832

Dijkstra

J. J.

Liebrand

W. B. G.

Timminga

(1998). Persuasiveness of expert systems. Behaviour and Information Technology, 17(3), 155–163. https://doi.org/10.1080/014492998119526

Dzindolet

M. T.

Peterson

S. A.

Pomranky

R. A.

Pierce

L. G.

Beck

H. P.

(2003). The role of trust in automation reliance. International Journal of Human-Computer Studies, 58(6), 697–718. https://doi.org/10.1016/s1071-5819(03)00038-7

10.

Dzindolet

M. T.

Pierce

L. G.

Beck

H. P.

Dawe

L. A.

(2002). The perceived utility of human and automated aids in a visual detection task. Human Factors, 44(1), 79–94. https://doi.org/10.1518/0018720024494856

11.

Hoesterey

Onnasch

(2022). The effect of risk on trust attitude and trust behavior in interaction with information and decision automation. Cognition, Technology, & Work, 25(1), 15–29. https://doi.org/10.1007/s10111-022-00718-y

12.

Hoff

K. A.

Bashir

(2015). Trust in automation: Integrating empirical evidence on factors that influence trust. Human Factors, 57(3), 407–434. https://doi.org/10.1177/0018720814547570

13.

Islam

M. R.

Ahmed

M. U.

Barua

Begum

(2022). A systematic review of explainable artificial intelligence in terms of different application domains and tasks. Applied Sciences, 12(3), 1353. https://doi.org/10.3390/app12031353

14.

Jian

J.-Y.

Bisantz

A. M.

Drury

C. G.

(2000). Foundations for an empirically determined scale of trust in automated systems. International journal of cognitive ergonomics, 4(1), 53–71. https://doi.org/10.1207/s15327566ijce0401_04

15.

Karras

Laine

Aittala

Hellsten

Lehtinen

Aila

(2020). Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2020, pp. 8107-8116. https://doi.org/10.1109/CVPR42600.2020.00813

16.

Lange

Kühn

Filevich

(2015). “Just Another Tool for Online Studies” (JATOS): An easy solution for setup and management of web servers supporting online studies. PLoS One, 10(6), Article e0130834. https://doi.org/10.1371/journal.pone.0130834

17.

Langer

König

C. J.

Back

Hemsing

(2021). Trust in artificial intelligence: Comparing trust processes between human and automated trustees in light of unfair bias. Journal of Business and Psychology. https://doi.org/10.31234/osf.io/r9y3t

18.

Langer

Landers

R. N.

(2021). The future of artificial intelligence at work: A review on effects of decision automation and augmentation on workers targeted by algorithms and third-party observers. Computers in Human Behavior, 123(1), 106878. https://doi.org/10.1016/j.chb.2021.106878

19.

Lee

J. D.

See

K. A.

(2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80. https://doi.org/10.1518/hfes.46.1.50_30392

20.

Lewis

Sycara

Walker

(2018). The role of trust in human-robot interaction. In Foundations of trusted autonomy (pp. 135–159). Springer International Publishing. https://doi.org/10.1007/978-3-319-64816-3_8

21.

Madhavan

Wiegmann

D. A.

(2007). Similarities and differences between human-human and human-automation trust: An integrative review. Theoretical Issues in Ergonomics Science, 8(4), 277–301. https://doi.org/10.1080/14639220500337708

22.

Mao

Wang

Tang

Qian

(2019). Opportunities and challenges of artificial intelligence for green manufacturing in the process industry. Engineering, 5(6), 995–1002. https://doi.org/10.1016/j.eng.2019.08.013

23.

McKinney

S. M.

Sieniek

Godbole

Godwin

Antropova

Ashrafian

Back

Chesus

Corrado

G. S.

Darzi

Etemadi

Garcia-Vicente

Gilbert

F. J.

Halling-Brown

Hassabis

Jansen

Karthikesalingam

Kelly

C. J.

King

Shetty

(2020). International evaluation of an AI system for breast cancer screening. Nature, 577(7788), 89–94. https://doi.org/10.1038/s41586-019-1799-6

24.

Mosier

K. L.

Manzey

(2020). Humans and automated decision aids: A match made in heaven? In Mouloua

Hancock

P. A.

(Eds.), Human performance in automated and autonomous systems: Current theory and methods (pp. 19–42). CRC Press.

25.

O’Neil

(2020). AI, ethics, and the law. In Work in the future (pp. 145–153). Springer International Publishing. https://doi.org/10.1007/978-3-030-21134-9_15

26.

Parasuraman

Riley

(1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230–253. https://doi.org/10.1518/001872097778543886

27.

Rice

Keller

(2009). Automation reliance under time pressure. Cognitive Technology, 14(1), 36–44.

28.

Rieger

Manzey

(2022). Understanding the impact of time pressure and automation support in a visual search task. Human Factors, 0(0), 1–17. https://doi.org/10.1177/00187208221111236

29.

Rieger

Roesler

Manzey

(2022). Challenging presumed technological superiority when working with (artificial) colleagues. Scientific Reports, 12(1), 3768. https://doi.org/10.1038/s41598-022-07808-x

30.

Roesler

Onnasch

Majer

J. I.

(2020). The effect of anthropomorphism and failure comprehensibility on human-robot trust. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 64(1), 107–111. https://doi.org/10.1177/1071181320641028

31.

Roesler

Rieger

Manzey

(2022). Trust towards human vs. automated agents: Using a multidimensional trust questionnaire to assess the role of performance, utility, purpose, and transparency. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 66(1), 2047–2051. https://doi.org/10.1177/1071181322661065

32.

Sha

L. Z.

Remington

R. W.

Jiang

Y. V.

(2018). Statistical learning of anomalous regions in complex faux x-ray images does not transfer between detection and discrimination. Cognitive Research: Principles and Implications, 3(1), 1–16. https://doi.org/10.1186/s41235-018-0144-1

33.

Wiczorek

Meyer

(2019). Effects of trust, self-confidence, and feedback on the use of decision automation. Frontiers in Psychology, 10(519), 519. https://doi.org/10.3389/fpsyg.2019.00519

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.43 MB