The Confidence Trap: Why People Overestimate the Accuracy of Smooth-Talking AI

As more people begin to use AI in new ways, one group of researchers looked at why humans seem to trust the answers provided by AI more than other humans, even though AI can still be unreliable and inaccurate. What the study revealed was AI's lack of emotion and visible cues may be the underlying reason.

AI is constantly evolving, with companies such as Google announcing advancements like Android Contextual Suggestions, which will quietly learn a user's routine and offer the right action at the right moment. There are also stories where AI has helped someone in a grand manner, such as Claude helping someone solve a decade-long crypto nightmare by helping them recover $400,000 in bitcoin. So, it isn't a big leap for someone to look at the headlines and have a false sense of trust in AI. It is this false sense of security perhaps that has also led to so many people placing more trust in AI than other humans.

"As AI systems become more and more sophisticated, people are increasingly relying on them for advice, for instance on which products to buy or what content to consume," commented Clara Colombatto, first author of the paper and part of the Department of Psychology at the University of Waterloo, in a recent interview.

Colombatto added that this behavior "mirrors something we do all the time when interacting with other people, as we often seek guidance and advice from others." She notes there is an important difference between the two: humans usually communicate in some way their level of confidence in what they are saying, with that level of confidence shaping how much trust someone ends up putting into the given advice. With AI, the cues humans provide are absent.

The researchers were interested in trying to understand how these factors influence how much trust someone puts into either another human, or an AI chatbot providing answers or advice. Colombatto added, "To understand how people attribute confidence to other agents, we designed a task in which participants observed either humans or AI systems making decisions and then reported how confident they thought the agent was in each decision."

After analyzing the data, the researchers found how different cues influenced the level of confidence participants placed in either AI agents or humans. The results also showed that participants believed the agents were more confident when they responded quickly, or when it appeared easier for them to answer. This held true even when the belief was not justified, according to Colombatto.

"When interacting with other people, we communicate confidence through a variety of cues, including our tone of voice, our facial expressions, our posture, etc.," added Colombatto. "AI systems, however, often lack these features, as most systems do not have human-like voices or physical presence. Understanding which kinds of confidence signals are most helpful, transparent, and trustworthy in AI-human interactions will therefore be an important direction for future research."

AI companies will likely view the results of this study and design AI systems to convey more confidence in the answers provided by their systems, directly and otherwise. Colombatto points out with human-to-human interactions the level of confidence associated with an answer plays a large role in how much trust is placed in the person giving the answer. What companies may attempt to do with AI is find new ways of providing social type cues to instill more trust in users than already exists. This will most likely play a bigger role in chatbots that have voice capabilities, and eventually those which include a visual reference such as a face. When this occurs, more studies will certainly follow.