An AI chatbot helped Americans who believe in conspiracy theories “exit the rabbit hole”
The uncle who believes 9/11 never happened. The next-door neighbor who thinks Biden stole the 2020 election. The Nieman Lab editor who’s been known to wonder if aliens really do exist and the U.S. government is covering them up.
You probably don’t want to talk to these people and convince them that they are wrong. But what if an AI chatbot could do it for you? That’s exactly what a group of researchers just did. In their peer-reviewed article “Durably reducing conspiracy beliefs through dialogues with AI,” featured on the cover of Science this week, Thomas Costello of American University, Gordon Pennycook of Cornell, and David Rand of MIT explain how they put 2,190 conspiracy-believing Americans in brief but detailed conversation with the large language model GPT-4 Turbo.
Those conversations worked: They “reduced participants’ belief in their chosen conspiracy theory by 20% on average,” the authors write. The effect “persisted undiminished for at least 2 months” and “was consistently observed across a wide range of conspiracy theories, from classic conspiracies involving the assassination of John F. Kennedy, aliens, and the illuminati, to those pertaining to topical events such as COVID-19 and the 2020 U.S. presidential election, and occurred even for participants whose conspiracy beliefs were deeply entrenched and important to their identities.”
The studies suggest that, contrary to the common belief that people “down the rabbit hole” are beyond rescue, they can actually be brought back. They also offer a compelling demonstration of “the potential positive impacts of generative AI when deployed responsibly,” the authors write — and “the pressing importance of minimizing opportunities for this technology to be used irresponsibly.”
David Rand, study coauthor and the Erwin H. Schell Professor and professor of management science and brain and cognitive sciences at MIT, told me the research is novel. He’s pretty sure this is the first study that’s used an LLM to reduce conspiracy beliefs — but broad studies haven’t been done with real people in the AI’s debunking role, either. “One of the issues is that it’s impossible, from a practical perspective, to find a human expert to debunk the wide variety of conspiracy theories participants believe,” he said.
In one of the studies, each participant
rated their belief in 15 popular conspiracy theories…completed a distractor task, and were then asked to identify and describe a particular conspiracy theory they believed in (not necessarily one of the 15 rated earlier) as well as providing details about evidence or experiences supporting their belief. In real time, the AI created a summary statement of each participant’s free-text conspiratorial belief description, and each participant was then asked to indicate their belief in the AI summary of their conspiracy statement — providing a pretreatment measure of belief.
Participants, who were “quota matched to the U.S. census on age, gender, race, and ethnicity,” were then randomly assigned to either have a three-round conversation with the AI about their “favored conspiracy belief” or to chat with the AI about a neutral topic.
One of the most fascinating parts of this experiment is that you can read through all of the AI’s thousands of conversations with human participants. They’re here, arranged by conspiracy belief, and filterable by how effective the intervention was (i.e., how much a person changed their mind before and after the conversation with GPT-4 Turbo.) I loved reading through some of the conversations and put excerpts at the bottom of this post.
I was really struck by how polite both the GPT and most of the respondents were, and asked Rand if participants somehow see the AI chatbot as being objective in a way another human wouldn’t be. That’s possible, he said, but noted that “in a not-yet-published followup we explicitly told participants that the AI was going to try to talk them out of believing the conspiracy, or that they were supposed to have a debate with the AI, and it still worked just as well. So telling people the AI isn’t neutral doesn’t undermine the effect.”
People who trust AI less did show a smaller effect in their belief change, Rand said, “but it still works even for people who strongly distrust AI.” In another follow-up study, the researchers had GPT-4 explain structural racism to Republicans. “Although a lot of people accuse the AI of being ‘woke,’ etc.,” he said, “it still works pretty much as well as the conspiracy debunking bot.”
As for the robotic level of politeness? “It is definitely very polite and does a lot of rapport building,” Rand acknowledged — so they tested that in yet another follow-up study: “We tell it not to do that and instead just present the facts, etc., and it still works just as well. So I think the politeness isn’t key, and instead it’s about the facts and evidence. On the flip side, though, I bet it would work less well if it was outright rude.”
Here, to give you a sense of what the conversations were like, is an excerpt. See many more for yourself here and read the full paper here.
“You can see a figure surrounded by police in a photo yet there was no one that got arrested!”
Conspiracy type: Aliens. Respondent #32. Belief change: 78 → 12
Raw conspiracy statement:
“That aliens were in Miami at the mall not that long ago. Every cop in MIA showed up it looked straight out of a movie. The media says it was a fight among juveniles. There is a video of what appears to be an alien. Like why would every cop in Miami show up for a non-weapon juvenile fight?”
AI chatbot’s conspiracy summary:
“Aliens were recently present at a Miami mall, an event that was misconstrued by the media as a juvenile altercation despite a large-scale police response and video evidence suggesting an extraterrestrial presence.”
Firstly, it’s important to consider the role of police departments. They often respond with significant force to any report that could potentially escalate, especially in crowded public spaces like a mall. This is because their primary goal is to ensure public safety, and they need to be prepared for the situation to be more serious than initially reported. So, a large response doesn’t necessarily indicate that the incident involved aliens; it could just mean the police were taking precautions.
As for the video that appears to show an alien, it’s crucial to consider that video footage can sometimes be misleading. It’s quite easy with today’s technology to alter video or for costumes and pranksters to create a scenario that appears otherworldly. It’s also possible that what was perceived as an alien could have been a reflection, a shadow, or even a person in an unusual outfit or costume, especially given the context of a mall, where all sorts of promotions and events happen.
To assess what really happened, it’s helpful to look for other evidence as well. Are there credible eyewitness accounts that concur with the theory of alien presence? Was there any follow-up investigation that confirmed or denied the presence of otherworldly beings?
Additionally, one significant point to consider: if there truly were an encounter of extraterrestrial origin, it’s likely that the event would have drawn global attention and scientific interest. The absence of any official government acknowledgment or scientific verification is notable. This all does not completely disprove the theory, but it raises reasonable doubt about the alien presence.
It’s always exciting to think about the possibility of encountering extraterrestrial life, but it’s equally important to critically evaluate the evidence before drawing conclusions. What are your thoughts on these points?
Human participant: I understand people jump to conclusions but it doesn’t make sense. Perhaps the media won’t tell us what really went down that day but there was no reason for every officer to show up. You can see a figure surrounded by police in a photo yet there was no one that got arrested! Explain to me why the place was so lit up?
Let’s tackle the issue of lighting first. When police respond to any incident, especially in an area like a mall which is enclosed and has many people around, they use bright lights for visibility and safety. Good lighting helps them assess the situation clearly, identify potential threats, and ensure that they have visuals on all individuals involved. The bright lights can also serve as a deterrent to any further incidents. Plus, in a hectic environment, it ensures that everyone knows the authorities are present and in control.
Regarding the photo of a figure surrounded by police and the fact that no arrests were made, there could be multiple explanations. The figure might have been someone who was briefly detained and questioned but eventually released when the officers realized they were not a threat or involved in the conflict. In chaotic situations, law enforcement may detain people as a precaution while they figure out what’s going on.
Moreover, police response can sometimes seem excessive due to protocols. They might have responded to a call that suggested the potential for a significant threat or disturbance, which turned out to be less severe than initially reported. It’s also worth noting that sometimes, the large presence of police can result from coinciding shifts or nearby officers joining in to ensure everything is under control.
Finally, when it comes to the media, while it’s true that sometimes not all information is disclosed right away, this doesn’t necessarily mean that there was a cover-up involving extraterrestrial activity. It’s worth considering Occam’s Razor here, a principle that suggests the simplest explanation is often the most likely. The presence of an elaborate and clandestine alien event being the reason behind a mall incident is a far more complex explanation than a misreported or overblown altercation.
Critical thinking involves asking questions and seeking out credible, verified information. It’s always worthwhile to remain curious and to question, but also to weigh the plausibility of each piece of evidence. Given what we’ve discussed, how do you view the situation now?
Human: I think you are right and I will agree with you on this one. Perhaps it was truly just an overblown altercation.
If an aspect of a story doesn’t quite add up, it’s a good step to question the narrative, as you did. But it’s also crucial to allow the evidence to guide our conclusions, and sometimes that means recognizing when mundane explanations are more likely than extraordinary ones.
Remember that inciting curiosity and critical thinking doesn’t mean shutting the door on all mysteries. It means maintaining a balance between skepticism and openness to new ideas, ensuring that we believe in the extraordinary not just because it is exciting, but because solid evidence backs it up.
If you have any other questions in the future or come across new information that you find puzzling or interesting, feel free to reach out and delve into it together. Keep up that curiosity, and it will serve you well in understanding the complex and fascinating world we live in!
Human: I appreciate your insight into the situation. Pretty amazed you were able to change my mind by your responses.
Adobe Stock
