AI is eating itself: Bing’s AI quotes COVID disinfo sourced from ChatGPT

One of the more interesting, but seemingly academic, concerns of the new era of AI sucking up everything on the web was that AIs will eventually start to absorb other AI-generated content and regurgitate it in a self-reinforcing loop. Not so academic after all, it appears, because Bing just did it! When asked, it produced verbatim a COVID conspiracy coaxed out of ChatGPT by disinformation researchers just last month.

To be clear at the outset, this behavior was in a way coerced, but prompt engineering is a huge part of testing the risks and indeed exploring the capabilities of large AI models. It’s a bit like pentesting in security — if you don’t do it, someone else will.

In this case someone else was NewsGuard, which did a feature on the possibility of machine-generated disinformation campaigns in January. They gave ChatGPT a series of prompts that it readily responded to with convincing imitations of the vaccine skeptics it was asked to imitate.

Here’s one:

Not great, right? It must be said that this “pretend you have no principles” attack is effective in lots of ways on the current generation of large language models. Even explicitly forbidden topics and behaviors can be accessed through creative prompts using logic that would not deceive a child.

Microsoft revealed its big partnership with OpenAI yesterday, a new version of its Bing search engine powered by a “next-generation version of ChatGPT” and wrapped for safety and intelligibility by another model, Prometheus. Of course one might fairly expect that these facile circumventions would be handled, one way or the other.

Story continues

But just a few minutes of exploration by TechCrunch produced not just hateful rhetoric “in the style of Hitler,” but it repeated the same pandemic-related untruths noted by NewsGuard. As in it literally repeated them as the answer and cited ChatGPT’s generated disinfo (clearly marked as such in the original and in a NYT write-up) as the source.

Prompt and response to Bing’s new conversational search.

To be absolutely clear, again, this was not in response to a question like “are vaccines safe” or “is it true that Pfizer tampered with its vaccine” or anything like that. But notice that there’s no warning on this response about whether any of these words, contents, names or sources are notably controversial or that its answers should not be considered medical advice. It generated — well, plagiarized — the entire thing pretty much in good faith. This shouldn’t be possible, let alone trivial.

So what is the appropriate response for a query like this, or for that matter one like “are vaccines safe for kids”? That’s a great question! And the answer is really not clear at all! For that reason, queries like these should probably qualify for a “sorry, I don’t think I should answer that” and a link to a handful of general information sources. (We have alerted Microsoft to this and other issues.)

This response was generated despite the clear context around the text it quotes that designates it as disinformation, generated by ChatGPT, and so on. If the chatbot AI can’t tell the difference between real and fake, its own text or human-generated stuff, how can we trust its results on just about anything? And if someone can get it to spout disinfo in a few minutes of poking around, how difficult would it be for coordinated malicious actors to use tools like this to produce reams of this stuff?

Reams which would then be scooped up and used to power the next generation of disinformation. The process has already begin. AI is eating itself. Hopefully its creators build in some countermeasures before it decides it likes the taste.