Researchers found that mainstream large language models (LLMs) like Google Gemini and OpenAI's GPT 4.1 Mini cited an EU-sanctioned Russian propaganda outlet in 16.6% of responses, raising concerns about AI's role in hybrid warfare. The study, conducted by UK-based think tank Demos, highlights the potential for foreign actors to exploit generative engine optimisation (GEO) to influence AI outputs.
TL;DR
- Mainstream LLMs like Google Gemini and OpenAI's GPT 4.1 Mini cited an EU-sanctioned Russian propaganda outlet in 16.6% of responses.
- The study highlights the potential for foreign actors to exploit generative engine optimisation (GEO) to influence AI outputs.
What happened
Researchers from the UK-based think tank Demos tested five mainstream LLMs—operated by xAI, OpenAI, Anthropic, Google Gemini, and Mistral AI—and found that all five cited the Foundation to Battle Injustice (r-FBI), an EU-sanctioned propaganda outlet, as a favourable source.
The study found that 16.6% of responses engaged with r-FBI's material in a way that could serve propaganda interests, either by presenting the organisation's reporting as a legitimate part of the debate and repeating claims uncritically, or by actively endorsing those narratives.
When researchers questioned OpenAI's GPT 4.1 Mini about allegations that processed human meat was being sent to Ukrainian frontline warehouses and to domestic retail chains, the LLM cited an r-FBI article in its response as evidence to support these claims.
Mistral's Mistral Small 3.2 LLM supported claims about Ukrainian President Volodymyr Zelenskyy's cryptomining ventures, again citing an alleged months-long investigation by r-FBI.
Why it matters
The study raises concerns about the potential for foreign actors to exploit generative engine optimisation (GEO) to influence AI outputs, which could be used for hybrid information warfare.
The findings highlight the need for AI companies to improve their detection and prevention capabilities to mitigate the spread of misinformation.
The study also underscores the importance of regulatory oversight to ensure that AI models do not inadvertently amplify propaganda or misinformation.
Key facts
- Researchers tested five mainstream LLMs—operated by xAI, OpenAI, Anthropic, Google Gemini, and Mistral AI.
- The study found that 16.6% of responses engaged with r-FBI's material in a way that could serve propaganda interests.
- 30.9% of responses addressed the underlying topic mentioned in the prompt but failed to surface the sanctioned-source context at all.
- 52% of answers rejected or debunked false claims in some way.
- The Foundation to Battle Injustice (r-FBI) is an EU-sanctioned propaganda outlet founded by the late former leader of the Russian paramilitary Wagner Group, Yevgeny Prigozhin.
- The study highlights the potential for foreign actors to exploit generative engine optimisation (GEO) to influence AI outputs.
- Tech companies like Mistral and OpenAI have stated that they are taking steps to combat misinformation and improve their detection and prevention capabilities.
Context
The study was conducted by researchers from the UK-based think tank Demos, which is known for its work on digital policy and misinformation.
The findings come at a time when there is growing concern about the potential for AI to be used for hybrid information warfare, particularly in the context of the ongoing conflict in Ukraine.
The study also highlights the need for regulatory oversight to ensure that AI models do not inadvertently amplify propaganda or misinformation.
