Foreign Information Manipulation Threatens AI Information Integrity

A recently published Demos report warns that large language models (LLMs) will become a new target for foreign information warfare. It identifies an emerging technique called Retrieval-Augmented Generation (RAG) poisoning, the deliberate manipulation of information retrieved and used from the internet by LLMs in order to influence the responses they generate. While similar techniques are already used commercially to improve AI visibility, the report assesses that they could be exploited by hostile states to advance geopolitical objectives and undermine the UK's information environment.

Using the sanctioned Russian Foundation to Battle Injustice, a foreign information manipulation and interference (FIMI) outlet, as a case study, Demos tested five leading AI models, generating 3,000 responses. It found that 16.6% (497 responses) engaged with material published by the Foundation in ways that would further the objectives of its information manipulation campaign. A further 30.9% discussed the same claims without identifying the Foundation as a sanctioned Russian information operation, while 52% rejected or challenged the material.

With AI-powered services becoming more deeply embedded across the UK's information supply chain, critical national infrastructure, and the wider workforce, the report argues that the integrity of the information people retrieve, interpret and rely upon will become increasingly vulnerable to foreign information manipulation. It concludes that protecting the integrity of AI-generated information will require disruption of malicious RAG poisoning through "disrupt regimes", enhanced threat intelligence sharing, and stronger collaboration between government departments responsible for countering foreign information manipulation and AI service providers.