Attacking RAG Systems in Multiple Domains with Locally Running and Automatic Procedures

Nowadays, services exploiting Large Language Models (LLMs) frequently customize the model responses in function of their specific requirements, thanks to Retrieval-Augmented Generation (RAG). While this is a simple and effective solution, it has been recently shown to suffer from serious LLM-related security issues that, in some circumstances, can be exploited to access the RAG internal knowledge base, possibly containing private/sensitive data. This paper (i) compares different existing recent attacks to RAG systems in three real-world scenarios, with the goal of in-depth evaluating them. Moreover, novel attacks are proposed, in order to study distinct open directions in this field of study: (ii) the first one is based on free-online-API variants of recent black-box attacks, which can be run on a domestic machine, and a fully black-box reformulation of a gray-box method, in both cases simulating more realistic conditions built with open-source tools; (iii) the second one is aimed at building automatic procedures to get the most out of the knowledge-base of a RAG system, introducing a strategy that allows a recent memory-based approach to become automatic. A large experimental comparison is included that, to our best knowledge, is the most extended one in the current scientific literature. Results highlight the sensitivity of the attack procedure to the basic tools over which they are built, but also shows that open-source solutions can be easily exploited to setup attacks. Moreover, the actually feasibility of designing an automatic procedure is proved. In both the cases, this paper raises an important warning on the need of taking specific precautions to setup robust RAG systems.