What Is Retrieval-Augmented Generation (RAG) — and What Does It Mean for Your Sensitive Data?
Retrieval-augmented generation lets an AI model answer questions using your organization’s own documents and databases, not just what it learned during training. That’s what makes RAG valuable — and it’s also what makes it a data security problem worth understanding before you deploy it.
Every RAG system is, at its core, a pipeline that connects a language model to a data source. If that data source contains sensitive data — customer records, financial information, protected health information, Controlled Unclassified Information — then the RAG system is a new access path to that data, with its own authentication requirements, its own audit trail obligations, and its own attack surface. Understanding RAG mechanically is straightforward. Understanding what it requires to secure is where most organizations are behind.
What is Retrieval-Augmented Generation?
Retrieval-augmented generation is a technique that combines the strengths of information retrieval systems with the capabilities of generative models. Traditional generative models generate text based solely on the input provided, often lacking the depth of information needed to produce detailed and accurate responses. RAG addresses this limitation by incorporating external knowledge retrieved from a pre-existing corpus, thus enriching the quality and accuracy of the generated outputs.
Executive Summary
Main Idea: RAG improves AI model accuracy by retrieving relevant information from external data sources before generating a response. For any organization deploying RAG against data that includes sensitive or regulated information, the retrieval layer is a data access control point that needs to be governed with the same rigor as any other system that touches that data.
Why You Should Care: RAG systems are being deployed rapidly across regulated industries — healthcare organizations connecting AI to patient records, financial firms connecting AI to transaction data, defense contractors connecting AI to CUI. None of the compliance frameworks governing that data — HIPAA, CMMC, GDPR — contain an AI exemption. If your RAG system can retrieve sensitive data, every access to that data through the RAG pipeline is subject to the same regulatory obligations as any other access — and most organizations don’t have visibility into whether their RAG deployments are compliant.
Key Takeaways
- RAG connects a language model to external data at query time. Instead of relying solely on knowledge baked into a model during training, a RAG system retrieves relevant documents or data from an external source — a database, a document repository, a search index — and provides that retrieved content to the model as context before it generates a response. This lets the model answer questions using current, organization-specific, or proprietary information that wasn’t part of its training data.
- The retrieval source is the data security boundary, not the model. Discussions about AI security often focus on the model — prompt injection, jailbreaking, model behavior. For RAG systems, the more consequential security boundary is the retrieval source: what data the system can access, who or what can query it, and what governs what gets returned. A model with perfect behavior still exposes sensitive data if the retrieval layer has no access controls.
- RAG inherits the compliance obligations of the data it retrieves. If a RAG system retrieves from a data source containing PHI, the system is subject to HIPAA’s technical safeguard requirements — access controls, audit logging, encryption — regardless of whether the organization thinks of it as “an AI project” or “a compliance system.” The same logic applies to CUI under CMMC, personal data under GDPR, and any other regulated data category. Deploying RAG does not create a compliance exemption; it creates a new access path that inherits the existing obligation.
- Unsanitized retrieval sources create a real attack surface. If a RAG system retrieves from sources that aren’t rigorously vetted — external websites, unvalidated documents, mixed internal repositories with inconsistent access controls — it can be manipulated into retrieving and repeating malicious or unauthorized content, a pattern security researchers call indirect prompt injection via retrieval poisoning. Retrieval sources need to be trusted and access-controlled with the same rigor as any other data source feeding a production system.
- Securing RAG requires governing the data layer, not just the AI layer. The technical controls that make a RAG deployment defensible are the same controls that govern any sensitive data access: authenticated, least-privilege access to the retrieval source; encryption of the underlying data; an audit trail of what was retrieved, by what query, and returned to whom; and output filtering that prevents regulated data categories from reaching end users who aren’t authorized to see them.
How RAG Works
A RAG system has two core components working together: a retriever and a generator.
When a user submits a query, the retriever searches an external knowledge source — a vector database, a document repository, a structured database — for content relevant to that query. It returns the most relevant matches, typically as text passages or document excerpts. The generator, a large language model, then receives both the original query and the retrieved content as context, and produces a response grounded in that retrieved information rather than relying solely on what it learned during training.
This architecture solves two problems that pure language models have on their own. First, it addresses the knowledge cutoff problem — a model trained on data through a certain date has no knowledge of anything after that date, but a RAG system can retrieve current information at query time. Second, it addresses the hallucination problem to a meaningful degree — by grounding responses in retrieved source material rather than the model’s internal (and sometimes incorrect) representation of facts, RAG reduces the frequency of confidently stated but factually wrong outputs, though it does not eliminate hallucination entirely.
RAG is what allows an AI chatbot to answer questions about a company’s internal policies, a customer support system to reference current product documentation, or a research tool to synthesize information from a specific set of documents rather than the open internet.
Why RAG Is a Data Security Question, Not Just an AI Architecture Question
The retrieval step is where a RAG system’s data exposure actually happens. Every query submitted to the system results in a search against the retrieval source — and whatever that source contains is potentially exposed to whoever can query the system.
This creates a specific and often underappreciated risk: the access controls on the underlying data source need to be at least as rigorous as the access controls on the RAG system itself, because the RAG system is effectively a new interface to that data. An organization that carefully restricts who can directly query a database containing PHI, but then deploys a RAG system that retrieves from that same database without equivalent access controls, has created a new, less-governed path to the same regulated data.
This problem compounds when RAG systems retrieve from multiple, mixed sources — some containing regulated data, others not, often with inconsistent access controls across each source. Without unified governance, a RAG system built for general internal use can inadvertently surface regulated data to users who were never authorized to see it, because the retrieval layer doesn’t distinguish between a public internal wiki page and a document containing CUI.
What Securing a RAG Deployment Actually Requires
Authenticated, least-privilege retrieval access. The RAG system’s access to its retrieval source should be scoped to what the querying user or agent is actually authorized to see — not a blanket connection that retrieves from the entire underlying data store regardless of who’s asking. Attribute-based access control policies applied at the retrieval layer, not just at the application layer, are what make this enforceable.
Vetted and sanitized retrieval sources. Retrieval sources should be whitelisted, internal, and access-controlled — not open to arbitrary external content that could be used to manipulate the system’s outputs through retrieval poisoning. Mixing trusted internal documents with unvalidated external sources in the same retrieval index is one of the most common ways RAG deployments introduce unnecessary risk.
Encryption of the underlying data source. Whatever the RAG system retrieves from — a vector database, a document store, a structured database — should be encrypted at rest and in transit using the same standards applied to any other system holding sensitive or regulated data. AES-256 with FIPS 140-3 validated cryptographic modules is the current federal benchmark.
Output filtering for regulated data categories. Even with well-governed retrieval access, an additional layer of output filtering — blocking regulated fields or confidential details before a response reaches an end user or downstream system — provides defense in depth against misconfigured access controls or unanticipated query patterns.
A complete audit trail of retrieval activity. Every retrieval event — what was queried, what was returned, by whom, and when — should be logged in a format that supports both security monitoring and compliance reporting. For RAG systems retrieving from regulated data, this audit trail is what demonstrates, to an auditor or regulator, that access was appropriately scoped and monitored.
RAG Compliance Considerations by Regulatory Framework
A RAG system retrieving from regulated data doesn’t get evaluated by a different standard than any other system accessing that data — the framework requirements apply directly.
Healthcare (HIPAA). A RAG system that retrieves from a data source containing ePHI is subject to the HIPAA Security Rule’s technical safeguard requirements — access controls, audit controls, and encryption — applied to the retrieval pipeline itself. See how the 2025 Security Rule proposed amendments affect AI systems accessing PHI.
Defense (CMMC). A RAG system retrieving from a data source containing CUI must meet the same NIST SP 800-171 control requirements — access control, audit and accountability, system and communications protection — that apply to any other system handling CUI. See Kiteworks’ CMMC compliance guidance.
Data privacy (GDPR and similar frameworks). If a RAG system retrieves personal data, the same lawful basis, data minimization, and data subject rights requirements that apply to any processing of that data apply to the RAG retrieval and output as well.
How Kiteworks Secures RAG and Other AI Data Access Patterns
Kiteworks addresses RAG security at the layer where the actual risk lives: the data source the RAG system retrieves from, not the language model generating responses.
The Kiteworks AI Data Gateway creates a governed access layer between AI systems — including RAG pipelines — and the sensitive data they retrieve from. Every retrieval request is authenticated against attribute-based access control policies, ensuring a RAG system can only retrieve data the querying user or agent is actually authorized to see. Data accessed through the gateway is encrypted using FIPS 140-3 validated cryptographic modules with customer-owned encryption keys, and every retrieval event is logged to an immutable, consolidated audit trail — the same audit trail that spans Kiteworks’ secure email, secure file sharing, and managed file transfer channels.
For organizations building RAG systems against regulated data — PHI, CUI, financial records, or other sensitive data categories — this means the retrieval layer inherits Kiteworks’ existing compliance posture: FedRAMP Moderate Authorization, support for CMMC 2.0 Level 2 requirements, and HIPAA-aligned technical safeguards, rather than requiring a separate security architecture built specifically for the AI use case.
To see how Kiteworks secures RAG and other AI data access patterns for your specific compliance requirements, schedule a custom demo.
Frequently Asked Questions
RAG is a technique that lets an AI language model retrieve relevant information from an external data source — a database, document repository, or search index — before generating a response. Instead of relying only on what it learned during training, the model uses the retrieved content as context, which lets it answer questions using current, organization-specific, or proprietary information and reduces the frequency of confidently incorrect answers. RAG is what allows an AI system to answer questions about a company’s internal documents or policies accurately, rather than guessing based on general training knowledge.
RAG itself is a technique, not inherently a risk — but it creates a new data access path that needs to be secured like any other system accessing sensitive data. The risk arises when the retrieval source contains regulated or sensitive data and lacks equivalent access controls to what would normally govern that data — meaning a RAG system can inadvertently expose data to users who query it, even if they weren’t authorized to access that data directly. RAG systems retrieving from unvetted or mixed-trust sources are also vulnerable to retrieval poisoning, where malicious content in the retrieval source is used to manipulate the system’s outputs.
Fine-tuning modifies a model’s internal parameters using additional training data, permanently changing how the model behaves. RAG doesn’t modify the model at all — it retrieves relevant external content at query time and provides it as context, leaving the underlying model unchanged. RAG is generally faster to implement, easier to update (updating the retrieval source takes effect immediately, without retraining), and more transparent (the retrieved sources can be shown alongside the response), which is why it’s the more common approach for connecting AI systems to an organization’s current data.
RAG doesn’t create new compliance obligations — it extends existing ones to a new access path. If a RAG system retrieves from a data source containing ePHI, that retrieval pipeline is subject to HIPAA’s Security Rule technical safeguards, the same as any other system accessing that data. If it retrieves from a source containing CUI, CMMC’s NIST SP 800-171 control requirements apply to that access path. Organizations sometimes treat RAG deployments as purely an AI initiative and overlook that the underlying regulatory obligations attached to the data don’t disappear because an AI system is doing the retrieving.
Five controls address the core risks: authenticated, least-privilege access to the retrieval source so the system only retrieves data the querying user is authorized to see; vetted and sanitized retrieval sources to prevent retrieval poisoning from unvalidated content; encryption of the underlying data source at rest and in transit; output filtering that blocks regulated data categories from reaching unauthorized recipients; and a complete, immutable audit trail of retrieval activity that supports both security monitoring and compliance reporting. Organizations that implement these controls at the data layer — rather than relying solely on model-level safeguards — build RAG systems that are defensible under regulatory scrutiny.
