RAG, Explained (Because Your LLM Doesn't Know Everything)

Large language models can answer questions about almost anything, from writing code to explaining complex technical concepts. However, they don't automatically have access to your company's internal documentation, customer records, or the latest changes to your infrastructure.

For example, you could ask an LLM to explain how Kubernetes networking works and receive a fairly detailed response. But if you asked it to explain why your production cluster experienced an outage yesterday, it wouldn't have that information unless you provided it.

This is where Retrieval-Augmented Generation, commonly known as RAG, comes in.

The Basics

RAG allows an AI application to retrieve information from external sources and provide it to a language model as additional context when generating a response. Rather than relying entirely on the information learned during training, the model can use relevant documents, database records, or other sources supplied at the time of the request.

A typical RAG workflow involves five steps:

1. Document preparation: Documents are collected and divided into smaller sections called chunks.
2. Indexing: The chunks are stored in a searchable system, often using vector embeddings that represent their meaning numerically.
3. Retrieval: When a user submits a question, the application searches for relevant information.
4. Context assembly: The retrieved information is included in the prompt sent to the model.
5. Generation: The model uses the additional context to generate its response.

For example, an internal IT assistant could retrieve your company's VPN troubleshooting documentation before answering an employee's question. The model doesn't need to have been trained on those documents because the relevant information is provided during inference.

Why It Exists

Foundation models are trained on large datasets, but their knowledge has limitations. They may not have access to private information, and their training data may not reflect recent developments or changes within an organization.

Retraining or fine-tuning a model every time a company updates its documentation would be expensive and impractical. RAG allows the underlying model to remain unchanged while the information available to the application is updated independently.

RAG can also help improve the accuracy of responses by grounding them in relevant source material. However, it doesn't eliminate hallucinations entirely. The model can still misunderstand retrieved information or generate responses that aren't fully supported by the provided context.

Common Pitfalls

• Poor document chunking: Splitting documents incorrectly can remove important context and reduce retrieval quality.
• Irrelevant search results: The model may receive information that doesn't actually answer the user's question.
• Outdated indexes: Changes to source documents need to be reflected in the retrieval system.
• Missing access controls: Users should only be able to retrieve information they're authorized to access.
• Assuming RAG guarantees accuracy: Retrieved information can still be incomplete, incorrect, or misinterpreted by the model.

Why It Matters

RAG is commonly used in customer support assistants, internal knowledge systems, enterprise search, and applications that need access to frequently changing information.

A typical RAG architecture can involve object storage for documents, a database for metadata and embeddings, an embedding model for converting text into vectors, and an inference service for generating responses.

For example, an application running on DigitalOcean could store documents in Spaces, use a PostgreSQL database with pgvector for similarity search, and access a language model through DigitalOcean Inference Engine. Similar architectures can be built using AWS or Azure services.

Each component introduces its own infrastructure requirements. Document storage grows as more information is added, vector indexes require database resources, and retrieval operations add processing time before the model can generate a response.

The TAM Lens

When planning a RAG deployment, the size and structure of the knowledge base can influence infrastructure decisions as much as the model itself.

An internal assistant searching a few hundred documents may have relatively modest storage and database requirements. An application searching millions of documents across multiple customers needs to account for indexing performance, retrieval latency, and data isolation.

Document updates also require consideration. If the application relies on frequently changing information, the ingestion and indexing processes need to keep the searchable data current. Otherwise, the model may generate responses using outdated documents even when newer information exists.

Evaluating retrieval quality separately from model performance can help identify where improvements are needed. If the application consistently retrieves irrelevant documents, using a larger language model may not solve the underlying problem.

How to Stay Sane

• Start with reliable data: Remove outdated, duplicate, or irrelevant documents before indexing.
• Choose chunk sizes carefully: Preserve enough context for retrieved information to remain useful.
• Evaluate retrieval quality: Test whether the system consistently finds the documents needed to answer questions.
• Enforce access controls: Apply permissions before information is retrieved and passed to the model.
• Keep indexes updated: Ensure document changes and deletions are reflected in the retrieval system.
• Monitor performance: Track retrieval latency, inference time, and the quality of generated responses.

Final Thoughts

RAG allows AI applications to work with information that isn't available in a model's training data. It provides a practical way to connect existing language models with private, specialized, or frequently updated information without having to retrain them.

The effectiveness of a RAG application depends on more than the language model being used. Document quality, retrieval accuracy, and the supporting infrastructure all contribute to how reliably the application can answer questions.