Why RAG Keeps Coming Up in Business Conversations
Many business owners and technical founders hear that the best way to make AI understand their business is to train a custom model from scratch. In practice, for the vast majority of business use cases, that path is expensive, slow to update, and often the wrong tool entirely. Retrieval-Augmented Generation, or RAG, offers a far more practical alternative: instead of baking knowledge into a model's weights through fine-tuning, RAG pulls relevant information from your actual source documents at the moment a question is asked, then feeds it into the prompt sent to the model.
What RAG Actually Is: From Raw Documents to Grounded Answers
At its core, RAG runs through five steps. First, documents are broken into smaller pieces called chunks, typically a few hundred tokens each. Second, each chunk is converted into a numerical representation called an embedding using an embedding model, where texts with similar meaning end up mathematically close to one another. Third, these embeddings are stored in a vector database that supports searching by semantic similarity rather than exact word matching.
When a user asks a question, the system converts that question into an embedding too, then searches for the chunks that are most semantically relevant (the retrieve step). Those chunks get inserted into the prompt alongside the original question, and the language model generates an answer grounded in that real context rather than relying purely on what it learned during training. One notable strength of this semantic search approach is its ability to surface relevant documents even when the wording barely overlaps with the query, something plain keyword search simply cannot do.
For small to mid-sized workloads, you don't always need a separate, dedicated vector database service. Extensions like pgvector for PostgreSQL can handle millions of vectors with solid performance for most business use cases, while avoiding the added complexity of keeping two separate systems in sync.
RAG vs Fine-Tuning vs Long Context: Which Fits Your Business?
These three approaches are often mistaken for interchangeable solutions, but each comes with distinct trade-offs.
| Aspect | RAG | Fine-Tuning | Long Context |
|---|---|---|---|
| Upfront cost | Moderate (build a retrieval pipeline) | High (training data, compute) | Low (just add documents to the prompt) |
| Data freshness | Real-time, just update source documents | Requires retraining to update | Real-time, just swap prompt content |
| Maintenance cost | Moderate, pipeline needs upkeep for chunking and retrieval | High, needs periodic retraining | Low, but per-query cost grows with context size |
| Data scale | Good for large knowledge bases | Good for changing model behavior or style | Good for small-to-medium knowledge bases |
Fine-tuning makes sense when you want to change how a model responds or teach it a new skill, not simply add more facts to draw on. Meanwhile, the long-context approach, dropping entire documents straight into the prompt, has become dramatically cheaper and faster thanks to prompt caching, which can cut costs substantially for documents that get reused across many queries.
The Under-200k-Token Rule: When You Don't Need RAG at All
This is the part most teams skip past: if your entire knowledge base is smaller than 200,000 tokens, roughly 500 pages of material, you can simply include the whole thing in the prompt with no RAG system required. Prompt caching makes this approach significantly faster and more cost-effective, since the same document doesn't need to be reprocessed from scratch on every new question. Only once your knowledge base grows past this threshold does building a full RAG system start to make sense at scale.
Many small and mid-sized businesses, especially those just starting to explore AI, actually sit below this threshold. Building a complex RAG pipeline for a problem that long context with caching could solve outright is a waste of engineering effort.
Where RAG Breaks in Practice
RAG sounds simple on paper, but real implementations run into predictable failure modes. Three of the most common:
Bad chunking. When documents are split carelessly, a chunk can lose the context that made it meaningful. A chunk that reads "the company's revenue grew by 3% over the previous quarter," for instance, is useless if the system has no idea which company or which quarter it refers to. The fix is prepending explanatory context to each chunk before it gets embedded, a technique shown to meaningfully cut retrieval failures.
Keyword misses. Embedding models excel at capturing semantic meaning, but they can miss precise keyword matches, like a specific error code or product SKU. Combining semantic search with BM25, a classic keyword-based ranking technique, helps close that gap.
Missing context at retrieval time. Even with solid chunking and search, a system can still fall short if too few chunks are retrieved or if the most relevant ones aren't ranked highly enough. Adding a reranking step, where an initial batch of candidates gets rescored by a dedicated model before reaching the LLM, has been shown to push accuracy meaningfully higher, at the cost of added latency and per-query expense.
Realistic Scope for a First RAG Project
Don't start with a knowledge base that spans the entire company. Pick one domain with clear boundaries and measurable business value, such as internal documentation for a support team, a product catalog for a sales chatbot, or support ticket history to speed up agent responses. Each of these has enough volume to be genuinely useful, while staying narrow enough that you can clearly evaluate retrieval quality before expanding scope.
Closing Thoughts
RAG isn't a silver bullet, but for most businesses that want AI to answer questions grounded in their own data, it's a far more practical and affordable path than training a custom model from scratch. The key is understanding the scale of your data first: if it's still small, long context with prompt caching may already be enough. Once it grows large, investing in a proper RAG pipeline, complete with good chunking, hybrid search, and reranking, genuinely pays off.
References
- Anthropic Engineering, "Introducing Contextual Retrieval"
- Encore, "You Probably Don't Need a Vector Database"
- Claude Cookbook, "Enhancing RAG with Contextual Retrieval"
