Retrieval-augmented generation is the easiest impressive demo in AI. Point a model at some documents, wire up a vector search, ask a question, and watch it answer with sources. You can put that together in an afternoon and get a room full of nods.
Then you hand it to real users, with real documents, and it starts coming apart in ways the demo never hinted at. None of the failures below are exotic. They’re just the parts that don’t fit in a five-minute demo, so nobody shows them to you. This is the list I wish I’d had the first time.
The demo is a rigged sample
A demo runs on a handful of clean documents you picked, questions you picked, and a user who is also you. Production has thousands of documents you didn’t pick, questions from people who don’t know your internal vocabulary, and near-duplicate files that quietly contradict each other. The model was rarely the risky part. The risk is in everything you feed it.
So when someone shows me a RAG demo, I don’t ask about the model. I ask how many documents, how messy they are, who’s actually going to be asking, and what should happen when the answer isn’t in the corpus at all. That’s the real project, and it’s the part the demo skips.
Retrieval decides everything
If the retrieval step hands the model the wrong passages, no amount of prompt wording rescues it. This is where most of the real work is, and it’s also where the least demo attention goes.
Chunking is the first thing to get wrong. Split documents too small and you cut a passage off from the context it needs; too large and the relevant sentence drowns in noise while you burn your token budget. There’s no setting that’s right for everyone, because it depends on your documents, which is why you can’t lift a chunking recipe off a blog post and expect it to hold on yours.
Plain vector similarity is also weaker than the demos let on. Embeddings are good at “these mean roughly the same thing” and bad at exact matches like part numbers, error codes, names, and dates. What usually works is hybrid retrieval: run keyword search next to vector search, merge the results, then re-rank the merged set with a model that reads the actual query against each candidate. Retrieving broad and then re-ranking does more for quality than reaching for a fancier embedding model.
The index goes stale immediately
A demo indexes its documents once. Real documents move. Policies get revised, prices change, old versions get deleted. If your index doesn’t follow those changes, the system will cite last quarter’s answer with total confidence, and it’ll look every bit as convincing as the correct one.
So freshness is an actual engineering problem, not a later cleanup: detecting that a source changed, re-embedding only what changed, and dropping what’s gone. Work it out early. Bolting incremental re-indexing onto a system that assumed a frozen corpus is a miserable retrofit.
Permissions live in the retrieval layer
This is the one that gets people fired, so I’ll be blunt. If your corpus holds documents that different users are allowed to see, the access check has to happen at retrieval time, filtered per user, before anything reaches the model. Retrieving across everything and trusting the prompt to keep a secret does not work. Ask the model nicely to ignore the confidential passage and it will still cheerfully summarize it.
Add permissions at the end and what you’ve shipped is a data leak with a chat box on top. Every chunk needs to carry who can see it, and every query has to filter on the asker before it retrieves anything.
Without evals you’re guessing
Here’s the uncomfortable one: with no evaluation, you can’t actually tell whether a change helped or hurt. You’ll adjust a prompt, spot-check a few questions, decide it feels better, and ship a regression you won’t notice for a month.
You need a real evaluation set: questions with known-good answers, taken from how the thing is actually used, plus a way to score new answers against them without doing it by hand every time. It doesn’t have to be fancy to start. A hundred representative questions and an honest rubric already beat going on feel. Build it early, because everything downstream (chunking, re-ranking, model choice) is a guess until you can measure it.
Latency and cost are product decisions
Retrieve, re-rank, then generate, and every stage costs time and money. The demo doesn’t notice because it runs once. Production runs all day, and users feel every extra second.
These shape the product; they aren’t something to optimize later. How much context can you afford per call? Would a smaller model with better retrieval be cheaper and still good enough? Can you cache the common questions? Decide while you’re designing, not after finance reads the first bill.
It still makes things up
Retrieval reduces hallucination. It doesn’t remove it. The model will still occasionally assert something the sources don’t support, or fuse two passages into a claim neither one makes.
Two things help. Make it cite: tie each claim back to a passage the user can open and check, so the system is auditable instead of a black box. And let it decline: when retrieval comes back empty or thin, the right answer is “I don’t have that,” not a smooth guess. A system that knows when to say nothing is worth more than one that always has something to say.
What I actually reach for
Keep it boring at the start. Hybrid retrieval and a re-ranker will beat an elaborate pipeline you can’t debug. Build the eval set before you tune anything. Put access control in the retrieval layer from the first commit rather than the last sprint. Make answers cite their sources, and let them refuse. And watch real queries once you’re live, because the questions people actually ask will reshape your chunking and your corpus more than any amount of planning up front.
None of this shows up in the demo, and all of it is the job. The distance between a RAG demo and a RAG system people trust with real work is the whole engagement, and it’s the part worth paying for.
If you’ve got a demo that wowed everyone and then stalled, that’s the gap I close.