Skip to contentCIELONT SOFTWARE PRIVATE LIMITED logoCIELONTSOFTWARE PRIVATE LIMITED
AI / ML

10 Lessons from Deploying RAG Systems to Production

Real-world lessons from building and shipping retrieval-augmented generation systems for enterprise clients — what nobody tells you before you start.

C
Cielont Engineering
Apr 10, 2026
12 min read

We've shipped RAG systems for enterprise clients across legal, finance, and operations — corpora ranging from 50,000 to 5 million documents. The gap between a compelling demo and a production-grade system is enormous. Here are the ten lessons that surprised us most.

Lesson 1: Chunking strategy matters more than model choice

Most teams spend weeks choosing between GPT-4o and Claude 3.5. They should spend that time on chunking. How you split documents into retrievable units determines retrieval accuracy more than anything else.

Fixed-size chunking (e.g., 512 tokens) is almost always wrong. Sentence-boundary chunking is better. Semantic chunking — splitting at topic boundaries detected by a lightweight model — is better still. For legal documents, section-based chunking that preserves clause numbering outperforms all of them.

Lesson 2: Embed the question, not just the document

Standard RAG embeds the document chunk and the user query separately, then measures cosine similarity. The problem: documents contain statements ("The widget must weigh under 5kg") and queries ask questions ("What is the maximum weight?"). These are semantically dissimilar even though they're clearly related.

The fix: generate hypothetical answers to your expected query types and embed those alongside the document chunks. This is called HyDE (Hypothetical Document Embeddings) and routinely improves recall by 15–30% in our evaluations.

Lesson 3: Retrieval is a pipeline, not a lookup

A production RAG pipeline has at least four stages: initial retrieval (top-K by vector similarity), metadata filtering (date range, document type, author), cross-encoder re-ranking (a separate model scores each retrieved chunk against the query), and context assembly (deciding which chunks to include given token constraints).

Teams that skip re-ranking ship systems where the most verbose document always wins, not the most relevant one.

Lesson 4: Your eval set is your most valuable asset

You cannot improve what you cannot measure. Build an evaluation set before you write a line of retrieval code. A good eval set has 200+ question-answer pairs with references to specific source documents. This is what you run after every model or chunking change to know whether things got better or worse.

We've had clients spend three months tuning a system and making it worse because they had no eval loop. Don't be those clients.

Lesson 5: Context window size creates false confidence

128k context windows don't mean "just stuff everything in." Long-context models are worse at needle-in-a-haystack tasks. Information buried in the middle of a 100k token context is frequently ignored. Keep retrieved context tight — we rarely exceed 6,000 tokens of retrieved content regardless of model context size.

Lesson 6: Structured output is mandatory for downstream integrations

If anything downstream processes the LLM output — a database write, a UI component, an email trigger — force JSON output via tool calling or structured output APIs. Parsing free-form text with regex in production is a reliability disaster waiting to happen.

Lesson 7: Cost scales with retrieval, not generation

The surprise: in a well-optimised RAG system, embedding API costs dwarf generation costs. A system that re-embeds queries at runtime, does multiple retrieval passes, and uses a re-ranker can cost more per query than a simple GPT-4o generation call.

Cache embeddings aggressively. Deduplicate queries before they hit the embedding API. Monitor cost per query from day one.

Lesson 8: Hallucination requires a retrieval fix, not a prompt fix

When an RAG system hallucinates, the instinct is to add "only use information from the retrieved documents" to the system prompt. This rarely works. The actual fix: improve retrieval so the right information is always available. If the model can't find relevant information, it will confabulate — no amount of prompting prevents that reliably.

Lesson 9: Source citations kill trust problems

Enterprise users don't trust AI outputs without verification paths. Inline citations — "According to Policy v3.2, Section 4.1..." — with clickable links to the source document transform the trust dynamic. Users go from skeptical to engaged. We include citation generation in every production RAG system regardless of client requests.

Lesson 10: The index is a product

Your vector index needs the same engineering discipline as any production system: versioning, rollback capability, incremental update support, and monitoring. An index that has no update strategy becomes stale within weeks. Stale retrieval means confident answers about outdated information — the worst possible failure mode for an enterprise system.

RAG systems that work in demos fail in production because of retrieval quality, not generation quality. Fix the pipeline before you agonise over the model. The model is the last 20% of the problem.

Work with us

Ready to build something great?

From custom software to AI systems — we ship production-grade products on time.