Your support team answers the same 40 questions every week. Your sales engineers spend hours hunting through product docs before customer calls. Your onboarding takes three weeks because new hires don't know where anything lives. A retrieval-augmented generation chatbot — one that actually knows your specific company's knowledge — can fix all three of these problems. The gap between a quick prototype and a system you'd stake your business on, though, is larger than most teams expect.
What "Production-Ready" Actually Means for a RAG System
A weekend demo that answers questions from a handful of PDFs is not production-ready. A production system handles messy, inconsistent source documents, maintains acceptable latency under concurrent load, returns a citable answer or an honest "I don't know" rather than a hallucinated one, and keeps sensitive content away from users who shouldn't see it. That's the bar. Everything that follows is in service of hitting it.
Stage 1: Ingestion — Garbage In, Garbage Out
The quality of your RAG system is determined almost entirely by the quality of what you put into it. Before you write a single line of retrieval code, spend time cleaning your source corpus.
- Identify your authoritative sources. Confluence pages, SharePoint wikis, Notion spaces, PDFs, Zendesk articles — pick the sources that are actually maintained and accurate. Stale content is worse than no content, because the model will confidently cite it.
- Normalize formats. Convert everything to plain text or Markdown before chunking. PDF extraction is notoriously unreliable — scanned PDFs need OCR, and two-column layouts produce garbled output. Budget time for this step.
- Tag documents with metadata at ingest time. Document title, source system, last-updated date, owner team, and access level should all be stored alongside the chunk. You'll use these for filtering and attribution later.
A connector layer that polls your source systems on a schedule (nightly is usually sufficient; real-time sync adds significant complexity) ensures the vector store stays current without manual effort.
Stage 2: Chunking — The Decision That Haunts You Later
Chunking is where most teams make their biggest early mistake. Split too coarsely and you retrieve paragraphs that contain the answer buried in irrelevant context, increasing noise and token cost. Split too finely and you lose the surrounding context the model needs to give a coherent answer.
A few approaches that work well in practice:
- Recursive character splitting with overlap (e.g., 512 tokens, 50-token overlap) is the safe default for prose-heavy docs. The overlap ensures a sentence that straddles a chunk boundary doesn't get lost.
- Semantic chunking — splitting on sentence-boundary similarity rather than character count — produces more coherent units for narrative content and is worth the added complexity for knowledge bases with long explanatory articles.
- Hierarchical chunking: store a small "child" chunk for embedding, but retrieve its parent section (or a summary) for the actual context window. This keeps embedding precision high while giving the LLM richer context.
Chunk size is not one-size-fits-all. FAQs chunk differently than technical runbooks. Build a simple evaluation harness early so you can measure retrieval quality as you tune.
Stage 3: Embeddings and the Vector Store
Your embedding model translates text into the dense vectors that power similarity search. The choice matters more than people assume — a model trained on general web text will perform worse than one fine-tuned on domain-specific language for technical or legal corpora.
| Embedding Model | Best For | Cost Model | Notes |
|---|---|---|---|
| OpenAI text-embedding-3-large | General enterprise content | Per-token API | High quality, no self-hosting needed |
| Cohere Embed v3 | Multilingual corpora | Per-token API | Strong for non-English knowledge bases |
| BGE-M3 (open-source) | Air-gapped / regulated environments | Self-hosted infra | Competitive quality, zero data egress |
| Domain fine-tuned model | Highly specialized vocabulary | Training + infra | Best retrieval, highest upfront cost |
For the vector store itself, Pinecone, Weaviate, Qdrant, and pgvector (if you're already on Postgres) are all reasonable choices at different scales. pgvector is worth considering for smaller corpora (under a few million chunks) because it eliminates a separate managed service and keeps your data inside your existing database perimeter.
Stage 4: Retrieval — Beyond Simple Cosine Similarity
Vanilla nearest-neighbor retrieval is a starting point, not a destination. Production systems layer several retrieval strategies:
- Hybrid search: combine dense vector search with BM25 sparse retrieval. Dense search finds semantically similar content; BM25 catches exact keyword matches. A simple re-ranking step (reciprocal rank fusion works well) merges both result lists. This alone often produces a measurable quality improvement over pure vector search.
- Query rewriting: run the user's question through a small LLM call to rephrase it as a declarative statement before embedding it. "What's the refund window?" becomes "The refund policy states a window of X days." This closes the lexical gap between questions and documents.
- Metadata filtering: use those ingestion-time tags. If a user is in the Finance department, filter to finance-relevant documents before running similarity search. This improves precision and enforces access control simultaneously.
- Re-ranking: pass your top-K retrieved chunks through a cross-encoder re-ranker (Cohere Rerank or an open-source equivalent) before sending them to the LLM. Cross-encoders are slower than embedding similarity but far more accurate at distinguishing relevance.
Stage 5: Generation, Guardrails and the Prompt Layer
The retrieval pipeline feeds context into the LLM prompt. How you structure that prompt determines whether you get crisp, attributed answers or confident hallucinations.
Core prompt principles for production RAG:
- Instruct the model to answer only from the provided context and to say "I don't have information on that" when the context is insufficient. This is the single most important instruction for reducing hallucinations.
- Include document titles or chunk IDs in the context block so the model can cite its sources. Surface those citations in the response.
- Set a strict token budget for context. Stuffing the maximum context window degrades quality and inflates cost — experiment to find the sweet spot, typically 3-8 chunks.
Beyond the prompt, add explicit guardrails:
- Input guardrails: detect and block prompt injection attempts, jailbreaks, and questions that fall completely outside the system's scope.
- Output guardrails: scan responses for PII before returning them to the user (a regex pass or a lightweight classifier is usually sufficient). Flag responses that lack a source citation for human review.
- Rate limiting and logging: log every query, retrieved chunk set, and response. This audit trail is non-negotiable for regulated industries and invaluable for debugging quality issues.
Evaluation: The Step Teams Skip
A RAG system without an evaluation harness is a system you can't improve systematically. Build a golden question set — 50 to 150 representative questions with known correct answers — before you go to production. Run this set against every change to the chunking strategy, embedding model, or prompt. Track retrieval recall (did the right chunks come back?), answer faithfulness (did the model stay grounded in the context?), and answer relevance (did the response actually address the question?). Frameworks like RAGAS make this quantifiable rather than subjective.
Frequently Asked Questions
How long does it take to build a production RAG chatbot?
A focused team can deliver a working prototype in two to four weeks. Getting to a production-grade system — with proper evaluation, guardrails, access control, and monitoring — typically takes eight to fourteen weeks depending on the complexity and cleanliness of the source knowledge base. Cutting corners on the ingestion and evaluation stages is the most common cause of projects that deliver a demo but never ship.
Which LLM should I use for the generation step?
For most enterprise RAG deployments, GPT-4o or Claude Sonnet-class models offer a good balance of quality and cost at the generation stage. If data residency or privacy is a constraint, open-source models like Llama 3 or Mistral deployed on your own infrastructure are viable. The LLM choice for generation matters less than retrieval quality — a great retrieval pipeline with a mid-tier LLM will outperform poor retrieval fed into the best available model.
How do I prevent the chatbot from leaking confidential documents to the wrong users?
Metadata-based access control applied at retrieval time is the most reliable pattern. Store each chunk's permitted user groups at ingest, and filter the vector search by those groups before returning results. Never rely solely on prompt instructions to keep confidential content away from unauthorized users — an adversarial query can often extract context the model was told to ignore.
What's the difference between RAG and fine-tuning for a company knowledge base?
RAG is almost always the right first choice for company knowledge bases. Fine-tuning bakes information into model weights, which means it can't be updated without retraining, you can't cite sources easily, and the model may still hallucinate confidently. RAG keeps knowledge in a retrieval store you can update in minutes, returns citable sources, and degrades predictably when it lacks information. Fine-tuning makes sense on top of RAG when you need the model to adopt a very specific tone or follow domain-specific reasoning patterns that retrieval alone can't address.
When you're ready to build this, Mexilet can help — explore our generative AI development and AI solutions.
If you're weighing whether to build this in-house or bring in a specialist team, the most pragmatic path is usually a small paid pilot sprint — two to three weeks, real documents, real evaluation metrics — before committing to a full build. The team at Mexilet Technologies runs exactly these kinds of scoped RAG pilots: you get a working proof-of-concept against your own knowledge base, a clear quality benchmark, and an honest assessment of what production will actually take. Reach out to discuss a pilot engagement before signing off on a longer project.