1. The Problem Isn't Necessarily the Model

After launching a knowledge base AI, many enterprises immediately blame the model for not being "smart enough." But in most real-world projects, the real issue lies in the RAG pipeline: document parsing loses structure, chunking strategy is crude, retrieval recall is insufficient, reranking is missing, and prompts don't constrain the model to cited sources. The model is just the last mile — if the context it receives is flawed, the answer will be too.

2. The Five Most Common Pitfalls

2.1 Chunking Too Aggressively

If each chunk is just one or two sentences, the model ends up with context-free fragments. For example, the phrase "requires approval" from a policy document means nothing without knowing whether it refers to procurement, reimbursement, or contracts. The fix is to split along heading and semantic boundaries and always preserve parent headings.

2.2 Chunking Too Generously

Overly long chunks mix multiple topics together. A chunk may appear relevant to a query but contains so much noise that the model's summary becomes vague and unreliable. Each chunk should carry exactly one clear topic.

2.3 Relying on Vector Search Alone

Vector search excels at semantic similarity but can be inconsistent with serial numbers, product models, customer names, and abbreviations. Enterprise queries often depend on exact keyword matches, so combining vector retrieval with keyword search is essential.

2.4 No Reranking Step

Initial recall only means "potentially relevant," not "best fit for this answer." Without reranking, the model may work with high-ranking but informationally incomplete chunks. Reranking significantly improves the quality of the evidence the model uses to answer.

2.5 No Refusal Mechanism

When the knowledge base lacks relevant material, the system should refuse to answer or prompt the user to supplement. If the prompt doesn't explicitly require "answer based solely on retrieved content," the model may fall back on its general knowledge to fill in the gaps — producing exactly the kind of hallucination enterprises fear most.

3. Diagnostic Approach

Don't just evaluate the final answer. Break every QA interaction into three stages: how the user's question was rewritten, what chunks were retrieved, and which chunks the model actually used to generate its answer. Open these three logs and most problems become immediately traceable.

  • If retrieved results are irrelevant: Optimize chunking, Embedding, keyword recall, and query rewriting.
  • If retrieved results are relevant but the answer is wrong: Improve prompts, citation constraints, and context ordering.
  • If the answer is correct but users don't trust it: Add source citations, document timestamps, and clickable references.

4. Optimization Priorities

Fix document structure first, then retrieval recall, then add reranking, and finally optimize the generation prompt. Many teams jump straight to tuning the prompt, only to make wrong answers read more fluently. RAG is fundamentally an information supply chain — if the upstream is unstable, no amount of downstream polish will save you.

5. Conclusion

A good enterprise AI doesn't come from plugging in a model — it comes from continuously refining the knowledge pipeline. Get chunking, recall, reranking, citations, and refusal mechanisms right, and RAG can evolve from "able to answer" to "answering accurately, traceably, and reliably."