PimSpace

Episode 2 of 12

Why RAG Gets It Wrong: Check Retrieval Before You Blame the Model

Watch on YouTube

Presented by an AI digital human: George's own likeness and cloned voice.

Your knowledge base is connected, but the AI is still wrong? The problem is often parsing, chunking or search, not the model. How I build retrieval in steps, and the two things to evaluate separately: did it find the right material, and is the answer faithful to it.

Key points

  • RAG retrieves documents first, then the model answers from what was retrieved; most failures happen before or during retrieval.
  • Typical causes: a PDF parsed wrongly, a table that lost its structure, chunking that cut the context in half, or a product-code question answered by semantic search alone.
  • Retrieval in steps: clean up the documents and their permissions, combine keyword and vector search, and re-rank results when needed.
  • Make every answer cite its sources.
  • Evaluate with a set of real questions, and check two things separately: did it find the right material, and is the answer faithful to it.

Transcript

You've connected AI to your company knowledge base, but the answers are still wrong. The first thought is the model isn't smart enough. Not necessarily. RAG means you retrieve documents first, then the model answers from what was retrieved. What breaks most often is before retrieval and the retrieval itself. The PDF was parsed wrong. A table lost its structure. Chunking cut the context in half. Or the user asks for a product code and the system only runs a semantic similarity search. Give the model the wrong material and however smart it is, it will struggle to answer correctly. I split retrieval into steps. Clean up the documents and their permissions. Combine keyword and vector search. Re-rank the results when needed. And finally, make every answer cite its sources. Even more important, build a set of real questions to evaluate and check two things separately. Did it find the right material and is the answer faithful to that material? The key to RAG isn't just the vector database. It's an engineering pipeline for data quality, retrieval, and answer checking. Check your retrieval recall first. Then talk about changing models.