The question I get most often from teams building retrieval-augmented generation (RAG) is some version of: “Which model should we use?” — and they almost always mean the generation LLM. They have picked the biggest flagship model they can afford, wired it to a vector database, and are surprised the answers are still wrong.
Here is the reframing I have learned to share first: a RAG system is not one model. It is at least three model decisions stacked on top of each other — the embedding model that decides what gets found, the (optional) reranker that decides what survives, and the generation model that decides how the answer reads. The flagship LLM sits at the end of that chain. If retrieval hands it the wrong three paragraphs, no amount of reasoning horsepower saves the answer. Garbage in, confident garbage out.
This is a practitioner’s decision framework for choosing every model in that stack. It is deliberately vendor-neutral: I name specific models only as time-stamped examples, because the lineup rotates every quarter and the method does not. The goal is that you finish this able to answer “which RAG model?” for your own use case — with a defensible reason, not a leaderboard screenshot.
1. First, name your use case (they are not the same)
“RAG” covers wildly different jobs, and each one has a different dominant constraint — the one failure mode that, if you get it wrong, sinks the project. Pick yours before you touch a model list.
| Use case | Dominant constraint | What “wrong model” looks like |
|---|---|---|
| Internal knowledge / helpdesk search | Recall (find the one right doc) | The answer exists but retrieval never surfaced it |
| Customer-facing chatbot | Faithfulness + latency | Fluent, fast, and confidently made-up |
| Coding / API docs assistant | Exact-term match (BM25 territory) | Semantic search misses error TS-999 |
| Legal / financial analysis | Grounding + citations | Right gist, wrong clause, no source |
| Regulated (health, finance) | Auditability + refusal | Answers when it should have said “insufficient evidence” |
| Multilingual / cross-lingual | Embedding language coverage | Query in German, doc in English, no match |
| Real-time / voice agents | End-to-end latency | Correct answer arrives 4 seconds too late |
| Multimodal (PDFs, tables, images) | Parsing + modality-aware retrieval | The number lived in a table the parser flattened |
The rule: you optimize the layer that owns your dominant constraint, and you buy “good enough” for the rest. A helpdesk-search team that spends its energy A/B-testing flagship generators while running a weak embedding model is optimizing the wrong layer.
2. Do you even need different models? Sometimes no.
Before adding models, check whether you need RAG at all. Anthropic’s own guidance is refreshingly blunt: if your entire knowledge base fits in roughly 200,000 tokens (about 500 pages), you can often just put the whole thing in the prompt and skip retrieval — prompt caching makes it cheap and fast. RAG is what you reach for when the corpus outgrows the context window.
And when you do need RAG, “different models” is a spectrum, not a mandate:
- Keep it simple when the corpus is small, single-domain, single-language, and low-stakes: one strong general-purpose embedding model + one solid generation model, no reranker. Adding a reranker and a router here just buys you latency and a bigger bill.
- Specialize when a specific constraint dominates: a domain/long-context embedding model for dense technical corpora, a reranker for precision-critical answers, a cheaper generator for the 70% of easy queries.
The honest limitation: every model you add is another thing to evaluate, version, and pay for. Complexity is a cost, not a feature. Add a layer only when a measured failure demands it.
3. The framework: choose each layer by its job
Here is the repeatable method. Work it top-to-bottom, because each layer constrains the next.
Step 0 Define the failure you can't tolerate -> your dominant constraint
Step 1 Choose the embedding/retriever -> owns RECALL
Step 2 Decide if you need a reranker -> owns PRECISION
Step 3 Choose the generation model -> owns FAITHFULNESS + COST
Step 4 Measure end-to-end on YOUR data -> RAGAS / eval harness
Step 5 Route, don't over-provision -> cheap model for easy queries
Step 1 — Embedding/retriever: the layer that decides what gets found
This is the highest-impact choice and the one teams under-invest in most. Public benchmarks are your starting map, not the territory: the MTEB leaderboard ranks embedding models across dozens of tasks, and BEIR (the retrieval benchmark behind much of it, Thakur et al., NeurIPS 2021) measures zero-shot retrieval across 15+ domains — which is exactly the “will it work on data it wasn’t trained on?” question you care about.
| Embedding tier | Pick it when | Example models (as of writing, 2026) |
|---|---|---|
| Open / small | Cost and on-prem/privacy dominate; corpus is general | all-MiniLM, bge-small, e5-small |
| Standard general | Default starting point for most apps | OpenAI text-embedding-3-large, Cohere Embed v3, bge-large |
| Premium / long-context / domain | Dense technical, legal, or long documents | Voyage, Gemini embeddings, NV-Embed |
| Multilingual | Query language ≠ document language | bge-m3, Cohere multilingual, multilingual-e5 |
Three differentiators that matter more than the leaderboard rank:
- Hybrid beats pure vectors for exact terms. Dense embeddings miss unique identifiers (
TS-999, a case number, a SKU). Combining embeddings with keyword search (BM25) via rank fusion is the single most reliable retrieval upgrade — Anthropic found embeddings + BM25 beats embeddings alone in every configuration they tested. - Context beats raw model quality. Prepending a one-sentence, LLM-generated description of where each chunk came from before embedding (“Contextual Embeddings”) cut Anthropic’s top-20 retrieval failure rate by 35% (5.7% → 3.7%) — a bigger gain than most model swaps.
- Dimensions and cost are a tradeoff, not a ranking. Higher-dimensional embeddings cost more to store and search. Test whether the recall gain justifies the index size on your corpus.
The limitation to respect: a leaderboard score on Wikipedia-style data tells you almost nothing about your niche corpus. Re-rank the candidates on your own queries — the next section is how.
Step 2 — Reranker: precision insurance, when you can afford the latency
A reranker (a cross-encoder that jointly scores query + document) takes the top ~100–150 candidates from retrieval and re-orders them so the best few reach the model. It is the highest-ROI add-on for precision-critical use cases.
The measured case for it: on top of contextual embeddings + BM25, adding a reranking step took Anthropic’s retrieval failure rate from 5.7% down to 1.9% — a 67% reduction.
| Add a reranker when… | Skip it when… |
|---|---|
| Wrong answers are expensive (legal, medical, finance) | You’re latency-bound (voice, real-time agents) |
| Retrieval returns many near-duplicates of mixed relevance | Corpus is tiny and top-k is already clean |
| You retrieve a wide net (top-100+) then need the best 5–20 | Every millisecond and every extra call hurts the budget |
The tradeoff is honest and unavoidable: a reranker adds one more model call and a little latency for a lot of precision. Rerank a wide candidate set (retrieve 100–150, keep 20) — that is where it earns its keep.
Step 3 — Generation model: faithfulness first, then cost
Only now do you pick the LLM everyone wanted to start with. For RAG, the generator’s job is narrow: stay faithful to the retrieved context and cite it — not to be the world’s best reasoner.
| Generation tier | Pick it when | Watch out for |
|---|---|---|
| Budget / small | High-volume, well-retrieved, simple Q&A | Weaker at refusing when evidence is thin |
| Standard | Default for most customer-facing RAG | The 80% case; verify faithfulness on edge queries |
| Flagship / reasoning | Multi-hop synthesis, ambiguous or conflicting sources | Cost and latency; often overkill for lookup |
Two generation-layer traps that are really retrieval problems in disguise:
- “Lost in the middle.” Models attend most to the start and end of their context and can miss facts buried in the middle of a long prompt (Liu et al., 2023). Stuffing 50 chunks in hurts; a reranker that puts the best 5–20 chunks up front helps more than a bigger model.
- Context window ≠ recall. A million-token window is not a license to skip retrieval. More chunks raise the odds the answer is present but also add distractors — Anthropic found 20 well-chosen chunks beat 5 or 10, but returns diminish past that. Retrieve precisely, don’t dump.
Step 4 — Measure end-to-end on your own data (this is the whole ballgame)
Every layer choice above is a hypothesis until you measure it on your queries. Use a RAG evaluation harness — RAGAS is the widely used open-source one (pip install ragas) — and track these four metrics as a system:
| Metric | Question it answers | Which layer it blames |
|---|---|---|
| Context recall | Did retrieval find the needed evidence? | Embedding / retriever |
| Context precision | Is the retrieved context mostly relevant? | Reranker / retriever |
| Faithfulness | Does the answer stick to the context? | Generator |
| Answer relevance | Does it actually address the question? | Generator + prompt |
The diagnostic power is in the split: low context recall is an embedding problem no generator can fix; high recall but low faithfulness is a generator (or prompt) problem no embedding upgrade can fix. Measure the layers separately or you will “fix” the wrong one.
Step 5 — Route, don’t over-provision
Most production traffic is easy. Sending every query to the flagship generator is the most common source of RAG overspend. Query routing sends simple queries to a cheap model and only the hard ones to the expensive model. The RouteLLM framework (Ong et al., 2024) reports routers that cut cost by up to 85% while keeping 95% of GPT-4-level quality on MT-Bench (GPT-4 being the paper’s 2024 reference model). The same idea applies to embeddings and rerankers: reserve the premium tier for the queries that need it.
4. A grounded before/after: the contextual-retrieval walkthrough
You do not have to take the framework on faith — Anthropic published the numbers for exactly this layer-by-layer approach on a mixed corpus (codebases, papers, fiction), measured as top-20 retrieval failure rate (source):
| Configuration | Retrieval failure rate | Reduction vs. baseline |
|---|---|---|
| Embeddings only (baseline) | 5.7% | — |
| + Contextual Embeddings | 3.7% | −35% |
| + Contextual BM25 (hybrid) | 2.9% | −49% |
| + Reranking | 1.9% | −67% |
Read it as the framework in miniature: they did not reach for a bigger generation model to fix bad answers. They fixed the retrieval layer — better chunk context, hybrid search, then a reranker — and cut failures by two-thirds. That is the whole thesis. The generator was never the bottleneck.
5. Expert tips (the shortcuts and the gotchas)
- Optimize retrieval before you upgrade the generator. It is cheaper and moves the metric more, almost every time.
- Always run hybrid (dense + BM25) unless you’ve measured that you don’t need it. Exact-match failures are silent and brutal in technical corpora.
- Chunking is a model decision in disguise. Chunk size, overlap, and whether you prepend context change retrieval quality more than swapping embedding models. Test it first.
- Benchmarks pick your shortlist; your data picks the winner. Use MTEB/BEIR to get to 2–3 candidates, then decide on your own eval set.
- Right-size the generator to the query, not the corpus. A big context window is not a reason to use a flagship model for a one-line lookup.
- Treat “insufficient evidence” as a feature. In regulated use cases, a model that refuses when retrieval is thin beats one that always answers.
- Version and re-run your eval harness on every model swap. “It felt better” is not a metric. Context recall is.
6. Build it yourself: three projects
The fastest way to internalize this framework is to run it. Each project derives from a layer above and produces a machine-checkable signal, not a vibe.
Project 1 — Beginner: stand up a RAG eval harness (½ day)
- Goal: Get objective RAGAS scores on your own docs so future model choices are measured, not guessed.
- Prerequisites: Python, an API key, ~20 question/answer pairs from your domain.
- Steps: (1)
pip install ragas; (2) index a small doc set with any default embedding model; (3) run a basic retrieve-then-generate loop; (4) score it with RAGAS on context recall, context precision, faithfulness, and answer relevance following the RAGAS quickstart. - Success signal: You have a baseline number for all four metrics you can re-run on demand.
- Time: 3–4 hours. Stretch: Add 20 adversarial “answer isn’t in the corpus” questions and confirm faithfulness catches hallucinations.
Project 2 — Intermediate: A/B two embedding models + add a reranker (1 day)
- Goal: Prove (or disprove) that retrieval upgrades beat generator upgrades on your corpus.
- Prerequisites: Project 1’s harness.
- Steps: (1) Swap in a second embedding model and compare context recall; (2) add BM25 and fuse; (3) add a reranker over the top-100 candidates. Use the runnable reranking and end-to-end evaluation notebooks in
NirDiamant/RAG_Techniquesas scaffolding. - Success signal: A table like Section 4 — failure rate dropping as you add hybrid + reranking, measured on your data.
- Time: ~1 day. Stretch: Add contextual chunk headers and measure the recall lift.
Project 3 — Advanced: route queries by difficulty (1–2 days)
- Goal: Cut generation cost without dropping quality by sending easy queries to a cheap model.
- Prerequisites: Projects 1–2 and a labeled easy/hard query sample.
- Steps: (1) Install RouteLLM (
pip install "routellm[serve,eval]"); (2) set a strong/weak model pair; (3) calibrate the threshold on your own query distribution; (4) measure cost and quality vs. always-flagship. - Success signal: Measurable cost reduction (RouteLLM reports up to 85%) at ≥95% of flagship quality on your eval set.
- Time: 1–2 days. Stretch: Extend routing to the embedding/reranker layers — premium retrieval only for hard queries.
Start here
Don’t start by picking a model. Start by running Project 1 this week: stand up the eval harness, get your four baseline numbers, and find out which layer is actually failing. Then choose the model that fixes that layer — and prove it moved the metric. That is how you identify the best RAG model for your use case: not from a leaderboard, but from your own measured failure.
References
| # | Source | Role |
|---|---|---|
| 1 | Anthropic — Introducing Contextual Retrieval (Sep 2024) | Primary — measured retrieval failure-rate reductions |
| 2 | MTEB Leaderboard | Measurement — embedding model rankings |
| 3 | BEIR: A Heterogeneous Benchmark for Zero-shot IR — Thakur et al., NeurIPS 2021 · repo | Measurement — zero-shot retrieval benchmark |
| 4 | Lost in the Middle — Liu et al., 2023 | Primary — long-context attention degradation |
| 5 | RouteLLM · paper — Ong et al., 2024 | Primary — cost-quality routing results |
| 6 | RAGAS docs | Primary — RAG evaluation metrics |
| 7 | NirDiamant/RAG_Techniques | Synthesis — runnable technique notebooks |
Model names are time-stamped examples as of 2026-07; tiers and capabilities are the durable framework. Retrieval-quality percentages are from Anthropic’s published experiments on their mixed corpus and will vary on your data — which is exactly why Step 4 exists.