02 Project
LLM ENGINEERING · INFORMATION RETRIEVAL · EVALUATION
RAG Benchmark
From Retrieval Quality to Grounded Answers
“How do retrieval strategies, context size, and language model selection affect the reliability and efficiency of a RAG system?”
Designed and implemented a reproducible benchmarking pipeline to evaluate sparse and dense retrieval, local language models, and their impact on answer correctness, source grounding, and response latency.
Finding Information vs. Answering Correctly
A reproducible document-grounded evaluation across retrieval strategies, chunking granularities, and local language models.
Finding information is not the same as answering correctly
Retrieval-Augmented Generation enables language models to answer questions using information from external documents.
However, the quality of the final answer depends on several interconnected decisions: how documents are segmented, which retrieval method is used, how much context is supplied, and which language model generates the response.
A system may retrieve passages that appear relevant but fail to contain the information required to answer a question. Similarly, an LLM may produce an answer that resembles the expected response without providing reliable evidence from the retrieved documents.
A reproducible document-grounded evaluation
The benchmark uses two NIST AI governance documents as its knowledge corpus, processed into structured text with source metadata and token-aware chunks of 256 and 512 tokens.
A curated evaluation dataset contains 100 synthetic questions covering lexical, semantic, factual, definition, reasoning, and multi-chunk information needs.
Each question is associated with manually reviewed relevant passages, allowing retrieval configurations to be compared against explicit relevance judgments. A subset of 50 questions is used for generation experiments.
Evaluating Retrieval and Generation Independently
Decoupling search quality from synthesis fidelity to diagnose where failures actually originate.
Sparse vs. Dense Passage Search
BM25 lexical matching compared against dense retrieval using BGE-small embeddings stored in Qdrant, tested across 256-token and 512-token chunks over the full 100-query benchmark.
Local Model Grounding & Context Size
The strongest retrieval configuration (Dense 512) supplies context to local Granite language models through Ollama, evaluating answer correctness, citation validity, and inference cost.
From PDF Documents to Evaluated Answers
Full end-to-end benchmarking pipeline illustrating document ingestion, decoupled retrieval testing, and model generation evaluation.
How Much Does Retrieval Strategy Matter?
Four retrieval configurations evaluated using the exact 100-query benchmark dataset.
| Method | Chunk Size | Precision@5 | Recall@5 | MRR | Mean Retrieval Latency |
|---|---|---|---|---|---|
| BM25 | 256 tokens | 20.80% | 50.87% | 0.5407 | 1.18 ms |
| BM25 | 512 tokens | 21.40% | 57.23% | 0.5598 | 0.73 ms |
| Dense (BGE-small) | 256 tokens | 25.00% | 58.15% | 0.6125 | 83.11 ms |
| Dense (BGE-small)Top Config | 512 tokens | 26.00% | 66.15% | 0.7165 | 74.79 ms |
Does a Larger Language Model Produce Better Grounded Answers?
Evaluating answer correctness vs. source citation grounding across 4 model configurations (50 questions each).
Granite 3B Top 3
Achieved the shortest mean response time (19.87s) and highest measured answer correctness (68%), making it agile for rapid prototyping.
Granite 8B Top 3
Achieved a 64% citation-based grounding rate (vs. only 4% for Granite 3B), demonstrating far superior instruction adherence to source attribution.
More Context ≠ Better
Increasing context from 3 to 5 chunks lengthened latency significantly (up to 78.57s) without improving measured answer correctness.
| Model | Context Chunks | Answer Correctness* | Citation Grounding | Mean Response Time |
|---|---|---|---|---|
| Granite 3B | Top 3 | 68% | 4% | 19.87 s |
| Granite 3B | Top 5 | 64% | 6% | 28.40 s |
| Granite 8BBest Grounding | Top 3 | 62% | 64% | 50.06 s |
| Granite 8B | Top 5 | 60% | 60% | 78.57 s |
Query-Level Failure Inspection
Why aggregate metrics are insufficient: a real case study comparing BM25 vs. Dense retrieval on the same question.
“What specific risk management measures are prescribed for autonomous decision-making systems under high-impact conditions?”
Lexical Mismatch & Distractor: Matches the isolated terms “decision-making” and “risk management”, but fails to capture high-impact regulatory prescriptions.
Semantic Understanding: Captures the conceptual relationship between “autonomous”, “high-impact”, and prescribed oversight controls despite varied vocabulary.
Technical Challenges & Experimental Limitations
Documenting the engineering hurdles and clear boundaries of the benchmark.
Evaluation Scope
Tested on 100 synthetic queries over two NIST AI governance documents. While controlled and reproducible, it does not claim universal generalization across all document formats, languages, or enterprise workloads.
Grounding Measurement
Grounding checks citation presence and labeled passage mapping. It does not establish exhaustive per-sentence human fact-checking of every individual claim in the generated text.
Document Processing
The pipeline ingests text-based PDFs. Optical Character Recognition (OCR) and multimodal inputs (tables, figures) are not included in the v1.0.0 benchmark scope.
Failures Beyond Aggregate Scores
High aggregate MRR and Recall can hide severe query-level hallucinations or missed evidence. Per-query failure diagnosis is mandatory to discover distractor chunks and lexical gaps.
Measuring Grounding Rigorously
Separating answer similarity from actual source grounding. Smaller models frequently hallucinate facts that match expected answers without relying on the provided context passages.
CPU Inference Latency Trade-Offs
Running local inference under CPU constraints revealed acute response-time penalties: 8B models took ~50–78s per query, defining real-world viability boundaries for local deployments.
Takeaways for Production RAG Engineering
Four core insights gained from running decoupled RAG evaluations.
Evaluate every stage independently: Retrieval quality does not automatically translate into trustworthy generated answers. Decoupling search testing from generation is the only way to diagnose errors accurately.
More context is not always better: Expanding context from 3 to 5 chunks lengthened latency by 43%–57% without yielding any measurable increase in answer correctness.
Attribution demands model scale: Granite 8B achieved 16× higher citation-based grounding compliance than Granite 3B, proving that smaller models struggle with strict source attribution formatting.
Inspect failures, don't just average metrics: Aggregate Precision and Recall averages routinely mask semantic drift and distractor passages that cause downstream answer hallucinations.
Source Code & Technical Write-up
Reproduce the benchmark experiments or read the comprehensive technical article.
RAG Benchmark v1.0.0
The public repository includes document preprocessing, sparse and dense retrieval scripts, per-query benchmark results, failure analysis logs, and local answer generation configs.