Agentic RAG Knowledge Assistant
Evidence-aware retrieval-augmented generation system with multi-pass planning, atomic claim auditing, and bounded retries.
Executive Summary // 30-Second Recruiter Brief
Standard feed-forward RAG architectures retrieve top-k chunks and immediately generate answers without verifying factual sufficiency. When queries are out-of-scope or contain false premises, naive generators fabricate plausible answers from parametric memory.
Built a closed-loop Agentic RAG pipeline in Python combining dense vector search and BM25 keyword matching via Reciprocal Rank Fusion (RRF, k=60). Implemented an explicit Evidence Auditor for atomic claim verification and dynamic PDF ingestion with SHA-256 Content-Addressable Storage (CAS) vector caching.
1. Problem Formulation & Motivation
Standard feed-forward RAG architectures retrieve top-k chunks and immediately generate answers without verifying factual sufficiency. When queries are out-of-scope or contain false premises, naive generators fabricate plausible answers from parametric memory.
To experimentally investigate whether an autonomous, closed-loop loop (Plan → Retrieve → Audit → Bounded Retry → Generate/Refusal) can eliminate unsupported hallucinations, while measuring the exact latency and accuracy tradeoffs.
2. Approach & System Solution
Built a closed-loop Agentic RAG pipeline in Python combining dense vector search and BM25 keyword matching via Reciprocal Rank Fusion (RRF, k=60). Implemented an explicit Evidence Auditor for atomic claim verification and dynamic PDF ingestion with SHA-256 Content-Addressable Storage (CAS) vector caching.
3. Architecture & Execution Pipeline
Closed-loop multi-pass architecture featuring query decomposition, hybrid lexical/vector search with RRF reranking, atomic claim auditing, and dynamic document caching.
Agentic RAG Knowledge Assistant — Topology
Interactive Visual Data Pipeline · Click any stage to inspect execution state
Input natural language query received by the orchestrator.
Execution Data Flow Sequence
6 Steps- 1User submits a query; Query Planner determines complexity and generates sub-queries.
- 2Hybrid retriever queries BM25 sparse index and NumPy vector store concurrently.
- 3Reciprocal Rank Fusion (RRF, k=60) merges sparse and dense rankings into top-K candidates.
- 4Evidence Auditor checks atomic claims, verifies premise validity, and diagnoses retrieval gaps.
- 5If evidence is insufficient and retry count < 2, the loop reformulates search queries for a targeted second pass.
- 6If sufficient, the generator synthesizes answers with citations; if definitively absent, the system outputs an explicit structured refusal.
4. Engineering Decisions & Trade-Offs
Reciprocal Rank Fusion (RRF k=60) Hybrid Retrieval
Explicit Evidence Auditing vs Immediate Generation
5. Implementation Snippet
def _reciprocal_rank_fusion(
self,
dense_results: List[SearchResult],
sparse_results: List[SearchResult],
top_k: int
) -> List[SearchResult]:
"""Combines dense and sparse results using Reciprocal Rank Fusion (RRF)."""
rrf_scores: Dict[str, float] = {}
chunk_map: Dict[str, Chunk] = {}
for rank, res in enumerate(dense_results):
cid = res.chunk.chunk_id
rrf_scores[cid] = rrf_scores.get(cid, 0.0) + (1.0 / (self.rrf_k + rank + 1))
chunk_map[cid] = res.chunk
for rank, res in enumerate(sparse_results):
cid = res.chunk.chunk_id
rrf_scores[cid] = rrf_scores.get(cid, 0.0) + (1.0 / (self.rrf_k + rank + 1))
chunk_map[cid] = res.chunk
sorted_chunks = sorted(rrf_scores.items(), key=lambda item: item[1], reverse=True)[:top_k]
return [
SearchResult(chunk=chunk_map[cid], score=score, retrieval_mode="hybrid")
for cid, score in sorted_chunks
]6. Evaluation & Measured Results
Evaluated across 8 system configurations on a 64-question curated benchmark (32 Dev / 32 Held-Out Test). Full Agentic system achieved 0% hallucination on unanswerables, 55.2% citation precision, and 52.1% evidence recall.
0/4 hallucinations on adversarial unanswerable questions (vs 4/4 for Hybrid RAG baseline)
Correctly identified and refused all 4 out-of-scope test questions
Human-reviewed accuracy on 24 answerable test questions under conservative evidence gating
87 passing unit & integration tests running with mocked API fixtures (0 external token cost)
7. Limitations & Production Considerations
- •Auditing loop introduces significant latency overhead (~17.7s avg for full agentic loop vs ~1.86s for single-pass baseline).
- •Conservative evidence thresholds produce higher false refusal rates (12.5%) on questions with sparse source documentation.
Related Technical Essays
Hybrid Retrieval Systems in Production: Combining BM25, Dense Embeddings, and Reciprocal Rank Fusion
Pure vector search frequently fails on exact keyword identifiers and domain jargon, while lexical BM25 misses semantic paraphrasing. This article breaks down the mathematical mechanics and implementation of Reciprocal Rank Fusion (RRF) for production RAG pipelines.
Evaluating Hallucination and Citation Faithfulness in Retrieval-Augmented Generation
Building a RAG pipeline is straightforward; systematically evaluating its faithfulness and hallucination rate is where most engineering teams struggle. This article details automated evaluation frameworks, grounded assertion metrics, and cost-effective offline testing with mocked embeddings.