Ask My Docs: building a hybrid RAG pipeline that actually answers correctly

Most "chat with your PDF" projects are a thin wrapper: split the file into chunks, embed them, do a vector search, hand the top results to an LLM. It works for demos. It falls apart the moment a question needs an exact term, a table value, or a fact the embedding model quietly glossed over.

Ask My Docs was built to fix that gap — a document Q&A system where retrieval quality, not just the LLM, does the heavy lifting.

The problem with plain vector search

Dense embeddings are good at semantic similarity but bad at lexical precision. Ask a pure vector-search system "what is the F1-score in Table 3" and it will often retrieve a passage that talks about F1-scores in general, not the specific number in that specific table — because embeddings blur exact tokens into a general meaning.

That's the exact failure mode Ask My Docs is designed around.

Architecture

The pipeline has four stages:

Ingestion — PDFs are chunked and embedded.
Hybrid retrieval — the same query runs through two independent retrievers at once:
- BM25 (keyword/lexical search) — catches exact terms, numbers, names.
- ChromaDB (dense vector search) — catches semantic/paraphrased matches.
Reranking — a CrossEncoder re-scores the combined candidate set by reading the query and each chunk together, rather than comparing pre-computed embeddings. This is slower but far more accurate at judging actual relevance.
Generation — the reranked top chunks go to Llama 3.3-70B via Groq, which generates the final answer grounded in that context.

Tech stack

Retrieval: BM25 (lexical) + ChromaDB (dense vectors)
Reranking: CrossEncoder
LLM: Llama 3.3-70B, served through Groq for low-latency inference
Backend: FastAPI
Deployment: Dockerized, hosted on HuggingFace Spaces

Why hybrid retrieval, specifically

Neither retriever alone is reliable:

BM25 alone misses paraphrased questions ("cost of the plan" vs. "pricing" in the doc).
Vector search alone misses exact-match questions (numbers, IDs, proper nouns).

Running both and reranking the union gets the precision of keyword search and the recall of semantic search, without picking one at the cost of the other.

Why a CrossEncoder reranker

A bi-encoder (standard embedding model) scores a query and a document separately, then compares vectors — fast, but it never lets the query and the document actually interact. A CrossEncoder feeds the query and the candidate chunk into the model together, so it can weigh relevance directly. It's too slow to run on the whole corpus, which is exactly why it sits after retrieval narrows the field down — cheap methods first, expensive method last, only on the shortlist.

Validation

The project was stress-tested on a known, dense technical document — the "Attention Is All You Need" paper — specifically to check factual accuracy against a source where the ground truth is unambiguous, rather than only on friendly, low-density text.

What this project actually demonstrates

Understanding of retrieval trade-offs, not just calling a vector DB API.
A working two-stage retrieval design (retrieve → rerank), which is closer to how production RAG systems (e.g. search engines, enterprise Q&A tools) are actually built.
End-to-end ownership: ingestion, retrieval, generation, deployment — not just a notebook.

What's next

Add citation/source-attribution so answers point back to the exact chunk and page.
Add evaluation metrics (retrieval precision/recall on a labeled question set) instead of relying on spot-checks.
Extend chunking strategy to be table-aware, since table data is where plain chunking loses the most information.

Deployed on HuggingFace Spaces via Docker. Stack: FastAPI, BM25, ChromaDB, CrossEncoder, Llama 3.3-70B (Groq).