Main Page

RAG, Measured

The claim

Reranking put the right passage first in 42% of 1,310 questions about research papers.

Why I built it

Interviewers ask how to get a long PDF into a language model. My only answer was one lab notebook with no test set. I wanted to watch each step work, and to measure which step matters.

The page walks one research paper from its PDF to an answer. You can change the chunk size, type a question and rerank the results. The models run in your browser, so the page needs no server and no key.

What I built

Five chapters. The first two measure the context window itself. The third runs the pipeline live on one paper. The fourth scores seven methods on two public benchmarks, QASPER and FinanceBench. The fifth prices pasting, caching and retrieving.

Chapter 4 of the walkthrough. Rows for seven retrieval methods and the whole document. Each row has a blue bar for how often the evidence was in the top five results and a grey bar for the tokens sent. Reranked paragraphs: 78% and 645 tokens. Fixed 2,000-character chunks: 82% and 1,782 tokens. The whole document: 4,877 tokens. Chapter 4 of the walkthrough. Rows for seven retrieval methods and the whole document. Each row has a blue bar for how often the evidence was in the top five results and a grey bar for the tokens sent. Reranked paragraphs: 78% and 645 tokens. Fixed 2,000-character chunks: 82% and 1,782 tokens. The whole document: 4,877 tokens.
Chapter 4 on the page. Left, how often the evidence was in the top five results. Right, the tokens those results add to a request.

Decisions

  • The same runtime everywhere. The benchmark runs the browser’s WebAssembly runtime in Node. Node’s own runtime was five times faster, but its vectors differed from the browser’s.
  • Answers written ahead of time. A live model call would need a server, a key and abuse limits. Claude Haiku 5.5 answered each preset question once, and the page labels the answers as pre-written.
  • One hit rule, fixed before the first run. A chunk counts when it shares most of its five-word runs with the evidence. Tuning the rule after seeing results would have flattered one method.

How I tested it

119 tests cover the chunkers, search, the hit rule, the spend guard and the page. A browser test compares embeddings and rerank scores against Node. The difference is zero. A spend guard capped all model calls at one dollar. They cost $0.32.

What broke

  • Batching changed the vectors. The quantised model scaled numbers across a whole batch, so the same sentence moved by up to 0.009 depending on its neighbours. Each text now runs alone.
  • The first benchmark measured the wrong runtime. The browser and Node disagreed by up to 0.021. I reran everything on the browser’s runtime.
  • Long documents surprised me. Haiku found a hidden code 138 times in 150. Every miss came when the code sat alone. With four similar codes nearby, it never missed.

What next

Test contextual retrieval, which adds a short summary to each chunk before embedding. Try a reranker trained on financial text, since this one did no better than plain search on company filings.