Building ragtrail: a search-path cache for agentic RAG
In 1873 an English engineer named Joseph Jagger hired six clerks to write down every number that came up on the roulette wheels at the Monte Carlo casino. His idea was simple. No wheel is perfect, and a slightly worn spindle or a table that isn't quite level makes some numbers come up more than others. Going through the clerks' notes, he found a wheel where nine numbers came up more often than they should, bet on them, and won a fortune in a few days. The details change depending on who tells it, but the story always keeps the same shape.
The casino eventually reacted, and one night it moved the wheels to different tables. Jagger kept betting on his numbers and started losing, until he realised the wheel in front of him was no longer his. He recognised his wheel by a scratch, found it across the room and started winning again. In the end the casino began taking the wheels apart and swapping their parts every night, and that was the end of his edge.
His method depended on two conditions. The notes had to be right, because a miscount would have had him betting on numbers with no bias at all. And the wheel had to stay the same one, because once it changed, the notes still looked valuable but no longer described anything. This post is about those two conditions applied to an agent.
An agent that answers questions over a set of documents has no memory. If you ask it the same thing twice, it searches twice, reads twice, and pays for both. ragtrail is my way of giving it Jagger's notes: a record of which sections worked last time, used only if what was written down was right and only if those sections haven't changed since.
More concretely, ragtrail is a cache for the part of agentic RAG that happens before the answer is written, which is the searching and reading the agent does to find the right sections. I tested it on a help center (WixQA, 200 questions) and on research papers (QASPER, 400 questions). When it found a similar question that had already been answered, the agent made about a third fewer calls, and some questions went from three or four calls down to one. Before reusing anything, ragtrail checks that each section is unchanged, and because of that it never served outdated text. Answer accuracy held steady.
The rest of the post explains how it works, how much it saves, and when you're better off not using it.
The orientation tax
An agent with search tools spends its first few calls getting its bearings. It runs a search, opens a result, decides that section was the wrong one, and searches again with different words. Only after that does it start working on the question itself. If somebody asked almost the same thing yesterday, all of that happens again from scratch.
Think of a question like "How do I return a product and what does it cost?". To answer it, the agent searches for the returns policy, probably searches again for shipping costs, and reads the sections it finds. If someone else asks the same thing in other words the next day, the documentation is the same, but the agent goes through the whole route again.
Each of those calls also puts search results and section text into the context, so the cost shows up in tokens as well as in latency. On the QASPER papers, an agent without a cache averaged 4.08 calls and 21.4k input tokens per question.
The figure below shows a real case from the runs, a question about resizing a gallery, answered three ways. Press Replay to watch the turns go by.
Why not cache the answer
The obvious fix would be to cache answers: embed the question, find a close neighbour, and return whatever was said last time. I didn't want to do that, for three reasons. Answers go stale, so if a page changed last Tuesday the cached answer still describes the old version and nothing tells you. Answers also cross permission boundaries, because an answer built from documents one customer can see might be served to a customer who can't. And a cached answer never gets better, since the model isn't involved anymore.
ragtrail caches the path and leaves the answer to the model. The agent still reads the current text and writes a new answer every time. What gets reused is the expensive part that doesn't depend on wording, which is knowing which sections were worth opening.
A few rules follow from that. Answers are never stored, only queries, the steps the agent took, section ids, content hashes and the signal that the answer was good. Every key starts with a scope, such as a tenant or a permission group, and no lookup crosses from one scope to another. Every stored path expires.
It also helps to be clear about what ragtrail leaves alone. It doesn't replace the agent or change how it searches. It sits on top of an agentic RAG loop that already works, adds a recall step before the first search and a record step after the answer, and leaves everything in between as it was.
What a trail is
A trail (a trajectory, in the code) holds a query, the steps the agent took, the sections the answer ended up citing, and a content hash for each cited section. Each step records which tool the agent called, with which arguments, and which section ids came back. A section here is whatever your index treats as a unit, which could be a heading section, a chunk or a whole document. ragtrail only needs an id for it and a way to ask for its current hash and text.
The sections that count as useful are simply the ones the agent says it used. When the agent records a trail, it lists the ids its answer relies on, and ragtrail stores that list as given. There's no relevance threshold inside ragtrail. The only condition it adds is on the write itself: nothing is stored unless the turn gets a positive signal, which can be a verifier's approval, feedback from the user, or the caller marking the answer as accepted.
Finding a similar question
Looking a question up has to be cheap, and I wanted it to work without a vector database. The core turns a normalised version of the query into a MinHash signature and splits it into bands with locality-sensitive hashing. Each band becomes a key in an ordinary key-value store. A lookup reads the band keys, collects the trail ids it finds there, and computes the exact Jaccard similarity against each candidate's signature. It needs only the standard library, gives the same answer every time, and fits SQLite on a laptop or DynamoDB and Cosmos DB in a deployment.
Lexical matching misses paraphrases, though, and paraphrases are most of what real repeat traffic looks like. So there's an optional matcher that uses a small embedding model (bge-small) with SimHash banding, which keeps lookups as key reads. In practice it made a large difference. With embeddings, 179 of the 200 paraphrased WixQA questions found a trail, against 25 with lexical matching alone.
Validating on read
Finding a similar question isn't enough to trust its trail, because the documentation may have changed since, the same way Jagger's wheel moved tables. Before a trail is used, ragtrail compares the hash it stored for each cited section with that section's current hash. If they all match and the questions are similar enough, the recall result is a prefetch and includes the text of those sections. If only some match, it becomes a hint that lists the sections still valid and names the ones that changed. If none match, it's a miss, and the trail is marked stale so it won't come back.
The figure below shows the three cases. Pick how many sections changed, and the check and the outcome update with it. In the hint case you can see the text the agent receives: it says a similar question was found, which sections are still good, which one changed and where to start. A prefetch also carries the text of the sections, always within a token budget.
I tested this directly, since it's the property the whole design depends on. After recording trails, I edited between 5 and 50% of the cited sections and replayed the repeat questions, once against ragtrail and once against a cache that stores and serves the text itself. With 10% of the sections edited, the text cache served outdated text on 12% of its hits on WixQA and 15% on QASPER, and the number keeps growing as more is edited. ragtrail never served a changed section, and at that 10% it kept 173 of 179 hits on WixQA and 260 of 274 on QASPER, turning some of them into hints. One caveat is that I picked the edited sections uniformly at random. Real edits tend to cluster around the same pages, and I didn't model that.
Does it help
For the evaluation, a Haiku agent ran through claude -p with the same tools, prompt and call limit in both arms, and a Sonnet judge that didn't know which arm it was grading scored the answers. The test questions are one paraphrase of each original question, 200 on WixQA and 400 on QASPER.
I started with an upper bound. If the cache holds perfect trails, ones that cite exactly the reference sections, the effect is large.
| Calls, without → with cache | Input tokens | Judged correct | |
|---|---|---|---|
| WixQA | 3.40 → 2.22 | 15.0k → 10.8k | 69.8% → 86.2% |
| QASPER | 4.08 → 1.48 | 21.4k → 9.2k | 60.8% → 71.6% |
That's the benefit when a hit is good. Nobody gets perfect trails in production, so the realistic test lets the agent record its own. In the first phase the agent answers the original questions with an empty cache and calls record with the sections it cited. In the second phase it answers the paraphrases against whatever the first phase stored.
| WixQA, self-recorded | Without cache | With cache |
|---|---|---|
| Tool calls | 3.40 | 2.08 |
| Input tokens | 15.0k | 10.0k |
| Judged correct | 69.8% | 74.2% (+4.5, 95% CI −0.8 to +9.8) |
The savings hold up and accuracy doesn't fall. It does rise by 4.5 points, but the interval runs from −0.8 to +9.8, so I can't claim that as a gain. The gap with the +16.5 points of perfect trails has a simple explanation: oracle trails contain the right section 100% of the time, and the ones the agent records itself, 69% of the time.
The figure below breaks the test questions down by what recall returned, using the v1 verifier gate described further down.
The admission problem
The 4.5-point average hides something, which shows up once you split the results with the cache by how the first answer was judged.
When the first answer was wrong, the agent recorded the path to a wrong answer, the paraphrase got that path back, and it did worse than it would have with no cache at all, dropping from 50% to 21%. A cache of paths spreads mistakes the same way a cache of answers does. It's only harder to see, because the answer is rewritten every time. These are the badly taken notes from the beginning: the agent bet on what had been written down without anyone checking it.
So a trail is only written when the answer gets a positive signal, and I tested three versions of that rule.
The first is an ideal gate that only admits answers the judge marked correct while looking at the reference answer. No deployment has a reference answer, so this can't run in production, but it shows the ceiling. With it, the agent averaged 2.52 calls and reached 75.5% correct, +5.8 points with an interval from +1.0 to +10.8. It was the first accuracy gain whose interval stayed above zero, and 20 of the 21 questions with a wrong first answer got no trail at all.
The other two are verifiers that a deployment could actually run. A model sees the question, the answer and the current text of the cited sections, never the reference, and decides whether the sections support the answer.
| WixQA | Calls | Judged correct | Incorrect answers admitted |
|---|---|---|---|
| Verifier v1 | 2.40 | 74.0% (+4.2, CI −0.8 to +9.2) | 10 of 21 |
| Verifier v2 | 2.81 | 75.2% (+5.5, CI +0.8 to +10.2) | 0 of 21 |
The first verifier let half of the wrong answers through. When I read those 21 answers, 19 of them said something like "the documents I found don't cover this" and then listed related facts from the cited sections. Those facts really are in the sections, so a check for support passes them. The second verifier adds one rule, which is that an answer saying or implying the sources don't answer the question counts as unsupported. It admitted none of the 21 wrong answers, and the trails it did admit were correct 80% of the time, against 66% with v1. The price is that it also rejected 54 of the 117 correct answers (v1 rejected 29). It admitted 79 trails instead of 134, so recall fired on 77 questions instead of 127, and the savings shrank.
QASPER behaved differently. The stricter verifier admitted far fewer wrong answers there too, 27 of 106 against 76, but accuracy didn't improve: 60.8% without a cache, 64.1% with v1 and 60.0% with v2. On those papers the wrong first answers weren't doing much damage to begin with (18% correct without a cache, 19% with the cache and v1), so the stricter gate mostly threw away useful trails along with the bad ones.
My reading is that strict gating pays off where a bad trail does real damage, and costs you where trails are cheap. Neither verifier replaces explicit feedback from users.
Something that surprised me
Shared context in the query swamped the similarity. My first QASPER setup put the paper's title into every question, in the form "About 'X': how did they measure Y?". A different question about a paper already in the cache then triggered recall 85% of the time with lexical matching and 97% with embeddings, and about 70% of the time it handed back the wrong sections. The title was doing all of the matching. The fix was to move fixed context, such as a document, a product or a customer, into the scope and match only on the question itself. With the paper as the scope and embeddings on, paraphrases still found a trail 274 times out of 400, always for the right paper, while other questions about the same paper triggered recall 13 times out of 880.
What I can't claim yet
- I used one agent model (Haiku), one paraphrase per question and one run per arm. I haven't measured how much results vary between runs, so differences of a few points should be read with their intervals, and the two verifiers can't be told apart on WixQA at n = 200.
- Without an oracle the accuracy gains are small. The result I'd rely on is the saving: fewer calls and fewer tokens, with no measurable loss of accuracy.
- Reusing evidence across questions in agentic RAG has been done before. MERMAIDS, for example, reuses verified evidence tuples across questions. What ragtrail adds is pointers validated by hash with a partial downgrade on read, reuse of a whole question's path over key-value LSH, isolation by scope, and a controlled comparison of oracle, self-recorded and gated trails.
Is it worth using
It depends mostly on how often questions repeat. ragtrail only saves anything when a question resembles one already answered, so it fits systems where many users ask similar things about the same documentation, such as a help center or an internal knowledge base. If almost every question is new, the recall step finds nothing and only adds a call. On the research papers, with the cache on, the agent took about a second longer on average.
To see how often it hits with realistic traffic, I mixed new questions with paraphrases of earlier ones in different proportions, starting with an empty cache.
The money saved is small. In these runs, answering a question without a cache on a small model cost around a tenth of a cent, so saving a third of the calls saves fractions of a cent. A more expensive model makes that number bigger, though it stays modest. What you notice most is latency, since a hit saves one or two round trips to the search tool.
It also needs section ids that stay stable, so the hashes mean something, a decision about what a scope is in your permission model, and a retention policy, because it stores the questions and those can contain personal data. Above all it needs a reliable signal of which answers are good, such as an approved answer or a thumbs-up. Without one, the cache ends up learning the mistakes.
Where it lives
ragtrail is MIT-licensed and written in Python. The core depends only on the standard library, sqlite3 and pydantic, and the cloud stores, the MCP server, the embedding matcher and the evaluation harness are optional extras. The harness and the reports behind every number in this post are in the repository at github.com/aralde/ragtrail. The paraphrases and the first-phase runs ship as a separate data bundle, because model runs can't be reproduced bit for bit.