Ariadne / Case study

Hybrid search in Ariadne: what I measured and what I cut

Ariadne is project memory for coding agents. It stores decisions and notes, and Claude Code queries it over MCP to find out why something was built the way it was. The whole thing rests on search, so when search is wrong by three positions, the agent gets the wrong answer.

This is how I checked whether the second search arm actually helps, and why it ended with cutting out its most expensive part.

1. The problem

Vector search understands meaning. It turns the question into a vector, turns every entry in the archive into a vector, and looks for the closest ones. That's how "which ORM did we pick" finds an entry saying "we went with Drizzle", even though the two sentences share no words.

That same property is its weakness. An embedding reads sense and blurs identity. A note that never mentions the search_text column sits about as close in vector space to a question about that column as a note that names it. In an archive full of database decisions, a dozen fragments look semantically identical and nothing tells them apart.

I noticed it by looking at what Claude Code actually asks about. Almost always a specific name: the searchNodes function, migration 0009, the code_anchors table. Exactly where embeddings are weakest.

2. The hypothesis

If meaning alone isn't enough, add a second arm that doesn't look at meaning at all, only at letters.

Postgres does this natively: a tsvector column, a GIN index, ts_rank_cd for ranking. No new dependencies. Search asks the same question twice, once by sense and once by name, and merges the results with Reciprocal Rank Fusion: position on both lists counts, not the score, because cosine and ts_rank_cd are two incomparable scales.

The hypothesis was that this would improve search. I had no idea by how much, or whether at all.

3. How I measured it

I built a golden set: questions with a known correct answer.

Corpus: 180 fragments cut from three project documents. Not ten test entries, because with ten and k=5 half the archive fits into the results, every variant scores near 1.0 and the comparison says nothing.

Questions: 62, all hand-written. That was a deliberate methodological choice. A model writing questions sees one fragment at a time and produces twins out of necessity, which then have to be sieved out with a similarity threshold pulled out of thin air. A person looking at the whole set at once simply doesn't write them.

The set splits into two kinds, and that split is the heart of the measurement:

• 40 questions in plain prose, with no proper name in them ("Why was running this on our own Oracle box dropped?"),
• 22 questions naming something specific ("Why does the search_text column use the simple configuration instead of a language dictionary?").

Every question is pinned to exactly one entry. The script halts if an anchor matches more than one fragment, because an ambiguous answer key is an ambiguous measurement.

I compared three variants over the same corpus and the same questions: the vector arm alone, the name arm alone, and the hybrid.

The metrics, in plain terms

recall@1 is the share of questions where the correct answer landed in first place.

recall@3 is the same, but a hit anywhere in the top three counts.

MRR is the mean reciprocal rank. First place scores 1.0, second 0.5, fourth 0.25, a miss 0. One number for how high the correct answer lands on average. A miss counts as zero rather than being skipped, because a search that answers one question in ten must not score better than one that answers nine.

Latency is the median time of one search, including the calls to Google.

4. Results

The aggregate number first. It's the easy one to show and the least meaningful.

Two percentage points. It looks like the second arm adds almost nothing and doubles the time. If I'd stopped here I would have cut it, with hard data to back me up.

Splitting the questions by kind shows something else entirely.

Plain prose, 40 questions

The hybrid hurts here. The name arm stays silent on 83% of these questions because there's nothing to look for, but in the few where it does speak up, it throws accidental matches into the fusion and pushes the correct answer down.

Questions with identifiers, 22 questions

Same mechanism, same database, opposite result.

5. What the data actually showed

The most important finding isn't visible in any table above until you look at recall@10.

On questions with identifiers, recall@10 is 100% for vector and 100% for hybrid. Vector was finding the correct answer anyway, if you look at ten results. The hybrid didn't find a single thing vector missed.

It doesn't find more. It orders better.

That changes what the whole experiment means. The second arm doesn't widen search reach, it settles the ordering where vector can't decide on its own.

The cleanest example in the set is the question about why the search_text column uses the simple configuration:

Vector understood perfectly well that the question was about text index configuration. The trouble is that this archive holds a dozen fragments about the database schema and they all mean roughly the same thing. The name search_text is unambiguous where the meaning isn't. That was enough to settle it.

And that's why RRF works: when two independent methods point at the same entry, their agreement is itself a relevance signal. Nothing more clever than that is going on.

6. Where the hybrid actually matters

Search is called from three places and the answer differs for each.

Claude Code over MCP. The most important case. The agent asks about searchNodes, migration 0009, code_anchors, which is almost entirely identifier questions. Ordering matters enormously here, because the agent often starts acting on the first hit and never reads the rest. Half a second inside an agent turn that runs for tens of seconds is invisible.

Chat in the app. Five entries go into the prompt, so recall@5 is what counts, not recall@1. The difference there is small: 98% against 95% on prose, 95% against 100% on identifiers. The time doesn't hurt either, because answer generation starts right after search and takes seconds.

The POST /search endpoint. The weakest case. A person types a query and waits for a list, so the delay is fully felt. It's also where prose questions show up most, which is the type the name arm helps least with.

7. The latency cost

The full hybrid pushed the median search from 351 ms to 856 ms.

That's a quality-latency trade-off: you pay in response time for better ranking. You usually can't have both, and the only question is how much you're willing to pay and for what.

Once I saw where those 500 ms came from, the question got a lot more interesting. Names were pulled out of the query by two mechanisms:

• regex (literalsByShape), matching by shape: snake_case, camelCase, paths, files with extensions, commit hashes. Effectively free,
• a cheap LLM (extractLiterals), catching names whose shape gives nothing away. About 500 ms.

I went through the 22 identifier questions: the regex handles roughly seventeen on its own. The model was only needed for names that look like ordinary words: react-force-graph, CORS, MCP Inspector, RLS, React Flow.

So we were paying half a second on every search to cover one case in five. And not on every fifth search, but on every single one: a prose question waited for the model too, which thought about it and returned an empty list.

8. The architectural decision

I cut the model. What stays is the hybrid: vectors plus regex-based lexical matching.

This isn't the mathematically optimal answer and I don't want to present it as one. It's a product trade-off. We lose names without a distinctive shape, roughly one case in five among questions that name anything at all. In exchange every search in the product stops paying half a second, and a whole failure class disappears with it: the model there was wrapped in error swallowing so that search wouldn't fail when name extraction did, which meant its rate limits and outages quietly degraded results with nothing to signal it.

An honest caveat: I did not benchmark the final architecture separately. The numbers in this document describe the hybrid with the model in it. The regex-only variant is a decision made from the distribution of those 22 questions, not from a run of its own. The harness is in the repository and measuring it is one command away, if it ever becomes worth doing.

9. What the benchmark taught me

The valuable part wasn't "the hybrid is better" or "the hybrid is worse". The valuable part was that the answer depends on the kind of query, and an aggregate number hides that.

Had I written only the 40 prose questions, the conclusion would have been: the lexical arm hurts, cut it. Had I written only the 22 with names, it would have been: the lexical arm is a big win, keep it and don't touch it. Both would have been honestly measured on their own set and both would have been false for the product.

The aggregate +2 percentage points is the average of −5 and +14. One number reading as a slight improvement where a twenty-point split in opposite directions is actually happening.

The practical takeaway: a golden set has to mirror the real traffic types in the product, not be a random list of questions. And results have to be reported per type, because only then can you recompute the outcome for any traffic mix without repeating the measurement.

The second lesson is less flashy but cost me more: the measurement itself can be broken in a way the result doesn't show. The name extraction model swallowed its own errors, so a Google rate limit produced empty names, the lexical arm scored worse than it really is, and nothing in the table said so. I added a check to the report: how many identifier questions got an empty response from the name arm. It came out 0 of 22, so the numbers hold. Had it come out 8, the whole document would have been worthless.

10. Final architecture

question
  |
  +-- embedding (gemini-embedding-001, 768 dimensions)
  |     -> 20 candidates by cosine distance (pgvector, HNSW index)
  |
  +-- regex on identifier shape
        -> 20 candidates by search_text (tsvector + GIN, ts_rank_cd)
  |
  fusion by position (RRF, damping 60)
  |
  trim to k

One model call per search instead of two. The lexical arm entirely inside Postgres, no extra dependencies. A question that names nothing skips the second arm, and then the whole thing is exactly the search it was before I added it.

The measurement is reproducible: backend/scripts/eval-corpus.ts builds the corpus, eval-questions.ts resolves the golden set, eval-search.ts computes the tables. Results land in backend/eval/results.md.

The document and the benchmark harness live in the project repository: r1zuuu/Ariadne