I built SearchBench to answer a question that kept bothering me:
What is your coding agent actually searching for?
Not just what file it eventually names.
Not just how many tokens a retrieval tool claims to save.
I wanted to know what evidence the agent found, how close it got to the real fix, what it spent getting there, and whether the search behavior could be improved instead of guessed at.
The first result looked like a clean win.
The figures in this article are static projections of SearchBench artifacts. The values are copied into the blog layer; the page does not query SearchBench at render time.
I first compared Iterative Context, or IC, with jCodeMunch on two small slices. All outcome bars use percentages. Token and hop figures stay in the prose because they have different units and, more importantly, different denominators.
Five runs. Outcome rates only; this figure does not compare token use.
IC reached 80% exact hits overall versus 60% for jCodeMunch while both systems completed every run.
On the 5-run JAX slice, both systems completed 5/5 runs. jCodeMunch produced 3/5 exact hits; IC produced 4/5. Because every run completed, exact hits overall and exact-hit rate on completed runs are the same: 60% versus 80%.
The cost result was also strong. Median tokens per completed run were 347,480 for jCodeMunch and 127,985 for IC. The corresponding means were 277,741 and 101,586.8; completed-run totals were 1,388,705 and 507,934. Mean gold hop improved from 4.8 to 2.4.
That was the eye-catching result: against a structured retrieval incumbent, IC localized more files and used fewer tokens on this slice.
The 10-run diverse slice complicated the story. jCodeMunch completed 3/10 runs and produced 1/10 exact hits overall. IC completed 10/10 and produced 6/10 exact hits. Conditional on completion, that is 33.3% versus 60%.
Median tokens per completed run were nearly tied: 56,874 for jCodeMunch and 56,588.5 for IC. The means were not tied: 67,679 versus 173,377.2. Mean gold hop was 8.0 versus 4.8.
The raw completed-run totals were 203,037 versus 1,733,772 tokens. That does not mean IC was eight times less efficient. It means IC paid for ten completed runs while the incumbent paid for three. Raw total spend is useful for capacity planning, but it is a bad efficiency bar when completion counts differ.
So the honest result is narrower:
IC won the diverse slice on reliability and exact localization. It did not establish a clean token-efficiency win.
jCodeMunch belongs to a growing category of structured code-retrieval tools for agents: tools that expose symbols, outlines, references, or compact code context instead of relying only on raw file reads.
IC inherits from that idea. It also moves more search work into deterministic computation performed on the model's behalf. When the model supplies a query, IC can resolve fuzzy anchors, expand nearby graph context, bound the candidate frontier, and record replayable evidence before returning a smaller search state.
So this is not structure versus no structure. It is closer to:
structured retrieval as a tool surface
vs
structured retrieval compressed into a policy loop
The harder control was ordinary shell search.
Bash completed 8/15 runs, and all eight completions were exact hits. IC completed 15/15 and produced 12/15 exact hits. In rate terms, Bash was at 53.3% completion, 53.3% exact hits overall, and 100% exact hits on completed runs. IC was at 100%, 80%, and 80%.
The figure makes the tradeoff visible: Bash was brittle but sharp. It does not show cost. Bash used a median 22,634.5 tokens per completed run; IC used 136,128. The means were 71,500.3 and 194,347.3, and the raw completed-run totals were 572,002 and 2,915,210. Mean gold hop was 0.0 for Bash and 2.4 for IC.
Those numbers come from the current 15-run diverse Bash baseline, not the older 10-run ablation slices.
The Bash backend had shell access from the repository root. It could use ordinary navigation tools such as rg, git grep, find, sed, and Python.
Outputs and timeouts were bounded, but the backend was not artificially crippled. It received the same task framing and was scored by the same exact-hit, hop-distance, and token machinery.
That matters because Bash is the default workflow for a reason. Modern coding agents already know how to search with shell tools.
IC looked excellent against jCodeMunch. Bash showed that the real default baseline was harder to beat: IC completed much more reliably and found more exact hits overall, but it was much more expensive.
SearchBench did not merely validate my expectation. It corrected it.
Once Bash showed that IC was reliable but expensive, the next question was obvious:
Which part of IC was helping?
The policy snapshots pointed in different directions.
The old baseline completed 90% of runs, reached 80% exact hits overall, recorded an 88.9% exact-hit rate on completed runs, and had mean gold hop 1.33. Its token median was 146,678.
Anchor-only and lookahead-only each completed 100% and reached 70% exact hits overall and on completed runs. Both recorded mean gold hop 3.6, but their token medians were very different: 274,151.5 for anchor-only and 115,558 for lookahead-only.
Current IC completed 100%, reached 80% exact hits overall and on completed runs, recorded mean gold hop 2.4, and had a token median of 136,128.
Those values come from the current baseline and Bash-family ablation snapshots. They do not all share one matched task/control configuration, so the chart is a map of the investigation, not a policy leaderboard.
The cleaner signal came from a direct 10-run follow-up: lookahead-only versus minimal-anchor-v1.
Lookahead-only completed 9/10 runs and produced 6/10 exact hits overall, or 66.7% on completed runs. minimal-anchor-v1 completed 10/10 and produced 8/10 exact hits, so both its overall and conditional rate were 80%.
Mean gold hop moved from 4.0 to 2.4. The recorded composite moved from 0.533 to 0.640. Median tokens per completed run fell from 365,740 to 255,908, and the mean fell from 263,078.8 to 230,979.6. Completed-run totals were 2,367,709 and 2,309,796 even though the challenger completed one additional run.
That is the strongest internal IC signal in this run set. It is still a small follow-up, not proof that one anchor policy generalizes. It also did not solve the token problem: both medians remained far above the current 20k token-efficiency budget.
The useful conclusion was not “IC is done.” It was:
Anchor quality can move localization, lookahead alone is not enough, and stopping discipline is now the next problem.
That was the product moment. SearchBench already had stable tasks, comparable roles, bundle artifacts, hop-distance scoring, token accounting, and reports. A surprising result could become the next challenger round instead of another bespoke script.
These runs use Long Code Arena bug-localization tasks. Given a real issue and a repository snapshot at the buggy commit, the system predicts which files belong to the human fix.
SearchBench scores exact hits against the dataset's gold changed files. It also builds a tree-sitter code graph and measures how far each prediction is from the fix.
hop 0 exact target
hop 1–2 nearby evidence
hop 12 wrong neighborhood
A completed run produced a scored answer instead of timing out, failing, or getting blocked. A completion can still miss. In these figures, score >= 0.8 counts as an exact hit.
That distinction is why the diverse jCodeMunch result matters. The tool finished three runs, and only one of those three was a hit. Completion, conditional quality, and total cost answer different questions.
The useful thing about IC is not that graph search magically beats Bash. It is that parts of the search process move out of the transcript and into explicit, deterministic knobs.
anchor quality
Where should the search start?
lookahead
Which nearby evidence should be inspected next?
stopping
When is there enough evidence to commit?
A good anchor is a file, symbol, or subsystem close to the gold changed files under the hop scorer. A bad anchor starts the run in the wrong neighborhood. IC often looked bimodal: hop 0, or the max-hop boundary. The follow-up's move from 4.0 to 2.4 suggests that the starting neighborhood deserves direct attention.
Lookahead is controlled expansion from that anchor. It can surface related files, references, callers, callees, or other policy-selected frontier nodes. Without a grounded start and a disciplined stop rule, however, lookahead becomes expensive wandering.
Stopping asks whether IC spends tokens before finding useful evidence or after it already has enough. That motivates measurements such as:
tokens_to_first_near_anchor
tokens_after_first_near_anchor
tokens_to_first_exact_anchor
tokens_after_first_exact_anchor
The ablation is a test of a decomposed search policy:
start better
→ look around better
→ stop sooner
To isolate anchor quality from lookahead, the task set, repository snapshots, model bounds, scorer, token accounting, failure categories, and artifact model must stay stable.
Without that, the experiment becomes scripts, logs, screenshots, and vibes.
With SearchBench, it becomes a challenger round: same matches, same scorer, comparable incumbent/challenger roles, exact and hop scoring, token accounting, and replayable projection artifacts.
That is what I want from an agent evaluation harness. Not just a leaderboard. A way to ask a better question the moment a result surprises me.
This case study does not prove that IC universally beats Bash, that jCodeMunch is bad, or that SearchBench replaces repair benchmarks such as SWE-bench.
The samples are small. Some slices are homogeneous. Completion counts differ, which makes cost comparisons conditional. The policy snapshots are useful for navigation, but only the direct follow-up supports a within-slice challenger comparison.
What the evidence supports is narrower:
IC can beat a structured-retrieval incumbent on small slices.
Bash is a serious default baseline.
Hop distance makes misses more diagnostic.
Anchor quality appears to be an optimization knob.
Stopping and token discipline remain open problems.
The product is not just better search.
Better search is a consequence.
The product is making codebase navigation legible.
Senior engineers carry a mental map of a repository. Agents do not. SearchBench turns search failures into evidence about that missing map: where the agent gets lost, which files become decoys, which policy enters the right neighborhood, and which tool keeps spending after it has enough evidence.
The next experiment is straightforward:
measure token waste around the first useful anchor
add Bash-like lexical anchor seeding
add token-aware stopping
run the optimization ladder against Bash
IC does not already win.
SearchBench made the behavior visible enough to improve.
Before you trust your coding agent, ask what it actually searched.