Jev picks the documents. One call answers.

What this means for a RAG system

• Use Jev to pick the context. Its top 8 of 1,000 documents held one with the answer on 94 of 100 questions, against 49 for keyword ranking. One LLM call on those 8 matched a search agent at 38% of the cost and 1/7 of the time. Methods
• Use Jev to decide whether to answer. Asked whether the documents can answer at all, before the LLM runs, Jev separated answerable from unanswerable questions almost perfectly, in an easy test. Unanswerable questions
• Jev as agent tools did not help in our setup. Each Jev call used up one of the agent's limited turns, and the agent with Jev tools cost more for the same accuracy. Other ways of wiring Jev into an agent are untested. Jev as a tool
• Checking a written answer was close to a tie. Sonnet's stated confidence flagged wrong answers slightly better than the Jev check, 0.93 against 0.87, but there were only 9 wrong answers and one case drives most of the gap. Jev check

At 1,000 documents per question. A larger corpus needs a cheap first search before Jev; see Limitations.

How Jev is used

Jev is a small, fast model from Typesafe. It answers typed questions about text with probabilities, not prose.

Where Jev sits in each method

Jev reads up to 12,000 characters per document and never sees the benchmark's answer. Sonnet never sees Jev's scores. Jev costs $0.042 per million input tokens: about $0.09 to screen 1,000 documents, and $18 of the study's $208.

The three methods

Every question comes with 1,000 BrowseComp-Plus documents, about 7 to 9 million tokens. A few of them hold the answer and the facts behind its clues; the rest are random. A Sonnet 5 judge marks each answer right or wrong against the benchmark's answer. We call one method better only when the 95% confidence interval for the difference, over the same questions, is entirely above or below zero.

The Jev reader scored 3 points above the plain RLM, run on the fixed setup described below, with a 95% interval of −3 to +9, so the difference is not clear. On the first setup both scored 0.94.

Why one call is enough: Jev screen's top 8 held a document with the answer on 94 of 100 questions. BM25, a standard keyword ranking, managed 49. BrowseComp questions describe the answer through indirect clues, which keyword ranking misses.

Unanswerable questions

We took 50 more questions and swapped out the documents that answer them, so the right move is to decline.

Sonnet decides, after reading

• The Jev reader declined more, at 1/8 of the cost and 16 seconds instead of 4.6 minutes. Its two confident answers filled a gap from memory, since one call over 8 documents cannot search further.

Jev decides, before Sonnet runs

We asked Jev one Choice over the same 8 documents: do they contain enough information to answer the question? The rules were set before the run, which cost $0.09 for 150 questions.

• AUROC 0.98, where 0.5 is a coin flip. Jev scored all three of Sonnet's confident unsupported answers low. Putting Jev first at 0.2 catches two of them for 2 correct answers lost, for about $0.0006 a question. Only the 0.5 rule was set in advance.
• Caveat: an easy test. The swapped-in documents are random and plainly off topic, while real unanswerable questions have near-miss documents. No Sonnet call was actually skipped.

Jev check: is a written answer supported?

The Jev check verifies an answer after Sonnet writes it, so its verdict depends on what Sonnet wrote; it does not decide whether to answer. A made-up case where none of the 8 documents answers the painter question, and one is a true biography of a different painter:

Every answer gets two numbers: the confidence Sonnet writes in its answer, and the Jev check's probability that the cited documents support it. A good check gives wrong answers the lower number.

• Both flag wrong answers well. Sonnet's confidence did a little better, but with only 9 wrong answers, one that both rated highly accounts for most of the gap.
• Two of the 9 wrong answers are declines such as "cannot be determined". The check was not built for these: Jev often agreed the documents support the decline, which counted against it.

Why an RLM, and how the numbers compare

This test uses the same setup as the Recursive Language Model paper by Zhang, Kraska and Khattab: 1,000 documents per question, too many for one prompt. An RLM gets the documents as a variable in Python, searches them with code, and sends parts to a smaller model. We chose it because it is among the strongest published methods here and its code is open, so we could run it with Sonnet 5 and add Jev to it.

• These are context, not a head-to-head. The paper and this study each drew 150 questions at random from the same 830. The paper did not publish its list, so we do not know how many questions overlap, and the random filler documents differ too. The paper also ran GPT-5 with a GPT-4.1 judge, where we ran Sonnet 5 with a Sonnet 5 judge. So our scores cannot be ranked against these.
• The public BrowseComp-Plus leaderboard searches a 100,000-document corpus instead, so we do not compare against it.

Jev as a tool inside the RLM

Every headline number on this page uses the fixed setup. We first ran the RLM with 20 turns and the RLM library's prompt for running out of turns, to which Sonnet often replied with more code instead of an answer. Each Jev tool call spends a turn, so this hit the RLM with Jev tools hardest. The fixed setup gives 30 turns and a last turn that tells Sonnet to answer without code. The same 100 questions on both:

• With the fixed setup, the RLM with Jev tools went from 10 points behind the plain RLM to 1 point behind, but still cost more, $0.42 per question against $0.35. Inside the RLM, Jev added cost without adding accuracy. In front of one call, as the Jev reader, it cut cost.
• On the 22 questions where the RLM with Jev tools ran out of turns in the first setup, the plain RLM got 17 right and the Jev-tools version 7 in that setup. A version whose Jev tool returned a ready-to-read shortlist in one call, to save turns, got 13 at $0.66 each, and the Jev-tools version in the fixed setup got 13 at $1.01 each.
• Rerunning the same questions moves the count by about 3 either way.

Limitations

• Scale. Jev screen scores every document, so its cost grows with the number of documents. At 1,000 documents it was $0.09 of the Jev reader's $0.13 per question. At 100,000 documents it would be about $9. A larger corpus needs a cheap first search, by keywords or embeddings, to cut it to about 1,000 documents before Jev. Jev can only pick from what that search keeps, and we have not tested this at that scale.
• An easy, small test. Each question's 1,000 documents always include its answer documents, and the rest are random. In a real corpus, many documents can look close to the answer, which would make both searching and screening harder. The sample is also small: 100 main and 30 to 50 held-out questions, one run each, scored by a Sonnet 5 judge. Rerunning moves counts by about 3.
• No other retriever. We compared Jev screen with BM25 keyword ranking only. An embedding retriever might also put the answer documents in the top 8.