Jev lost to my reranker. Then it did something a reranker can't.

Tested September 2026 · Jev jev-1.13 (TypeSafe AI) via OpenRouter · flashrank 0.2.10 (ms-marco-MiniLM-L-12-v2)

The short version

I tested Jev, a new type of AI model, in the agentic RAG behind our multi-agent assistant at Nands, to decide whether to put it into production.

At sorting documents, it didn't earn its place. It tied a free, open-source tool on average, and on my own platform's content it was worse than changing nothing at all.

But Jev can also say "this doesn't help", which a sorting tool can't. Used as a filter, it cleared out ~90% of the clutter and never threw away the right passage.

And when the answer wasn't in the documents, it said so, in 20 out of 22 cases, instead of handing the AI irrelevant text to guess from.

My decision: don't replace the sorter; add Jev as a filter. About $0.90 per 1,000 questions.

Why I ran this

At Nands, we've been building an internal AI assistant. It's a multi-agent system: specialist agents for coding, research, visuals (images, figures, slides) and building websites, plus a general agent for everything else. The agents also do agentic RAG: they search the platform's articles and a user's uploaded documents themselves, as a tool, whenever an answer needs facts they don't have.

Two decisions in that system quietly decide whether an answer will be any good:

Agentic routing. Every message first goes to a router, which picks the agent that should handle it. Send "plot sin(x) and cos(x)" to the coding agent instead of the visual one, and the user gets code when they wanted a chart.
Agentic RAG. When an agent needs facts from the platform's content or a user's documents, it calls a search tool (RAG, retrieval-augmented generation) and gets back a handful of passages. If the passage holding the answer isn't among them, the agent can't use it, however good the agent is.

Then the hype around Jev, from TypeSafe AI, caught my attention, and I started experimenting with it. The pitch is that it's a different kind of model. Most models you've heard of (ChatGPT, Claude, Gemini) write answers. Jev is a decision model: it doesn't write, it decides. You give it a question and a fixed set of options, and it tells you which option applies and how sure it is.

Agentic routing and agentic RAG are both exactly that shape of problem. So I set out to answer one practical question: can Jev do either of these jobs better than what we run today, at a speed and price we can actually ship? Not "is Jev impressive", but "does it earn a place in our assistant". This post covers agentic RAG. Part 2 covers agentic routing, plus a few other decisions I tested along the way.

Where Jev could fit in agentic RAG

When an agent calls the search tool, it doesn't get every document back. The tool works in steps, and the agent only sees what survives them:

How the AI finds what to read before it answers

50 passages. The one with the answer is 7th.

the passage with the answerother passages

Search grabs about 50 passages that look related to your question.
Sort reads them more carefully and puts the most useful ones on top. This step is usually called a reranker.
Filter throws away passages that don't actually help. This step didn't exist in my pipeline — Jev is what made it possible.
Answer: the AI reads only the top few and writes its reply.

The AI never sees what gets left behind. If the passage with the answer ends up in position 7, the AI either says it doesn't know or, worse, answers confidently from the wrong text. So what reaches the AI matters as much as how smart the AI is.

Jev can do steps 2 and 3, because for each passage I can ask it one question: "Does this passage help answer the user's question? Yes or no?" Jev answers with a percentage, like 3% or 99%. A percentage can be sorted and it can be cut at a threshold, which is the whole reason this experiment had two findings instead of one.

The percentage is trustworthy enough to act on, which matters for the filter. In my router tests (part 2) Jev's confidence behaved like a good weather forecast: when it said 90%, it was right about 9 times in 10. So "throw away anything below 50%" means what it sounds like.

🔧 For engineers: Jev takes a JSON state plus typed questions (yes/no, choice, or score) and returns a probability per option. The yes/no question I used:
{ "state": { "question": "…", "passage": "…" },
  "questions": { "helps": { "type": "noul",
    "instructions": "Does this passage contain information that directly helps answer the question?" } } }
→ { "helps": 0.03 }
The output can only ever be one of the options you defined, so there's no parsing and no malformed responses.

How I tested it

I took 62 real test questions where I already knew which passage holds the answer, gave every contestant exactly the same 50 search results per question, and checked where the right passage ended up.

The contestants:

Search alone: the order the search step returned (my starting point)
flashrank: a small, free, open-source sorting tool that runs on an ordinary server
Jev: a "does this help?" percentage for every passage
Laya: a free, open-source model from the same family as Jev

🔧 For engineers: 20 questions over my platform's articles (expected document known) and 42 over research papers in a test account's document store (expected passage known; these questions were auto-generated from the papers, and a few are a little odd). Search is my production hybrid retrieval (BM25 + embeddings, fused with RRF), top 50 per question. The right passage was in those 50 for 58 of 62 questions, which is the ceiling for any reranker. flashrank is a cross-encoder running on CPU.

Two questions, two very different outcomes

The clearest way to show what Jev did is one question it handled brilliantly and one it got wrong.

Where it shone — a question over an uploaded research paper:

"What is the title of the paper by R. Jozsa cited as reference [69]?"

Search result	What it actually contains	Jev: "does this help?"
#1	references [32]–[33] from the same paper	3%, thrown away
#2	references [61]–[62], so close	3%, thrown away
#3	"[69] R. Jozsa, Fidelity for mix…"	99%, kept
the other 47		all thrown away

Search did what search does: it found passages that look like the answer (the same reference list, the same formatting). Both flashrank and Jev moved the right passage to the top. But Jev also cleared away the other 49, so the AI would read exactly the right text and nothing else.

Where it failed — a question a real learner would ask on my platform:

"How do qubits use superposition and entanglement?"

The right article was "Quantum Computing: Beyond Classical Bits". My production search already had it at #2. flashrank promoted it to #1. Jev pushed it down to #4.

Nothing about that question is a trick. It's a broad, conceptual question, and the pool is full of passages that are all genuinely somewhat relevant — several articles touch superposition. Jev's yes/no question ("does this help?") can't separate "the best article on this topic" from "an article that mentions this topic", because both honestly deserve a yes. A reranker trained to order things can. That difference turned out to be the whole story of Finding 1.

Finding 1: at sorting, it depends on the content — and on mine, it lost

Reranking lifts the right passage to the top: Jev and flashrank tie

Questions where the right passage is ranked #1 (of 62)

Jev · graded scale

82%

flashrank

81%

Jev · yes/no per passage

77%

Jev · all 50 in one call

77%

No reranking (search order)

58%

Laya (off the shelf)

36%

020406080100

Same 50 retrieved passages per question for every contestant. Jev ≈ flashrank (ΔMRR −0.002, 95% CI ±0.08).

What to notice: the blue bars (flashrank and three ways of asking Jev) are nearly level, and all of them are far ahead of search alone (gray).

Both flashrank and Jev put the right passage first in about 8 out of 10 questions, up from about 6 with search alone. The gap between them was too small to call a winner. The open-source Laya, used off the shelf, did worse than doing nothing.

That's the average — and the average is misleading, so here is the same result split by where the questions came from. My 62 questions were two different jobs: 20 real questions over my platform's articles, and 42 auto-generated questions over research papers a test user had uploaded.

Right passage first	Search alone	flashrank	Jev
My platform's articles (20 questions)	75%	80%	55%
Uploaded research papers (42 questions)	50%	81%	88%

On my own platform's content, Jev made things worse than changing nothing — 55% against the 75% my production search already managed. Everything good about the 77% average came from the paper questions, where it was the best of the bunch.

Why the split? Those two sets ask different kinds of question. The paper questions are specific and factual ("which reference is [69]?"), where "does this passage help?" has one honest answer. The platform questions are broad and conceptual ("how do qubits use superposition?"), where a dozen passages all deserve a yes and the job is ranking them, not judging them. Jev is a judge, not a ranker.

These are 20 and 42 questions, so treat the sizes with suspicion — but the direction is the opposite on each set, which no amount of rounding explains away. If I had only looked at the average, I would have shipped the wrong thing.

	Search alone	flashrank	Jev	Laya
Right passage first	58%	81%	77%	35%
Right passage in what the AI reads (top 5)	89%	92%	94%	60%
Time per question	–	1.9 s	0.66 s	1.4 s
Cost per 1,000 questions	free	free	~$0.90	free

So the sorting question answered itself: a free tool does the job, and on my content it does it better. flashrank stays.

🔧 For engineers: I asked Jev three ways: yes/no per passage (MRR 0.847), a graded 4-level scale (0.868), and all 50 passages in a single call (0.845). flashrank scored 0.849 and search alone 0.709. Jev vs flashrank: −0.002 (95% CI −0.08 to +0.08), a tie; both vs search alone about +0.14 (CI +0.05 to +0.23). The table shows the single-call setup, the practical one at 0.66 s and $0.0009 per question. Laya's published checkpoint is a specialist for other tasks; part 3 of this series fine-tunes it.

Finding 2: Jev works as a bouncer

A sorting tool can only order passages. Because Jev gives a confidence for each one, you can also set a bar and throw out everything below it, like a bouncer checking each passage at the door.

Jev can drop ~90% of passages and still keep the right one

Each point is one threshold. 58 questions where search retrieved the correct passage. Top-left is best.

What to notice: Jev's lines (blue, orange, green) stay at the top even at the far left, where only a handful of passages are kept. Laya (yellow) loses the right passage as soon as it starts cutting.

Jev threw away about 90% of the passages and kept the right one every single time (in all 58 questions where search had found it).

When I combined the two tools, flashrank sorting first and Jev filtering after, the AI got the same or slightly better results while reading about half as much:

	flashrank alone	flashrank, then Jev
Right passage first	81%	82%
Right passage in what the AI reads	92%	94%
Passages the AI reads (typical)	5	2–3
Right passage wrongly thrown away	–	0 of 58

Less to read means fewer distractions for the AI, and a shorter, cheaper request.

🔧 For engineers: thresholds were P ≥ 0.5 (yes/no), "partially useful" or better on the 4-level scale, or ≥ 1.5 of 3 in the single call. All three kept 4.6–5.9 of 50 passages with 100% survival; Laya at the same threshold kept 17.3 with 84% survival. This matches the open-source jev-reranker result on a public benchmark (92% dropped, every relevant passage kept).

Finding 3: it knows when to say "not found"

This was the most useful result. If the bouncer lets nothing through, the honest reply isn't an answer at all: it's "I couldn't find that in your documents."

To test it, I wrote 22 questions my documents genuinely can't answer:

11 near misses: on-topic for one of the documents, but asking for something it doesn't contain. "How many hectares burned in the 2025 fire season according to the wildfire satellite paper?" This is the realistic, hard case.
11 off-topic: "How many moons does Neptune have?"

	Jev let nothing through
Off-topic questions	11 out of 11
Near misses	10 out of 11 (9 with the yes/no setup)
Answerable questions where search had found the answer (58 of 62)	wrongly emptied 0 out of 58

The remaining 4 of my 62 answerable questions were ones where search had never found the answer to begin with. Two of those came back empty from the filter too — and there, "not found" was the right reply: the answer genuinely wasn't in anything the filter saw. Without the filter, the AI would have received five loosely related passages for each of those questions and had to decide by itself that none of them answered it.

🔧 For engineers: unanswerability was verified by searching all 572 chunks of the corpus for each question's key terms. 6 of 28 drafted questions were excluded because a term appeared somewhere. Near misses: 9/11 empty with yes/no, 10/11 with the single-call setup. Same thresholds as Finding 2, not tuned on these questions.

What other teams have found

Others report big gains from using Jev for sorting, but almost all of them start from basic keyword search, which leaves a lot of room to improve. My search was already decent: it put the right passage first 58% of the time before any sorting. How much a sorting tool helps depends on how good your search already is. How much a filter helps doesn't.

🔧 For engineers: reported results from other teams, with their starting retrieval:

Source	Starting retrieval	Reported
MindStudio	BM25	top-1 21% → 54%
TypeSafe cookbook (legal search)	BM25, 30 candidates	top-1 5% → 18%, top-10 38% → 62%
jev-reranker	hybrid	nDCG@10 0.83 → 0.97; filter dropped 92%, kept all positives
refix.ai	–	relevance, contradiction, injection and citation checks as separate questions

Takeaways

If you build AI products:

Sort with a free tool like flashrank. It matched Jev on average, beat it on my own content, and runs on my own server for nothing.

Filter with Jev. One call per question clears ~90% of the clutter without losing the answer, for about $0.90 per 1,000 questions.

Use the empty result. When nothing passes the filter, say "not found" instead of letting the AI guess.

Split your results before you believe them. The same test said "Jev ties" on average and "Jev is worse" on the content I actually serve. One number per contestant would have sent me the wrong way.

If you're choosing or buying AI tools: The most valuable thing an AI can say is sometimes "I don't know". Ask vendors how their system behaves when the answer isn't there, and test it yourself with questions you know it can't answer.

What I'll test next: whether the filter improves the AI's final answers, not just what it reads; screening documents for hidden instructions aimed at the AI (a known risk with uploaded files and web pages); and checking each claim in an answer against its source.

What this test doesn't prove

It measures what the AI reads, not what it writes. Better passages should mean better answers, but I haven't measured that yet.
It's a small test: 62 answerable and 22 unanswerable questions. Small differences (a few points) should be read as ties.
One known answer per question. Some passages Jev threw away might also have contained useful information.
Some tests came after the first round. The single-call setup and the "not found" questions were added later, but I didn't tune any settings on them.

Part 2 looks at where decision models earn their place across the rest of an AI product, and where a simple rule still wins. Part 3 is about training the free, open-source model to match Jev on your own tasks.