Part 2 of 3 · Decision models in a real AI product
Where a decision model earns its place in an AI product (and where a regex still wins)
TL;DR
- Behind every message to our multi-agent assistant at Nands sit several small decisions: which specialist answers, does the user want an image, a web search, a plan first, what kind of website? I tested Jev, a hosted "decision model", against the code that makes those decisions today.
- Jev won where the decision needs understanding the wording: catching image requests (92% vs 31% for my regex), web-search requests, trick questions, and tagging news articles (86% agreement with the platform's editors).
- My existing code won or tied where a rule already captures the answer: arXiv IDs, attached images, website build types.
- On the prompts I wrote myself, Jev's routing advantage shrank to nothing measurable. Averages hid that; slices showed it.
- I ended up with a four-question test for "should this decision use a decision model?", plus a map of where it fits in my product, including labeling, moderation and recommendations.
Where this fits
At Nands, we've been building an internal multi-agent assistant: specialist agents for coding, research, visuals and building websites, plus a general agent. In part 1 I tested Jev on agentic RAG, the search the agents run when they need facts. This post is the other half of the question: agentic routing, the step that decides which agent gets each message.
Routing turned out not to be one decision but several. Before an agent runs, the system also checks whether the user wants an image, a web search, a plan first, or a particular kind of website. Each of those is made today by a regex, an embedding router or a small LLM call. So for each one I asked the same question as in part 1: does Jev beat what we already run, at a speed and price we can ship?
What a decision model is
A large language model writes answers. A decision model picks from a list you give it and tells you how sure it is:
LLM: "Which assistant should handle 'draw me a cat'?"
→ "Great question! I think the visual assistant would be best because…"
Jev: same question + options [general, coding, research, visual, build]
→ visual (0.99)
That's the whole interface. You get a choice, a yes/no probability or a score on a scale, always one of the options you allowed, with a confidence that means something. Jev (TypeSafe AI) is the best-known one; Laya is an open-source model of the same shape.
I measured three properties first, because everything else depends on them:
- Consistency: the same 100 routing questions, asked three times each. Answers moved by 0.004 on average, and no decision changed.
- Honest confidence: when Jev said ~90% on routing, it was right about 90% of the time (calibration error 0.034).
- Many questions for the price of one: nine questions in one call gave the same answers as nine separate calls (average difference 0.0025, 1 flip in 900), for 3.5× less money, because the message is billed once.
Cost, for scale: batched this way, a chat message's full bundle of decisions (routing, nine tool intents, build type) came to roughly 5 cents per 1,000 messages in my later experiments — part 3's 5,368 labeling calls cost $0.25 total. Money is not the argument against Jev anywhere in this post; latency sometimes is.
The contestants and the exam
For each decision I compared today's code (the regex, embedding router or LLM call that runs in production), Jev, and Laya off the shelf. The exam: ~300 messages, half written by me with known answers, half synthetic, covering tricky phrasings and deliberate trick questions. Labels came from two independent LLM annotators, with disagreements settled by hand. Every contestant took the same final exam, and all tuning happened on a separate practice set.
Decision by decision
1. "Does the user want me to make an image / search the web?"
Before my assistant's AI runs, a set of regular expressions checks for common requests and shortcuts straight to the right tool.
| Intent | Regex: F1 (recall) | Jev: F1 (recall) |
|---|---|---|
| Make an image | 0.47 (0.31) | 0.73 (0.92) |
| Web search | 0.17 (0.09) | 0.88 (1.00) |
| 3D from a description / 3D from an attached image / compare papers by arXiv ID | 1.00 / 1.00 / 1.00 | 0.57 / 1.00 / 0.67 |
The regex never fires falsely on image requests, mostly because it barely fires at all. "Draw me a cat" and "design a poster" slip past it because they don't say "image". "Can you look up…" and "check online if…" slip past the web-search pattern. Jev reads the intent.
But where the answer is structural (is there an arXiv ID? an attached image?), a regex is perfect and instant, and Jev adds nothing. (These rows have only 1–3 positive examples each, so they're indicative.)
Jev also has a characteristic weakness: it reads markers literally. Every message starting with [image attached] got flagged as "the user
is asking about their documents". The lesson: send decision models clean, structured state (has_image: true), not text markers.
Verdict: a hybrid. Jev for the meaning-based intents, regex for the structural ones.
2. "Which assistant should handle this?"
My router compares the message's embedding with example phrases for each assistant.
| Router today | Jev | |
|---|---|---|
| Overall accuracy | 76.3% | 88.2% (+11.8, 95% CI [+5.7, +18.5]) |
| On prompts I wrote myself | 84.3% | 86.1% (no measurable difference) |
| On trick questions ("what's the best image model?") | 52.9% | 97.1% |
| Time per decision | ~100 ms | ~415 ms |
This is the result I'd most like other teams to see, because the average hid the story. Jev's big overall win comes almost entirely from phrasings my router's example sentences don't cover, above all questions about a topic that look like requests for it. On the kind of prompts I write myself, the two are tied. One example from the trick set:
"What's the best free software for 3D modeling?"
My router matches it against example phrases, sees "3D modeling", and sends it to the visual assistant — which makes things. Jev reads it as a question about 3D software and routes it to research. The user wanted a comparison, not a model.
Same pattern on "plot sin(x) and cos(x) from 0 to 2π" (router: coding; Jev: visual) and "write a caption for a photo of a lab bench" (router: coding; Jev: general).
Jev also missed my speed budget for an in-chat decision (300 ms). But its honest confidence suggests a better design than replacing the router: a second opinion. My router already knows when it's unsure (it was in its "weak match" band on 37% of messages). Send only those to Jev: most messages keep the fast path, and the confusing ones get the model that's good at confusing ones. Jev alone could decide 43% of messages at ≥ 98% accuracy. I haven't built this. The closest thing I measured is the reverse order: use Jev's answer when Jev is confident, otherwise keep the router's. Replayed over the test set, that scored 89.1%, against 76.3% for the router alone. It still calls Jev on every message, so it doesn't fix the latency; the router-first version is the one to build, and part 3 ends with a plan to ship it behind a feature flag.
3. "Does this research question need a plan first?"
While building this benchmark I discovered that my production classifier for this had never worked. It asked a small thinking model for a one-word answer with a 5-token budget. The model spent all 5 tokens thinking silently, returned an empty string, and the code read "empty" as "simple", every time. Fixing one flag made it work (F1 0.69). Jev scored 0.78, but with only 9 complex examples in the test set, I can't tell those apart statistically.
4. "Single page, multi-page site, or interactive app?"
The regex scored 94.5% (100% on the prompts I wrote); Jev 90.9%. Keep the regex. The rules here are crisp ("multi-page", "calculator", "dashboard with filters"), and a decision model can't beat a crisp rule on its home turf.
5. Tagging news articles by topic
There's no automated version of this today: editors pick a category. Against the editors' choices:
| Always guess "AI" | Laya | Jev | |
|---|---|---|---|
| Agreement with editors | 36% | 52% | 86% |
Jev could tag 74% of articles on its own at ≤ 5% disagreement, and most of its "disagreements" were defensible (the editors filed
"OpenAI launches GPT-5.2" under Technology; Jev said AI). Along the way I found the category field uses 16 different spellings for five topics
(ai, AI, AI & ML, AI & Machine Learning…).
6. RAG reranking
Covered in part 1: at ordering passages Jev tied a free local reranker on average, and lost to it on Nands' own articles. As a filter, though, it dropped ~90% of candidates while keeping the correct one every time.
7. Laya, off the shelf
It was 25× faster than Jev (11–35 ms on a laptop GPU) but never beat my existing code in any of these tests. The published checkpoint is a specialist for other tasks. Part 3 shows what happened when I fine-tuned it on Jev's answers for my decisions: it matched Jev.
The four-question test
Put a decision model on a decision when all four are true:
- The answer is one of a fixed list: a category, yes/no, low/medium/high.
- Deciding needs understanding the wording, not spotting a pattern. "Wants a picture?" needs understanding; "contains a YouTube link?" doesn't.
- A few hundred milliseconds is acceptable, because it runs in the background or replaces something slower.
- The alternative is a brittle regex or a slow, expensive LLM call.
How my decisions scored:
| Decision | 1 | 2 | 3 | 4 | Verdict |
|---|---|---|---|---|---|
| Image / web-search intent | ✅ | ✅ | ◐ | ✅ | Jev (hybrid) |
| YouTube / arXiv detection | ✅ | ❌ | – | ❌ | Regex |
| Routing | ✅ | ✅ | ❌ in-chat | ◐ | Jev as a second opinion on unsure cases |
| Build type | ✅ | ◐ | – | ❌ | Regex |
| News tagging | ✅ | ✅ | ✅ background | ✅ nothing today | Jev |
| RAG filtering | ✅ | ✅ | ◐ | ✅ | Jev filter |
What other teams use decision models for
- Routing: support tickets to queues, and even choosing which AI model handles a task (jev-router picks the cheapest capable model per coding task).
- Moderation and safety: spam, scams, toxicity, policy checks on user content.
- Agent guardrails: approve, confirm or block a tool call before it runs (LangChain).
- Replacing reliable-but-slow LLM classifiers: one team moved its classifiers to Jev yes/no questions because "the LLM was just the slow, expensive way to run them."
Where it fits in my product
Nands has about 800 pieces of content (articles, posts, news, events, projects, courses, jobs, hackathons) but, before launch, very little behaviour data: under a hundred users and a few hundred interactions. That shaped my map:
- Labeling (strong fit). One call per item, many questions at once: topic, sub-topics, level (beginner → advanced), format, audience, evergreen vs time-sensitive, region, and for events and courses "online?", "free?", "beginner-friendly?". Backfilling everything costs well under a dollar.
- Moderation (strong fit). Comments, threads, posts and DMs, including prompt injection aimed at my agents, since some agents read platform content. Honest confidence gives a clean three-way policy: allow / hide / send to a human.
- Recommendations (a component, not the engine). With so little behaviour data, "people like you also liked…" can't work yet, so recommendations have to start from what the content is about, which is exactly what the labels provide. A decision model also fits the filter step ("right level for this user?"). Finding candidates and ordering them still belongs to embeddings and scoring: calling a model for every item × every user doesn't scale, and ordering isn't its strength.
Caveats
- The test set is prompts I wrote plus synthetic ones; real user traffic will differ. A human check of the labels is still pending.
- On a tiny adversarial slice (6 prompts written to confuse the router), Jev actually scored worse than the baseline (3/6 vs 4/6). Six prompts prove nothing either way, but it's a reminder that "better on average" isn't "better everywhere".
- Some intents have very few positive examples in the test set, so treat their per-intent numbers as indicative.
- Latency numbers include the network round-trip from my dev machine.
Part 1 covers the RAG results in detail: Jev vs flashrank as a reranker, and Jev as a relevance filter. Part 3 is about distilling Jev into a free model I run myself.