LLM vs Decision models like Jev and use-cases

I ran a small benchmark on my laptop this week. Eight triage questions, four model rows, one shared ground truth. The general-purpose 4B chat model got all eight right, in both thinking and non-thinking mode. So did the best decision model. The clock is where they split: 1,035 ms with thinking off, 26,783 ms with thinking on, against 176 ms for the decision model and 34 ms for the smallest one.

That is the pitch for decision models like Jev in a single table. It is also why the pitch needs care, because the accuracy row tells a different story than it did a year ago.

What a decision model does

A decision model takes a state and a set of questions and returns typed answers: a choice from a fixed option map, a score on an ordered scale, or a boolean with a probability attached. One forward pass handles it. No generation loop, no sampling, output_tokens in the usage block always 0. The probabilities come out of fitted heads with per-type temperatures baked into the GGUF, which is what "calibrated" amounts to in practice.

A chat model can answer the same questions if you write a JSON-only prompt, describe the schema in every request, and parse the reply back into a structured shape. It works. It costs time and plumbing.

Eight questions, four rows

Accuracy no longer separates the two camps. A 4B chat model tied the best decision model and tripled the smallest one on these prompts. What it cannot do is answer in 176 ms: 5.9× slower with thinking off, 152× with it on, and it hands you plausible numbers (1.00, 0.90, 2.00) with no fitted temperature behind them.

The smallest model's 3/8 deserves a look too. It filled in every answer field correctly and chose wrong: routed a billing ticket to shipping, called an angry customer not angry, rated a blocking bug as minor, and accepted secure-verify-bank.xyz as a real bank domain. Its misses were knowledge gaps. Worse, its sentiment score flipped from 0.96 to 0.04 on a reworded state alone, which I verified is sensitivity in the small encoder rather than a bench artifact. Fast and fragile are a package deal unless the model knows your domain.

Where the time goes

Decision models are input-bound. Latency tracks prompt length: about 0.5 ms per token for vllm-d3, 0.08 for Julia. Chat models are output-bound. Latency follows generated tokens at roughly 54 tok/s plus a prompt-eval term the prompt cache usually erases. Thinking mode explains almost the entire gap in the table: 843 to 2,079 generated tokens per request with the budget on, 29 to 43 with it off, identical answers, 26× difference. The obvious knob (reasoning_effort: "none") silently fails on this template; chat_template_kwargs: { enable_thinking: false } is the one that works.

Caching flips the comparison for long inputs. A bidirectional decision pass renders the whole prompt every request, so warm and cold measurements match within noise. A chat model replays a cached prompt, so the same 1,205-token state costs 807 ms warm and 2,448 ms cold. At 2.2k tokens vllm-d3 always sits at 901 ms; the chat model beats it warm and loses to it cold by 2.7×. A unique long state, which is what production usually sends, is where the decision model wins.

What people actually use them for

Routing and orchestration is the most cited case: deciding which LLM to call (large model or small), which tool or sub-agent to invoke, and whether a sub-agent should interrupt its parent. When2Call is the dedicated benchmark for that last question.

High-volume classification follows. Support tickets (one report: 49 of 50 correct), email intent (Banking77: 94.5%), sentiment, article tagging, SOC triage, and sorting errors without burning reasoning tokens on each line of a log.

Guardrails sit in front of actions: is this shell command safe to run, is this input a jailbreak, does this text contain PII. One caveat from the threads: with Base64 or hex-encoded commands Jev drops to around 91% accuracy. Decoding first with deterministic code puts it back near 98.4%.

Confidence thresholds work as architecture. Above 95% act automatically, between 70 and 95% escalate to a strong LLM, below 70% hand off to a human. The value used for that comparison is the model's calibrated distribution, which matters because a chat model's self-reported confidence is prose.

Verification is a quieter use: re-checking every fact an LLM claims to have extracted, catching a wrong side or sex in a radiology report template, logging decision ID, model version, distribution, threshold, and the verified outcome afterward.

Micro-decisions inside agent loops are where the savings compound. A stopping decision in DeepSearcher went from 2.23 s to 0.55 s at roughly 7.5× lower cost with unchanged Recall@5. A real-time signal scanner spends about $0.00002 per decision in 80 ms and calls the expensive LLM only on doubtful cases. Retry, stop, escalate, ask a human: each is one cheap classification.

Beyond that: home automation and Kubernetes autoscaling where a wrong free-text answer breaks things, on-device voice agents and robotics (the setting people keep calling the endgame), game AI mapping unit state to options, and the boring long tail such as CV screening, insurance classification, and search reranking. One case study reports the same accuracy as Gemini, faster and cheaper.

When to keep the LLM

Generation, coding, deep reasoning, math, difficult multi-hop retrieval, open-ended answers. On knowledge and reasoning tasks general models still hold the edge: Haiku 5.5 scored 61 against Jev's 58 on the AA Decision Index, at 11× the latency and 4.4× the cost. My bench points the same direction from the other side, since the 4B chat model swept a test full of domain knowledge the small decision model failed.

Caveats worth repeating

Calibration holds up on classification and tool-calling and degrades on knowledge and reasoning. Confidence is not permission: 94% means probably right, not authorized to act. Compare against the baseline you would otherwise ship, a rules engine or a small fine-tuned classifier, rather than only against a frontier LLM. And mind the formula: Jev's confidence is (p_max − 1/n) / (1 − 1/n), which is not the max probability, so thresholds written for one will misbehave with the other.

A cheap decision is not a saving until you check the outcome.