Jev, a judgment-only model released by TypeSafe AI on September 15, 2026, generates no text at all—it takes decision inputs and typed questions and returns only choices and probabilities. A third-party benchmark released two days after launch reported that when asked to classify 2,000 emails (phishing and legitimate) using a single judgment question, Jev scored 62.6% while Claude Haiku 4.5 scored 81.3%. In a separate measurement from the same verification, splitting the judgment into five questions, fitting the weights on half the data (1,000 labeled emails), and scoring the remaining 1,000 pushed Jev to 95.0%. The idea of a dedicated model that returns only judgments isn't new—the weight format demonstrated by BERT in 2018, bert-base-uncased, is still downloaded more than 47 million times a month from Hugging Face (as of September 19, 2026). How you read that 95.0% figure depends on whether the improvement came from a new class of model or simply from how the questions were split.
In a benchmark run two days after launch, Jev lost to Haiku on a single phishing question
Jev launched in early access on September 15, 2026, and skeptical foreign press coverage appeared the very next day. Two independent third-party benchmarks with published code followed on September 17 (the dates are those recorded by each repository). One tested phishing detection against regex rules and Claude Haiku 4.5 as controls; the other tested agent tool-call risk classification but merely compared two model names—jev-latest and jev-preview—without any external control. On September 18, the Python SDK typesafe_sdk 0.7.0 was published to PyPI.
The first benchmark measured 2,000 emails (1,000 phishing, 1,000 legitimate) from a dataset called PhishNChips v5.2, run from France. The email bodies were synthetic text generated by an LLM, and the ground-truth labels came from a URL reputation feed rather than human readers going through every message. Each system used a single prompt; the comparison model, Claude Haiku 4.5, ran without thinking mode, and the author explicitly frames it as the cheaper, faster option.
On the single judgment question—"Is this email phishing?"—Jev's accuracy was 62.6% versus Haiku's 81.3%. AUROC, a measure of ranking performance, came in at 0.689 versus 0.837, and calibration error (ECE, expected calibration error, 10 bins) was 0.154 versus 0.097—Jev trailed on both. Conversely, median response time was 239 milliseconds versus 687 milliseconds, and cost per 1,000 emails was $0.038 versus $0.462—roughly 2.9x faster and about 12x cheaper. Network-floor latency differed sharply too: 163 ms for Jev versus 18 ms for Haiku. These figures don't match the 193.6x speed and 1/444.6 cost claims TypeSafe itself advertises, because the comparison baseline is different.
Querying Jev looks nothing like prompting an LLM. According to official documentation, the API accepts a state object summarizing the decision inputs along with typed questions. Question types come in three flavors: Choice (pick one from candidates), Score (return an ordered scale value), and Noul (return a probability of "yes" between 0 and 1). All questions are evaluated in parallel and independently against the same state, and no text is generated at any point. Billing applies only to input tokens—$42 per billion tokens—with output tokens free, so adding more questions doesn't increase output-side costs. This pricing design and parallelism are the premise behind the "add more questions" strategy.
BERT's template of "just add one output layer," and 47 million monthly downloads of bert-base-uncased
The canonical example of a dedicated judgment-only model dates to the BERT paper, posted October 11, 2018. Jacob Devlin and three co-authors wrote that a pretrained BERT could be fine-tuned—by simply adding one output layer—into a model that set new records across a wide range of tasks, requiring almost no task-specific architecture changes. In practice, it set new records on 11 natural language processing tasks, with a GLUE score of 80.5% (a 7.7-point absolute improvement; OpenAI GPT stood at 72.8 on the leaderboard at the time).
The paper's headline advantage was how cheap fine-tuning was. While pretraining required significant compute, the paper states that all results could be reproduced on a single Cloud TPU in at most one hour, and SQuAD reached a Dev F1 of 91.0% in about 30 minutes. BERT-Base has 12 layers, a hidden size of 768, 12 attention heads, and 110 million parameters; BERT-Large has 24 layers, a hidden size of 1024, 16 heads, and 340 million parameters.
Shared weights, once built, can be reused—but the tradeoff is that fine-tuned models multiply with the number of tasks. A team running sentiment analysis, spam detection, and intent classification simultaneously ends up maintaining three separate models.
The lineage never broke. RoBERTa (July 2019) concluded that BERT had been significantly undertrained; the 1.5-billion-parameter version of DeBERTa (June 2020) scored a SuperGLUE macro average of 89.9, surpassing the human baseline of 89.8; and the most recent entry, ModernBERT (December 2024, from Answer.AI and LightOn), came in base (149 million parameters) and large (395 million parameters) versions with an 8,192-token sequence length.
Monthly download figures checked on Hugging Face on September 19, 2026 showed bert-base-uncased at 47,236,960 and ModernBERT-base at 4,489,330. The 2018 original checkpoint is downloaded more than 10 times as often as its most recent successor. Download counts can include repeated fetches from continuous-integration pipelines, so they're only a proxy for demand—but the fact remains that eight-year-old weights are still being pulled at massive scale.
One dividing line with Jev is exactly this "separate model per task" pattern. TypeSafe's documentation states explicitly that all accounts share the same weights under the current jev-1.13.0, with no customer-data fine-tuning or LoRA adaptation performed. Under BERT's template, the task is fixed at training time; under Jev's contract, the task is determined at runtime by the question text itself.
Did LLMs replace dedicated classifiers?
The GPT-3 paper, posted May 28, 2020, described a 175-billion-parameter model applied to every task purely through text-based task descriptions and a handful of examples—no gradient updates, no fine-tuning. This spread the view that training dedicated classifiers was no longer necessary. That view has met two different fates depending on the layer of the problem.
At the formatting layer, the question is essentially settled. OpenAI announced JSON mode at DevDay on November 6, 2023, allowing the response_format parameter to constrain output to syntactically valid JSON. In its August 6, 2024 Structured Outputs announcement, OpenAI itself noted that JSON mode alone didn't guarantee schema conformance, and reported that in its own evaluation of complex JSON schema adherence, gpt-4o-2024-08-06 hit 100% while gpt-4-0613 scored under 40%. Google likewise described, in a September 3, 2024 blog post, introducing Controlled Generation for Gemini 1.5 Pro at Google I/O and later extending it to 1.5 Flash.
A representative example of the same underlying idea is the Outlines paper, posted July 19, 2023. It formalizes generation as transitions through a finite-state machine, indexes the vocabulary, and zeroes out the probability of schema-impossible candidates at the moment the next token is chosen. This mechanically guarantees the output won't break format—but it says nothing about whether the value placed into that schema is actually correct.
At the judgment layer, results split by task. A November 7, 2024 comparison across five political science tasks (using GPT-4o) found that for 8-class political manifesto classification, a BERT fine-tuned on 200 samples scored 53.9%, while GPT-based prompting methods topped out at 48.8%.
Meanwhile, on a 20-class COVID policy classification task, fine-tuning with 500 samples scored 65.7% versus prompting's 65.8%—essentially tied. Fine-tuning on the same task improved from 55.3% at 200 samples to 71.3% at 1,000 samples.
The cost crossover point also appears in the same experiment. For the 20-class COVID policy task, assuming inference at 100 samples per second, the paper shows that cumulative zero-shot prompting costs catch up to fine-tuning costs somewhere between 150 and 200 samples. That figure is specific to that task's conditions, not a general break-even point for classification.
On a legal contract clause dataset, the gap widened considerably. For single-label classification (CUAD-SL), a fine-tuned DeBERTa scored 87.8% versus zero-shot GPT-4 prompting at 67.2% (2025, Artificial Intelligence and Law journal). With results tied on one 20-class task, and diverging sharply on an 8-class task and a legal-clause task, no single paper supports a blanket claim that "LLMs win" or "classifiers win."
Using LLMs as classifiers over long context introduces yet another form of degradation. Martin & Roger (posted May 12, 2026) reported that when a dangerous manipulation appeared after 800,000 tokens of benign activity, Opus 4.6, GPT 5.4, and Gemini 3.1 each missed it 2 to 30 times more often than when the same manipulation appeared in isolation. Jev's contract—evaluating all questions in parallel and independently against the same state—is designed to avoid cross-question interference. But because each question still reads a state of the same length, this design does nothing to address degradation caused by a long state itself. That failure mode is one Jev discloses about itself.
What's disclosed, and what isn't written anywhere
Rate limits are 250,000 tokens per second and 1,200 requests per minute; context length is 64k tokens, with the state plus the longest single question capped at 32k tokens. Input is text-only. There is exactly one model, jev-1.13.0; both jev-latest and jev-preview point to the same underlying model.
At the same time, items normally found in a model card are missing. TypeSafe publishes concrete numbers for pricing, rate limits, and context length, and it lists its own failure modes voluntarily. But architecture type, parameter count, layer structure, training data, and RLCD's reward function and training scale aren't described anywhere in the official documentation or official blog. This excludes member-only materials or pages that may be added later, and such gaps aren't unusual for early-access product documentation. Still, going only by what's public, there's no material to answer even the most basic question—whether it's encoder-type or decoder-type.
Response-time figures, by contrast, appear repeatedly across multiple documentation pages. The How to Build with System One doc states that "most queries complete in about 100 milliseconds"; the Use Case Map touts "frontier-level intelligence at real-time speed (150 milliseconds)"; and the Speculative Fan-Out page explains that adding more questions barely increases response time. The official blog gives an even wider range—70 to 500 milliseconds—and claims a 40x to 200x speed advantage over comparably intelligent LLMs. All of these are self-reported figures, with no accompanying table of measurement conditions. The blog's numbers carry a note that they were measured from a laptop on the U.S. West Coast to a service also located on the West Coast.
The homepage claim of "193.6x faster, 1/444.6 the cost" also needs context. It comes from an equally weighted internal evaluation across four business workflows, with ground-truth labels derived from the average of answers produced by GPT-6 Astra and Claude Fable 5.1 run in high-reasoning mode. As a real-world gain, it sits on the high end, and the company itself notes that because the evaluators are its own model capability team, bias is possible. The "0% hallucination" figure likewise isn't a measured result—the official blog states it was plotted at zero because schema conformance is guaranteed by design, not because it was empirically observed.
Some items are disclosed in the opposite direction. The official "jaggedness" page voluntarily lists nine failure modes: inability to count, reading dates as text rather than as an ordered quantity, no training on text generation, degraded accuracy with a large state, and so on.
The same page also demonstrates numerically that structural consistency isn't guaranteed. It gives an example where asking the identical question via Noul returns 0.22, while asking it via Choice as a yes/no question returns "no" at 0.99 with a confidence of 0.97—and it warns against reusing a Noul threshold for a Choice question.
These properties are disclosed, but the architecture that would explain why they occur is not. The BERT paper wrote layer count, hidden dimension, and total parameter count directly into its body text, giving follow-on research a concrete starting point for improvement. Asking the same question of Jev hits a wall: the official blog states that the company built a "new model architecture" in-house, but does not describe what that architecture is. There is a fork of LLaDA (a diffusion-based language model) in the company's GitHub organization, but forking an existing repository is not evidence of a model's actual lineage.
A synthetic 95.0% from five split questions, versus Haiku's 93.2% on the same five questions
The author split 2,000 emails into groups A and B of 1,000 each, used group A to determine signals, thresholds, and logistic regression coefficients, and reported all numbers using only group B. Every condition was held identical—the same dataset, the same split, one API call per email with no parallelism—and accuracy differences were tested with a paired, exact McNemar test comparing right/wrong outcomes on the same emails.
On the held-out 1,000 emails, Jev's best single signal scored 89.4%; a two-line regex rule using no AI scored 91.8% (Jev performed worse, p=0.0032); and Claude Haiku 4.5's best single signal scored 94.2% (versus Jev, p<0.0001). But Jev's and Haiku's "best single signal" were actually different questions, each independently selected as the top performer on group A. Jev's was sig_free_hosting (threshold ≥0.70), asking whether a link points to a shortened URL or free hosting; Haiku's was sig_generic_sender (threshold ≥0.08), asking whether the sender uses a free-email address while claiming to represent an organization.
Combining five questions pushed Jev from 89.4% up to 95.0%. Haiku, however, moved in the opposite direction: its single-signal score of 94.2% actually beat its five-question composite of 93.2%. Jev was the only model where decomposition closed the gap, and the difference between the two composites (95.0% vs. 93.2%) wasn't statistically significant under McNemar's test (p=0.063). Haiku's AUROC of 0.991 still exceeded Jev's 0.982.
Haiku's 94.2% and 93.2% differ by only 1.0 percentage point, and their 95% confidence intervals from the README overlap ([92.6, 95.5] and [91.5, 94.6]). No significance test is reported for the gap between a single model's single-signal and composite scores. Only for Jev can a clear direction be claimed—and its five-question composite does beat the two-line regex-with-regression score of 91.8% (p=0.0018).
データを表で見る
| Accuracy (%) | |
|---|---|
| Jev single signal | 89.4 |
| 2-line regex rule | 91.8 |
| Haiku single signal | 94.2 |
| Jev 5-question composite | 95 |
| Haiku 5-question composite | 93.2 |
All five of these figures are measurements on the same held-out 1,000 emails. The 62.6% vs. 81.3% figures from the single-question test mentioned at the top of this article were measured on the full 2,000-email set under different conditions entirely—they're not the same yardstick. It would be incorrect to read this as "62.6% became 95.0%."
TypeSafe's own documentation recommends breaking a broad judgment down into atomic questions and keeping the weighting logic on the code side. For phishing detection, that means asking easily observable factual sub-questions via separate Noul calls—such as whether a link points to a shortened URL—rather than requesting a single overall verdict in one question. That said, the author explicitly states that these particular five questions were written after reading the dataset's own URL-evasion classification scheme, and were tailored to match how this specific dataset was constructed.
When a model is asked to render an overall verdict in a single pass, it cannot expose which clues it weighted, or how heavily. Decomposing the task moves part of the judgment—the weighting of five separate probabilities—onto this side of the system, handled by logistic regression trained on 1,000 of our own labels, where it becomes something that can actually be inspected and tuned.
Separate lines of research have repeatedly shown that LLM judgment degrades with long input. Liu et al. (TACL 2024) demonstrated, across multi-document QA and key-value retrieval, that performance peaks when relevant information sits at the beginning or end of the input and drops sharply when it's buried in the middle of a long context. Chroma's technical report, dated July 14, 2025, tested 18 models across the Claude, GPT, Gemini, and Qwen families and found performance degrading consistently as input length increased, coining the term "context rot" for the phenomenon (this is a vector-database company's technical report; the term has no formal academic definition). Wang & Sun showed, using a proximal-interference paradigm from cognitive science, that retrieval accuracy degrades log-linearly as accumulated interference builds up, even when the target value sits immediately before the query.
None of these three studies connect directly to the phishing benchmark. The emails used were short, synthetic text, and Haiku answered all five questions in a single call—leaving little room for long-context degradation to play a role. Nor can Jev's jump from 89.4% to 95.0% be cleanly attributed to combining five signals versus simply fitting weights on group A's 1,000 labels—the two effects aren't separated in this test. TypeSafe's Introduction page states that because each question is evaluated independently, adding more questions doesn't trigger context rot and barely affects response time. That's the company's own claim; no third-party measurement confirms it.
The read-out format itself doesn't require a dedicated model. A technical article on Zenn implemented a scheme with Gemma3 270M that assigns answer candidates to single tokens and reads only the logit of the first token, reporting 77x the speed of having the same model generate a full JSON response. The author explicitly caveats that the comparison skewed toward an older, non-reasoning model generation—what was verified was read-out speed, not any accuracy gain from decomposition.
The cost and speed of gathering five signals also appear in the same benchmark. Jev answered nine questions—the five signals plus four others, including an overall verdict—in a single call, at a cost of about $0.04 per 1,000 emails. Haiku answered its five questions in one call for $1.02, with a median response time of 1,199 milliseconds. The author's conclusion is that what Jev retains is the ability to run decomposition cheaply: for signals of comparable quality, it's roughly 27x cheaper and about 5x faster.
Also worth noting: a two-line regex rule—maintaining lists of shortened-URL and free-hosting domains and comparing eTLD+1 (the registrable domain portion)—scored 91.8%, beating Jev's single signal outright. The author himself writes that this dataset is "nearly separable by construction," and this same regex rule scores 91.6% across the full 2,000-email set. This was a single run by a single author, with no independent replication.
Results favorable to Jev also appeared the same week. NearHere (byline: Jon Reed, September 16, 2026) reported that across 50 event-screening cases, Jev scored 96%, Gemini 3.5 Flash-Lite scored 86%, and Mistral Small 4 scored 84%, with cost per 1,000 judgments at $0.043 for Jev versus $2.496 for Gemini. However, the author cautions this is an applied-use study where results could shift with different prompts or samples, and expected judgments weren't verified through independent human adjudication. In a trial covering 12 pieces of writing, published by Every on September 15, Jev's median response time of 0.35 seconds contrasted sharply with Fable 5.1's 8.83 seconds—but of 7 known flaws, Jev caught 6 while Fable 5.1 caught 7. Author Mike Taylor writes that he'd want much more rigorous accuracy verification before deploying it in production.
Can confidence scores actually be trusted?
Jev's Choice and Score responses include, alongside the probability, a "confidence" value from 0 to 1 summarizing how concentrated the distribution is; Noul returns the probability of "yes." Using these values to route between automated processing and human review is central to the product's value proposition, which raises the question of calibration.
TypeSafe's own AI primer states that an assigned probability of 0.2 should correspond to an outcome that actually occurs about 20% of the time. That's calibration—a property measured over a population of predictions, not a guarantee about any individual answer's correctness. A model can have high accuracy but poor calibration if its probabilities run overconfident; conversely, a model with modest accuracy can still be well-calibrated if its outputs match reality's actual base rates. For workflows that use probability to design escalation rules, it's the latter property that matters.
Guo et al. (ICML 2017) reported that modern neural networks were worse-calibrated than networks from a decade earlier, and prescribed temperature scaling as a fix. Three years later, Desai & Durrett reported that BERT and RoBERTa were already well-calibrated in-domain, and that their out-of-domain calibration error was up to 3.5x lower than non-pretrained baselines (EMNLP 2020). The debate shifted again in the LLM era. The GPT-4 technical report states plainly, in its Limitations section, that pretrained models are highly calibrated on subsets of MMLU, but that calibration degrades after post-training (RLHF).
TypeSafe positions RLCD (Reinforcement Learning for Calibrated Decisions) as a third form of post-training, following RLHF and RLVR, explaining that it optimizes models to return decisions and calibrated probabilities rather than text. The company doesn't explicitly link the finding that RLHF degrades calibration to RLCD's own motivation—that connection remains an inference, not a stated claim.
Even so, no numbers demonstrating actual calibration have been published. Neither the confidence page nor the AI primer page includes ECE, a reliability curve, or any measured value. confidence is defined merely as a statistic summarizing how concentrated a probability distribution is, on a 0-to-1 scale, and operational guidance stops at recommending a threshold design with 0.5 as a floor.
Raw confidence figures did appear in two third-party tests dated September 17, 2026, but they measured calibration under different conditions and can't be directly compared. On the single judgment question across 2,000 phishing emails, Jev's calibration error of 0.154 was worse than Claude Haiku 4.5's 0.097. On the other hand, in the 60-case tool-call risk classification test—which had no external comparison model—jev-latest's ECE was 0.0712, and the author concludes calibration claims hold up for that task set. But in that same test, 50 of the 60 cases fell into the 0.9-to-1.0 confidence bin, meaning most of the ECE value is driven by that single bin. Different quantities, different tasks, different sample sizes—neither result can be called more "correct" than the other.
The 60 cases were hand-labeled by the author, split into 34 clear-cut, 14 ambiguous, and 12 adversarial examples. jev-latest and jev-preview (two names documentation says point to the same weights) both scored 91.7% accuracy, with ECE of 0.0712 and 0.0505 respectively, and median response times of 421.6 ms and 378.5 ms. Among the 40 cases where confidence hit exactly 1.000, there were zero wrong answers.
The README lists confidence values for incorrect answers ranging from 0.130 to 0.785, yet the reliability table shows 1 wrong answer out of the 50 cases in the 0.9-to-1.0 bin—the prose and the table don't line up. The comparison column meant for frontier LLMs is left empty, with the author explicitly stating he neither confirms nor refutes the vendor's advertised speed multipliers. Even within this 60-case test, the two model names can't be distinguished—repeated runs produced 91.7% and 93.3%.
Whether confidence scores can be trusted isn't settled by the material currently public. If you're putting this into production, the faster path is simply to count accuracy-by-confidence-bin yourself, on your own data.
Is this a step backward or a genuinely new category? Where the public evidence draws the line
What has genuine continuity is the requirements themselves: returning fixed-label judgments as probabilities, using calibration of those probabilities as an evaluation axis, and running a separate, judgment-only model that returns typed decisions. All three existed well before Jev.
Christiano et al. (June 2017) trained a separate reward predictor from human preference comparisons, solving Atari and simulated robotic control tasks using feedback on less than 1% of interactions. Llama Guard (December 2023) instruction-tuned Llama2-7b on safety classification data to classify prompts and responses as safe/unsafe, reportedly matching or beating existing moderation tools. The basic framework of a separate, judgment-only model has been in motion for at least three years.
What's actually changed is the unit of training and invocation. Under BERT's template, the number of fine-tuned models grew with the number of tasks. Jev, by contrast, keeps a single set of weights, and the task itself is determined at runtime by the question text. Specifying a task at runtime through natural language is exactly how LLM prompting already works—what's distinctly Jev's own is a combination of three things: typed output, parallel and independent evaluation against a shared state, and pushing the composition of results onto the code side of the system.
The phishing benchmark partially supports this operational model. But decomposition boosted the score only for Jev—when given the same five questions, Haiku's composite of 93.2% didn't reach its single signal's 94.2%. Whether decomposition is a Jev-specific advantage can't be concluded from this one test alone. What can be confirmed empirically is the speed and cost gap against a non-thinking Haiku: roughly 2.9x faster and about 12x cheaper on the single judgment question, and, in the multi-signal calls (where Jev answered nine questions in one pass), about 27x cheaper and roughly 5x faster.
At the architecture layer, no line can be drawn at all. Without any disclosure of parameter count, layer structure, or training data, there's no external way to determine whether this is a reinvention of encoder-based classifiers or something genuinely different.
For Japanese-language readers, there's one more thing worth checking first. TypeSafe's documentation states that its primary training language is English, and that other languages, including CJK languages, are "handled but not equivalent." No public data exists on Japanese-task accuracy or latency measured from Japan. Meanwhile, Japanese-language encoder models keep being actively developed. SB Intuitions' sbintuitions/modernbert-ja-310m has 315 million total parameters, was trained on roughly 4.09 trillion Japanese and English tokens, is released under the MIT license, and reports an average score of 89.83 across 12 Japanese evaluation datasets.
Even in Japanese, which model wins depends on the task. In a JGLUE comparison published by Fintan's machine learning team on May 11, 2023, ChatGPT (few-shot) scored 0.980 versus LUKE Japanese large's 0.965 on MARC-ja—but on JNLI, LUKE's 0.927 dramatically outperformed ChatGPT's 0.633.
When choosing between a judgment-only model and an encoder-type model, one of the deciding factors is how many labeled examples you have. Jev's 95.0% on the phishing task, too, was the result of fitting weights on 1,000 labels from group A. bitnovus/jev-spam-eval, tested on an 18,514-email dataset, reported that Jev given detailed criteria scored 98.33% versus TF-IDF logistic regression's 98.39%—essentially tied—and estimated from learning curves that matching that particular Jev configuration would require roughly 10,000, 200, or 100 labeled examples depending on the task. That said, the author cautions that the detailed spam criteria were developed through an exploratory process involving inspection of labeled mistakes, and that the learning-curve comparison used a specific sample rather than representing a general rule. The author also notes that splitting judgment into nine narrow questions made errors easier to investigate, but didn't yield an equivalent accuracy improvement.
When you have several hundred labels or more, a fine-tuned encoder model remains a strong candidate—exactly as in the political science example, where fine-tuning on 200 samples of 8-class data beat GPT-based prompting. For use cases where the classification criteria change frequently, Jev's contract—specifying the task at runtime through question text—may offer a real advantage. Once parameter counts and training data are disclosed, and a third-party benchmark appears comparing Jev against fine-tuned encoder models under identical tasks and label sets, this choice can be made from a table rather than a guess.