I got access to TypeSafe’s Jev a few days ago. It’s an odd kind of model that doesn’t generate text at all. You send it some content plus typed questions (yes/no, pick one of these options, rate this on a scale) and it gives you back probabilities. Setup: 49 tasks, about 8,200 items, all from public labelled datasets (SST-2, MMLU, ARC, MS MARCO, XNLI, SQuAD, Banking77 and so on), plus some synthetic tests where code computes the right answer. Both models got the same question wording. The baseline was gpt-5.6-luna with reasoning off, and again with reasoning on low. What I found: Jev matched or beat the baseline on 42 of 49 tasks! ~105ms median server time vs 700-800ms about $0.04 per 1,000 items vs $0.16-0.19 calibration error roughly half the baseline’s, so the probabilities actually mean something The part that surprised me was reasoning. LogiQA 0.77 vs 0.59, WinoGrande 0.89 vs 0.66, ARC-Challenge 0.97 vs 0.87, MMLU 0.94 vs 0.87. I assumed it had memorised the benchmarks, so I generated 120 new math word problems with a script using Fable 5.1. It got 0.75 on those, about the same as its 0.72 on GSM8K. Luna with reasoning off got 0.17 on the same problems. Reranking was also a clear win (NFCorpus nDCG@10 0.73 vs 0.63), and Luna needed 1-3 seconds per query there. Where it lost: counting things in a list (0.87 vs 0.99 once Luna could reason) picking 1 of 77 intents (Banking77, 0.81 vs 0.87) asking a question and then its negation did not give probabilities that add up to 1, off by about 0.3 on average in a separate private test with ~100 long documents in one request, it gave high relevance scores to a few completely unrelated ones. None of the clean benchmarks show that. because I know someone will ask. These are public benchmarks, so contamination is possible for any model, and the new math set is my only control for that. The baseline was deliberately held to no/low reasoning. 200 items per task means anything under about 0.05 is noise. Only 7 of the leads are clearly outside the error bars. And it can’t write text, call tools or explain itself, so it doesn’t replace your LLM. It replaces the small classifier, reranker and “is this relevant” calls around it. Repo has the task builders, runner, scorer, my raw per-item results and the full table. Python stdlib only. You can plug in any OpenAI-compatible model as another baseline with one argument, along with the full results : https://github.com/OmarMujahid/jev-decision-bench submitted by /u/LowNefariousness9966
Originally posted by u/LowNefariousness9966 on r/ArtificialInteligence