TypeSafe's Jev Skips Text and Returns a Single Floating-Point Number
A.I. / news
TypeSafe's Jev Skips Text and Returns a Single Floating-Point Number
Jev answers a question with a calibrated probability instead of a sentence, and developer Simon Willison says that opacity makes it a bad tool for ranking job applicants.
TypeSafe AI launched early access to Jev, a model that answers a question with one calibrated number instead of a sentence, founder Diogo Almeida said in a company blog post. The launch went out Sept. 15, and Almeida described it as TypeSafe's first "System One Model," built for automated decisions rather than chat.
What a decision model actually returns
Unlike a large language model, which generates a reply one token at a time, Jev evaluates a set of typed questions against a block of text or data, called a state, and returns each answer with a probability attached, according to TypeSafe's documentation. The three question types are Choice, which picks one option from up to 255 candidates; Score, which places the state on an ordered scale of two to 10 levels the caller defines; and Noul, which returns a 0-to-1 confidence that a single statement is true.
| Question type | Returns | Limit |
|---|---|---|
| Choice | One option, with probabilities | Up to 255 options |
| Score | A point on an ordered scale | 2 to 10 levels |
| Noul | A 0-to-1 confidence value | One yes-or-no statement |
TypeSafe trains Jev with what it calls Reinforcement Learning for Calibrated Decisions, rather than the reinforcement learning from human feedback most large language models use, according to the company's launch post. It prices the model at $0.042 per million input tokens with no charge for output, and its own published figures put end-to-end latency at 70 to 500 milliseconds. TypeSafe's own benchmarks, which The Terminal previously reported no independent lab has reproduced, claim decisions land up to 193.6 times faster and 444.6 times cheaper than comparable large-language-model workflows.
Why Simon Willison calls it a black box
Developer and blogger Simon Willison, who has tracked model releases closely since GPT-3, wrote Sept. 21 that Jev represents "a regression even further towards black box machine learning systems." An LLM's chain-of-thought text at least gives a reader something to inspect, he wrote: "Jev doesn't even give you that: put in all the text you want, the only thing you're going to get back is a floating point number. If Jev marks something as spam, which content signals tipped it off?"
Willison singled out hiring as the use case that worried him most. "I really hope nobody uses Jev to rank job applicants," he wrote, "that floating point number could conceal all manner of unseen bias baked into the models, and experimentally picking that bias apart is going to be a tricky business." His conclusion was practical rather than a call to ban the product: "evals and structured experiments are even more important than they are for regular LLM projects."
The name TypeSafe chose, and the one Willison prefers
TypeSafe's "System One" label references psychologist Daniel Kahneman's distinction between fast, automatic System 1 thinking and slower, deliberate System 2 reasoning, the company said in its launch post. Willison wrote that he prefers the term "decision models," a name he attributed to developer Maggie Appleton, arguing it describes what Jev does without importing a psychological metaphor the product does not actually implement.
TypeSafe's launch landed the same week OpenAI and Anthropic were cutting their own model prices against each other, The Terminal reported. TypeSafe has not published accuracy figures for any of the three question types on a named public benchmark, only the speed and cost comparisons in its launch post, and has not said when, or whether, it plans to release one.
Sources
More in A.I.
- 01Anthropic's Levent Alpöge Claims a Claude-Assisted Riemann Proof, and Lean Checks Are InImperial's Kevin Buzzard and MIT's Andrew Sutherland have examined the formalization, according to Scientific American. No journal review has been reported.
- 02Thinking Machines Ships Inkling 2 Mini, a 26.4B Apache 2.0 Model Sized for One DGX SparkThe model card lists 3.4B active parameters, a 1M-token window and a 20GB checkpoint, but does not say who ran most of its benchmark rows.
- 03Anthropic Closed Its Debut at $97.17, 17% Over the $83 IPO Price, as Nasdaq Fell 2%The $81.34 billion offering opened 34% higher, then gave back most of the gain while Nvidia, AMD and Micron dropped.
- 04Google's Gemini Tool Retrieval Preview Adds 26 Tokens Per Idle Tool and Publishes No Savings FigureAnthropic and OpenAI each reported token reductions for the same defer_loading flag. Google's November 19 post reports none.