An evaluation · Twin-2K-500 · jev-1.13.0 vs gpt-4.1

Does a native probability beat a verbalized one?

I asked TypeSafe’s Jev, a decision-only model, and GPT-4.1 to play the same 300 survey respondents. Jev returns a probability per option because that is its output type. GPT-4.1 is a chat model, so it was made to fill one into an enforced JSON schema. Six measures separate them.

Jev Noul 6–0 vs GPT-4.1
probabilities

Respondents: 300 of 2,058
Questions: 108
Jev cost: $4.02 billed
GPT-4.1 cost: ~$136

Asked as a Noul, Jev leads GPT-4.1 probabilities on all six measures, at a thirty-fourth of the cost. Plain GPT-4.1 hard answers still score two points higher on accuracy, and return no distribution.
How you ask mattered more than which model you used. Switching the 65 yes/no questions from a two-option Choice to a Noul moves the distribution gap — how far the model’s spread of answers sits from the humans’, 0 identical, 1 as far apart as possible — from 0.1985 to 0.1530, at the same price.
A persona of 620 of the person’s own past answers lifts both models at both levels, and Jev keeps pace with GPT-4.1.
No arm predicts an individual yet. Per-person correlation with the human sits near zero on 15 of the 16 tasks, whatever the persona carries.

What a synthetic survey respondent is

A language model is given a few facts about a real person, walked through questions that person also answered, and scored against those answers. Real panels are slow. A model that could stand in for one would let you pretest a survey. Whether it can is the measurement.

Constructed demo · not a real respondent

Fourteen demographic facts

Twin-2K-500’s cheapest persona is 14 items. The holdout exam is a battery of behavioral-economics tasks the person already answered. Nothing below is a real row.

Region South
Sex Female
Age 42
Education Bachelor’s degree
Race White
Citizen Yes
Marital Married
Religion Protestant
Attendance A few times a year
Party Democrat
Income $75,000–$99,999
Views Moderate
Household 3
Employment Employed full-time

One cell · pricing block · QID9-style

A held-out yes/no, with a piped price

Please consider the following product category: butter. You see Land O Lakes Salted Stick Butter, 16 oz, 4 sticks, priced at $6.49. Would you or would you not purchase this product?

Yes, I would purchase the product
No, I would not purchase the product

Forty of the 108 scored columns are this task. The price is randomized per person, which is why every figure in the repo is averaged by task first.

Illustrative vectors · teaching only

Two ways to get a probability

Jev · native vector

Returns numbers. No prose.

GPT-4.1 · verbalized

Asked to write the same numbers out.

The exam is the human’s own answer

Population first, person second

In this invented cell the human said No. One person gives one answer, so their bar is all-or-nothing — that is the thing every model vector above is scored against. A survey mostly wants the share who said No across everyone, not whether the model picked the right person, which is why the tables lead with a distribution gap rather than accuracy. Humans themselves only reproduce their earlier answers 81.68% of the time.

The data is Toubia et al., Twin-2K-500, CC BY 4.0. I did not collect it.

1 / 4

The ground both models stand on

The same engine walked gpt-4.1 through the whole Twin-2K-500 sample, three times. No Jev in this chapter: it sets the floor and the ceiling everything after it is read against.

Accuracy against the human test–retest ceiling and the paper’s published twin. All three arms land 9 to 12 points below what the humans manage against their own earlier answers. Adding 620 of a person’s own past answers to the same fourteen demographic facts buys +2.66 points; letting the model see its own earlier answers buys less than a quarter of a point.

Source: the grounding panel.

Jev against GPT-4.1

Same 300 respondents, same 108 columns, same stateful demographics persona. Two tests decide it. The first is a conjunction: Jev must win both the 65 yes/no columns and the 43 ordinal ones, and asked as a Choice it trails on the yes/no half. The second takes TypeSafe at its word — System One models are “trained for calibrated decisions” — and sets the bar at 0.05. No arm comes close: Jev Noul 0.1472, Jev Choice 0.2029, GPT-4.1 0.2393. Jev is the better calibrated of the two, by 39%, and still three times over the bar. Every arm failing at once is what makes this a statement about demographics-only grounding rather than about Jev.

Lower is better except accuracy. The hard-answer arm has no vector: its ECE and Brier score a one-hot spike. It is here because it wins accuracy, which is the point about accuracy.

Distribution gap, yes/no: Jev Choice 0.1985 against GPT-4.1 0.1789. Wilcoxon p = 1.0000, GPT-4.1 ahead. The ordinal half goes the other way, 0.6864 against 0.7272, p = 0.031.

Accuracy is last on purpose. The arm that wins it, GPT-4.1 hard answer at 69.32%, is last on the distribution gap.

Source: the planned comparison. Every measure here — accuracy, the two distribution gaps, calibration, Brier — is defined on the metrics page. Reproduce with prob_scoring.py.

Where Jev is the interesting option

Chapter 03 is the comparison as it was first set up. These are the measured results that came after it, and they are why this page leads with Noul. The last paragraph on this page is opinion, and is labeled as such.

“Jev Noul” means Noul on the 65 yes/no columns and Choice on the other 43, since a Noul takes a yes/no condition and cannot be asked of a multi-option question. Making that swap moves the gap from 0.1985 to 0.1530. Saturation at probability zero falls from 16.6% of cells to 0.0%. Same price. Built after seeing the Choice result, so it sets up the next test and does not settle the first one.

Add 620 of the person’s own past answers to the same 14 demographic facts and both levels improve. Not enough to predict an individual: 0.1045 is near zero on most tasks. Both bars hold statefulness fixed, so grounding is the only thing that moved — GPT-4.1 shifts the same way, 0.0945 to 0.1068.

A crosstab asks two questions, and each arm answers only one of them. What did this group say? Jev lands closer to the real answer on all 13 demographic variables, a median distance of 0.233 against GPT-4.1’s 0.295, over all 2,058 respondents. How far apart are two groups? That is the chart, and it goes the other way: Jev spreads its groups to 40% of the real distance, GPT-4.1 to about 110%. A breakdown needs both answers right, and neither arm gives both.

On the 40 pricing columns Jev tracks price more tightly than the humans, at the wrong operating point: it is far too willing to buy. The ranking is sound; the threshold is not.

What one run over the same 300 respondents cost, and how long it took. Jev bills input only and output is free; the GPT-4.1 bar prices that arm's own token counts at OpenAI list rates and is not a bill. Both wall clocks are measured, but at different batch sizes — 16 walks in flight against 50 — so the bars are not a speed test. Per call Jev averaged 0.26 seconds; GPT-4.1 logged no per-call timing.

Author’s take · not a result

I would use Jev. Asked as a Noul it leads GPT-4.1 probabilities on both halves, answers in about a quarter of a second a call, and costs a thirty-fourth as much. For ordinal-scale marginals under cost pressure I would reach for it today, with Noul for anything yes/no.

What this does not say

The 300 are the first 300 rows of the sample — older, whiter, more conservative. Read the paired comparisons. Do not read any absolute number as a population estimate.
The exam and the persona are different subject matter. The persona carries personality and economic-preference answers; the holdout is a cognitive-bias battery, built so that answers should not track who you are. Predicting an individual fails for every arm here, and that is as much the instrument as the models.
Noul was built after the two-option result, so it cannot carry the verdict. Option descriptions were fixed in advance of their own run and changed nothing.
Jev was given the same plain input GPT-4.1 got. TypeSafe’s own prompting levers are mostly untested, not refuted.

The repository The write-up Grounding What this licenses

By jjd-lab, September 2026. Independent work: I have no affiliation with TypeSafe or OpenAI, received no funding, credits or early access from either, and paid for all model usage myself. Neither company had any input into the design or the write-up, and neither has seen it.

The data is Twin-2K-500 on Hugging Face. Jev is TypeSafe’s. Every measure on this page is defined on the metrics page, and CITATION.cff says how to cite this.

Code and write-up MIT. Twin-2K-500 is CC BY 4.0, Toubia et al., arXiv 2505.17479. Figures on this page are generated from site/data/figures.json.