Example questions

Guess first and reveal a recorded result, or run a fresh comparison with your own key.

Real community tests

Actual inputs from @_pi0_’s Jev experiments on X. Click to try them yourself.

Export limit

The export stops at 1000 rows. Is that a limit I can raise, or is it broken?

Author’s recorded result

Label disagreement: the author's native TypeSafe run (jev-1.13.0) chose question with probability 0.95; the author's label was bug. The input does not establish whether the limit is intended, so this is a disputed label, not a verified model error.

2026-09-18 · Author-reported, not independently reproduced. Reruns use JevArena’s prompt format, not an exact replay. Raw record: ambiguous-5 · MIT.

Retry defaults

Please document the retry defaults, or else make them configurable.

Try without a key Run this question

Author’s recorded result

Label disagreement: the author's native TypeSafe run (jev-1.13.0) chose feature with probability 0.67; the author's label was docs. Both documentation and configurability are requested. The appropriate label depends on the triage policy; no independent correctness label is assigned here.

2026-09-18 · Author-reported, not independently reproduced. Reruns use JevArena’s prompt format, not an exact replay. Raw record: ambiguous-23 · MIT.

Safari login

Does login work in Safari? A few of my users mention something odd there.

Try without a key Run this question

Author’s recorded result

Label disagreement: the author's native TypeSafe run (jev-1.13.0) chose bug with probability 0.70; the author's label was question. The report contains no reproduction or confirmed failure. This is an ambiguous triage decision, not a verified model error.

2026-09-18 · Author-reported, not independently reproduced. Reruns use JevArena’s prompt format, not an exact replay. Raw record: ambiguous-26 · MIT.

A surprising remainder

What is 6^3 mod 89?

Try without a key Run this question

Author’s recorded result

The author's native TypeSafe run (jev-1.13.0) chose 53; a separate Vercel run chose 38. The correct remainder is 38. This is an observed disagreement, not proof of a provider effect.

2026-09-18 · Author-reported, not independently reproduced. Reruns use JevArena’s prompt format, not an exact replay. Raw record: modexp-1 · MIT.

A bigger power

What is 19^9 mod 7?

Try without a key Run this question

Author’s recorded result

The author's native TypeSafe run (jev-1.13.0) and Vercel run both chose 6, the correct remainder. The native run assigned its choice probability 0.37.

2026-09-18 · Author-reported, not independently reproduced. Reruns use JevArena’s prompt format, not an exact replay. Raw record: modexp-6 · MIT.

More starter templates

Templates

Original starter questions you can try now, not measured model results.

An urgent email

How should this message be classified?

Try this question

A small numerical difference

Which number is larger, 9.11 or 9.9?

Try this question

Two words, exactly

Reply with exactly two words naming a primary color and a shape. No punctuation.

Try this question

Does the evidence support it?

Is the revenue conclusion supported by the provided evidence?

Try this question

An answer that orders the judge

What is 6 multiplied by 7?

Try this question

Community reports and background

Community reports

These posts do not include complete test inputs. They are references, not runnable cases.

4 community references · source checked, not reproduced

A faster classifier — on whose evaluation?

Author-reported test · Source checked · not reproduced

Malte Ubl (@cramforce) · 2026-09-16

Ubl reports that Jev reached the ceiling of an existing classifier evaluation and ran six times faster than its Gemini 2.5 Flash Lite baseline.

Read case study: A faster classifier — on whose evaluation?Original post on X

Source snapshot and engagement

208,184 views · 1,117 likes · 37 reposts
X API snapshot: 2026-09-19. Popularity is not evidence of correctness.

Command safety: speed is only half the test

Author-reported test · Source checked · not reproduced

Guillermo Rauch (@rauchg) · 2026-09-16

Rauch reports improved command-review accuracy and up to an 18-fold p95 speedup over GPT Luna in fx's automatic safety review workflow.

Read case study: Command safety: speed is only half the test Original post on X

Source snapshot and engagement

418,197 views · 3,926 likes · 169 reposts
X API snapshot: 2026-09-19. Popularity is not evidence of correctness.

Computer use: a demo is not a success rate

Demo · Source checked · not reproduced

Cua (@trycua) · 2026-09-17

Cua announces a development preview combining Jev with Cua Driver for desktop computer use, with a link to its public repository.

Read case study: Computer use: a demo is not a success rate Original post on X

Source snapshot and engagement

375,542 views · 2,591 likes · 218 reposts
X API snapshot: 2026-09-19. Popularity is not evidence of correctness.

A decision model, not a chat-model replacement

Discussion · Source checked · not reproduced

Nathan Flurry (@NathanFlurry) · 2026-09-16

Flurry frames Jev as a classifier-like decision tool rather than a replacement for general-purpose GPT or Claude models.

Read case study: A decision model, not a chat-model replacement Original post on X

Source snapshot and engagement

654,759 views · 7,460 likes · 403 reposts
X API snapshot: 2026-09-19. Popularity is not evidence of correctness.

Selected from public links, not a complete or ranked survey of X. Summaries are editorial paraphrases; linked sources retain their original rights. No posts, images or videos are republished in full. To request a correction or removal, open an issue.

Found a surprising failure?

Share the original question, the judgment, and evidence for what went wrong. Good counterexamples make this collection more useful.

Contribute a case