Command safety: speed is only half the test

What the source reports

Rauch reports improved command-review accuracy and up to an 18-fold p95 speedup over GPT Luna in fx's automatic safety review workflow.

What has been verified

The source post, author attribution, timestamp, and public engagement snapshot were retrieved through the X API on 2026-09-19. This checks the source of the report, not its experimental conclusions. JevArena has not reproduced this report or established a model winner.

Evidence limits

This is the author's report, not a JevArena result. The retrieved post does not provide the command set, error breakdown or full evaluation configuration.

Complete replay inputs are unavailable in the retrieved source. This case study is not a runnable benchmark fixture.

Proposed reproduction protocol

Use a labeled, inert command corpus. Measure unsafe commands incorrectly allowed separately from safe commands incorrectly blocked. Never execute the commands.

1. Obtain permitted inputs, expected labels, and the original evaluation configuration. If they remain unavailable, label any new study as an independent adaptation.
2. Freeze the input set, model versions, prompts, output choices, and measurement conditions before running the comparison.
3. Record actual outputs, failures, sample count, and timing procedure. Evaluate judgment quality separately from latency and cost.
4. Publish the permitted evidence and differences from the original report so others can inspect the result.

This is a proposed study, not a completed experiment. Reading this page makes no model or X API calls.

Help make this reproducible

Have permitted inputs or a documented reproduction? Contribute the source, setup, and evidence for review. Keep new results distinct from the author’s original report.

Contribute evidenceExplore runnable templates