Superscalar Bench

The six aims

Key images: one wordless picture per aim, generated with Codex CLI's built-in image tool from published prompts (how they were made). The text, numbers and ladder on the sheet are drawn by code.

The aim ladder

Each aim is one task; within an aim every model is judged by the same hidden suite. The aims form a ladder ordered by task scope, designed to get harder as it climbs — the panels show what was measured, not a smoothed trend.

	aim	task	hidden tests	reach	first answer	median trust time	median trust cost
	1	mechanical multi-file edit	25	100%	95%	3.5 min	$0.35
	2	spec-complete implementation	73	100%	91%	4.8 min	$0.48
	3	multi-step feature + schema migration	49	100%	98%	6.3 min	$0.62
	4	diagnose-and-fix from a symptom report	37	100%	99%	4.6 min	$0.48
	5	design + build a small language interpreter	96	98%	90%	9.8 min	$0.86
	6	crash-safe transactional key-value store	58	94%	80%	12.9 min	$1.08

Medians are over the 97 model × harness × effort combinations of each aim (n = 3 per cell; Gemini 4 Argon: 18 cells with n = 2; Fable 5.1: one cell with n = 2). A combination that never reached on an aim has no trust cost and is left out of that aim's trust medians — this happens only on aim 6.

Aims 1–4 fall within $0.35–$0.62 and 3.5–6.3 min per verified result without a steady rise: aim 2 trips the first answer more often than aims 3–4, and aim 4 is cheaper than aim 3. Aims 5 and 6 sit above that range — about twice the aims 1–4 median in cost, and two to nearly three times in time. Aim 6 costs more than aim 5 at the median, takes longer and misses more; aims 5 and 6 are the only rungs where runs fail to reach.

What is measured

• Unit: model × harness × effort — the same model behaves differently under different harnesses.
• Reach: the hidden acceptance suite passes completely; no partial credit. First answer = reached with no refinement round.
• The first answer, then up to three refinement rounds (four in all); feedback is failing test names and assertion messages, never the test code. Round budget 90 minutes. Each run works in an isolated copy of the task repository and may run node there; other shell commands are denied.
• Cost is API-equivalent list price, not a subscription charge: Claude Code and Grok Build report it; Codex and Antigravity runs are priced from token counts × published prices. Invalid runs are counted apart from unreached runs — 177 records were set aside (runs under measurement-condition defects that were fixed and re-measured, quota and entitlement refusals, sessions that did no work, and one run whose submission inspected the grader).

The rows for Claude Haiku 5.5, Opus 5, Sonnet 5, Opus 4.8 and GPT-5.6 Sol · Terra · Luna were measured after the first measurement (2026-10-07 → 10-13), Gemini 4 Argon after those (2026-10-16 → 10-27) and GPT-6.1 Terra last (2026-10-27). Every combination ran on all six aims. Claude Code sends each Claude model its own built-in system prompt — that is part of the harness, so a comparison between two Claude models is a comparison of model plus harness prompt, as it is under any harness.

Findings

Recommended effort by aim

The ★ / ☆ marks in the figures. Four steps, per model and aim. (1) Eligible: efforts that reached on the first answer in every run. (2) Exception: an effort that needed refinement still qualifies if its cost or time per verified result is at most half that of the best eligible effort — far beyond the ~20% noise band. (3) Excluded: a lower effort that costs more than 20% more than some higher effort cannot get ★, and one that takes more than 20% longer cannot get ☆ — a lower effort that spends more must not look recommended. (4) Ranked: ★ on every remaining effort within 20% of the cheapest cost per verified result (the tie rule, so ties are all marked), ☆ on every remaining effort within 20% of the fastest that is not already ★. L · M · H · X = low · medium · high · xhigh (max is written Mx). — means no effort passed on the first answer in every run. A cell with a missing cost record gets no ★; Claude Haiku 4.5 has a single effort level and is not listed. † = provisional: a mark on a row with a single run (n=1) is judged on that run and is re-judged once more runs land.

Rows with fewer than three runs: Gemini 4 Argon has two runs per task × effort so far (run 3 of Gemini 4 Argon is being measured; the figures will be updated). Its rankings are mostly reliable, but read a gap under ~20% between its cells as a tie.

All six aims

x = trust cost (log) · y = trust time (log) · each line one model in one harness · points = effort levels (mark shape = vendor: six-ray star Anthropic, hexagon OpenAI, four-point sparkle Google, bold X xAI; a filled mark reached on the first answer in every run, an outlined one needed a refinement round in some run or missed; n=1 models are drawn with a short, faint dashed line and labelled «n=1»; ★ / ☆ = recommended effort, see above, with † on n=1 rows). The legend tags a model's efforts that were not reached, have unknown cost or have partial cost. Lower-left is better. The marker at the top right shows the rung on the ladder.

Limits

• Single observer, single machine — nothing here has been reproduced independently.
• Tasks and hidden suites are not published (kept out of training data and out of reach of the runs), so the benchmark cannot be rerun yet — read it as a report, not a standard.
• Repetitions: n = 3 per cell; Gemini 4 Argon: 18 cells with n = 2; Fable 5.1: one cell with n = 2. Three runs are enough to see the spread, not enough for fine rankings. Rows with fewer than three runs: Gemini 4 Argon has two runs per task × effort so far (run 3 of Gemini 4 Argon is being measured; the figures will be updated). Its rankings are mostly reliable, but read a gap under ~20% between its cells as a tie.
• Run order: Grok Build's third runs on aims 1–5 and all its aim 6 runs were measured on 2026-10-06, after the other rows of the first measurement. The rows added later were measured after that, on different days and under different machine load: Claude Haiku 5.5, Opus 5, Sonnet 5, Opus 4.8 and GPT-5.6 Sol · Terra · Luna on 2026-10-07 → 10-13, Gemini 4 Argon on 2026-10-16 → 10-27, GPT-6.1 Terra on 2026-10-27.
• Versions: the Claude Code CLI version is recorded in the Claude transcripts (runs stopped at the round budget left none). For Codex, before GPT-6.1 Terra a version was recorded only in the third runs of GPT-5.6 Terra and GPT-5.6 Luna (codex-cli 0.160.1); every GPT-6.1 Terra run used codex-cli 0.164.0-alpha.11.1, the build bundled with the VS Code extension, pinned because the standalone 0.162.1 could not start shell commands on this machine. It is the only Codex row on that build and no same-build control was run, so read its time against other Codex rows as a mix of model, build and date; cost comes from token counts and is the safer axis. Model behaviour is as of the measurement dates. Antigravity CLI versions differ between rows: Gemini 4 Argon on 1.3.6/1.4.5/1.4.7; Gemini 3.1 Pro and Gemini 3.8 Flash without a recorded version (those runs predate version recording; the 2026-09-30 adapter probe found 1.2.13).
• Time is the weaker axis: wall time depends on machine load (how many chains ran at once), run order and the serving account. Two Claude accounts were used, switching on 2026-10-03; cost is unaffected, time may not be. The early Claude runs that loaded the machine's plugin hooks (before 2026-10-01 11:00) carry extra seconds per tool call.
• Cost is API-equivalent: what the tokens would cost at list price, not what a subscription plan charges.
• Missing costs: some max-effort runs reached but hit the round budget (Fable 5.1 on aims 5 and 6, Sonnet 5.5 on aim 6). Claude Code writes no cost record for a stopped session, so those cells' means are over the remaining runs, are lower bounds, and are marked «partial cost» in the figures.
• CLI version control: Claude Haiku 5.5 needs Claude Code 2.1.293 or later, so it ran on a side-by-side install of that version; every other Claude row ran on 2.1.284. To check the version itself, Haiku 4.5's default cells on aims 1–3 were run again on 2.1.293 (one run each) and compared with their published means: cost came out about unchanged (the geometric mean of the ratios falls within the ±20% tie band), time lower. Cost is the axis to read here; time also moved with machine load, which was lower for these re-runs than for the original runs.
• Antigravity CLI version control: Gemini 4 Argon ran on Antigravity CLI 1.3.6/1.4.5/1.4.7; Gemini 3.1 Pro and Gemini 3.8 Flash were measured before the CLI version was recorded (the 2026-09-30 adapter probe found 1.2.13). To check the version, 9 existing Gemini 3.8 Flash cells (aims 1, 2, 3 × low / medium / high) were run again on 1.3.6 (one run each) and compared with their published n=3 means: cost 0.82× (geometric mean; 8/9 within the ±20% tie band), time 1.43× (1/9 within the band; by effort low 0.93×, medium 1.79×, high 1.77×). 9/9 re-runs reached, the same as their published cells. CPU busy before each round averaged 24% on the re-runs against 29% on the original runs, so machine load does not explain the slower time. Reading rule: between the Gemini 4 Argon rows and the older Gemini rows the cost axis is comparatively safe, while the time axis can mix a harness-version difference with measurement-date and server-side differences. 9 cells with one run each cannot separate a version effect from the date and server state.
• Restored costs: the cost of GPT-5.6 Luna medium on aim 4 (run 1) and GPT-5.6 Luna max on aim 5 (run 3) was restored from the session log's last cumulative token count × published prices. In the first, the session finished its work and then returned a capacity error, so no final usage event reached the runner. In the second, the round hit the 90-minute budget after the work was already done — the session log ends in a stalled response stream, so the time recorded for that run includes the stall.
• Output cap: Claude Code's default output cap (64,000 tokens per response) ended 41 rounds in 6 cells: Sonnet 5 max on aim 5 (1 of 2 rounds in 1 of its 3 runs; reached on a later round), Opus 4.8 xhigh on aim 6 (7 of 10 rounds; reached on a later round), Opus 4.8 max on aim 6 (6 of 9 rounds; 2 of its 3 runs reached on a later round), Sonnet 5 high on aim 6 (4 of 4 rounds in 1 of its 3 runs; that run did not reach), Sonnet 5 xhigh on aim 6 (11 of 12 rounds; did not reach), Sonnet 5 max on aim 6 (12 of 12 rounds; did not reach). The responses ran out of room while still thinking. No run of the other Claude Code rows (Fable 5, Fable 5.1, Haiku 4.5, Haiku 5.5, Opus 5, Opus 5.5 and Sonnet 5.5) hit the cap. The cap was kept at the harness default, so the capped cells are measured as the harness ships — a cell that did not reach here says more about the default cap than about the model.

Setup details, aim 6 model by model, the measurement defects found and fixed, and an incident where empty grader feedback led a model to probe the evaluator: SuperscalarBench.md →