Pre-registration: Opus 5.5 short tasks (v3)

The plan for one benchmark run: the tasks, the number of pairs, the cost basis and the analysis. It was committed before the run's first pair.

# v3 run, Opus 5.5 short tasks: pre-registration — 2026-09-29

Written and pushed before the first pair. No pair has run on this commit.

## Freeze

- Gateway commit: `408333a48d082830f5813b3f539ec03993830a89` ([edited: the fix's internal name and one clause removed] no re-read of this conversation's own
  memory rows). [edited: the gateway Worker's build] succeeded at
  2026-09-29T11:00:55Z.
- Real-client smoke on this commit: 8/8 requests ok before the deploy and 8/8 after, 0 upstream 4xx
  ([edited: link to an unpublished note removed]).
- Client: Claude Code 2.1.283, `claude-opus-5-5`.
- Later commits on `main` may deploy during the run. A pair counts only if the deployed commit differs from the freeze
  in `docs/`, `scripts/e2e/`, `site/` or a root `*.md` file only. The gateway bundle does not read these; `site/` is a
  separate Worker (`context-mode`). This commit widens the old rule (`docs/`, `scripts/e2e/`) by `site/` and root
  `*.md`, because `4ba8b4ce` changed the root `README.md` and the landing page is being edited in parallel. Test:
  `scripts/e2e/start-cost-ab-lib.test.mjs`, written first and seen to fail.

## Pairs

tests 16, log 16, git 16, grep 16, json 12, edit 12 (88 measured pairs). One warm-up pair per workload runs first and
is discarded. The arm order alternates by pair (even: plain then gateway; odd: gateway then plain). No pairs are added
after seeing data. Cap $30.

## Command

```
F=408333a48d082830f5813b3f539ec03993830a89
node scripts/e2e/start-cost-ab.mjs --model claude-opus-5-5 --freeze $F --only tests,log,git,grep,json,edit \
  --n 12 --n-for tests=16,log=16,git=16,grep=16 --conc 4 --cap 30 \
  --out docs/verify/launch-ab-2026-09-29/v3-opus55-short.json
python3 docs/verify/launch-ab-2026-09-29/opus55-short-table.py docs/verify/launch-ab-2026-09-29/v3-opus55-short.json
```

## Rules

- The harness reads the deployed commit with `gh api` (newest `main` commit whose [edited: the gateway Worker's build]
  check succeeded) before and after every pair. A pair during which it changed is invalid, kept in the file, and runs
  again. A deployed commit with other gateway code stops the run.
- A 429 makes the pair invalid; the harness waits, then runs it again.
- Every run: fresh `CLAUDE_CONFIG_DIR`, empty strict MCP map, no plugin. Gateway arm: one disposable tenant, preflighted.

## Cost basis

Per cost-basis.md and [edited: a cost script in the gateway repository]. Plain: the client's `total_cost_usd`.
Gateway: the sum of [edited: the gateway's per-call cost record] over the run's sessions, hidden calls included. A gateway run whose ops
rows hold fewer tokens than the client saw in any class is invalid (`ops_below_client_tokens`). List-price equivalent
(OAuth subscription credential).

## Analysis (fixed now)

On every measured valid pair, no outlier dropped:

- paired % difference, gateway against plain (mean(gw − plain) ÷ mean(plain)), with the 95% t interval. "less" or
  "more" only when the interval excludes zero; otherwise "same";
- median paired % difference;
- wins: pairs where the gateway cost less;
- success per arm (the workload's oracle).

What would count against the freeze: a lower success rate on any row, or any upstream 400.

## Result

Written after the run, below this line.