Pre-registration: Long agentic coding session, 1M window, amendment 4 - Source file: docs/verify/recall-ab-2026-10/amendment4/PREREG.md - Commit that added it: 5e8b5c85, 2026-10-03T12:07:40Z - Evidence: /benchmarks/long-session/amendment-4/ab.json (its "prereg_check" names the same commit and the first pair's start time). - This copy is the file as it stood at that commit. - Edits: private fields only (tenant ids, local paths, the internal Worker name and internal endpoint names), each marked [edited: ...]. The workload, the pass rule and the analysis are as committed. --- # Recall A/B, amendment 4: pre-registration (written and pushed before any run) Earlier: `../final/PREREG.md` (amendments 2 and 3) and `../final/README.md` (4 valid pairs, ship rule FAIL). This file changes only what is listed here. Everything else stays as in `../final/PREREG.md`: the workload (30 steps, seed 424242), the client, the model, the task checks, the re-ask checker, the exact checker (`probeExact`, with the banner line set aside), the validity rule (`finalPairValid`), the ship rule and the eight lessons. ## 1. Why In the final A/B the recall arm answered 5.75 of 10 memory questions exactly, plain 10 of 10. Every miss was a decline: the question forbade every tool, including `context-mode-search`, and that search is how Recall is built to bring back what it moved to the archive. The model said the content was in the archive and did not answer. This amendment lets the gateway arm use that one tool on the question turns. ## 2. What changes - **Gateway arm, question turns only.** The tool sentence of the question is `PROBE_SEARCH_ONLY` (`natural-lib.mjs`): the model may use one tool, `context-mode-search`, to search the archive of this conversation, and no other tool. The rest of the question text is the same as before, word for word. The `PreToolUse` hook gets that tool's name as its allowed tool (`probe-gate-hook.py`, second argument): while the probe marker exists, a call to it passes and any other call is refused. (The gateway runs this tool itself, so the client hook may never see it; the hook still refuses every client tool.) - **Plain arm: unchanged.** Same question text, same `PROBE_NO_TOOLS` sentence, same hook with no allowed tool. Plain has no such tool, and its whole history is still in its context. - **The tool list stays the same on every turn of every arm (lesson 3).** Nothing is added or removed on a question turn. A change makes the pair invalid, as before. - **Checker: exact match, unchanged.** - **Dry run (`--plan dry4`, at most $3).** Arms `plain` and `recall-b4k` (the same levers with `;b=4000`). Steps: `s01`, `r1`, and the first constant question (`q-const-series-max`; its file is read in `r1`, so at B = 4,000 it is archived before the question and only the search can bring it back). It must show the eight lessons as in `../final/PREREG.md` §5, plus: the recorder below sees the gateway's search records, and the plain question turn makes 0 tool calls. ## 3. What is recorded (new) - **Per question, gateway arm:** whether the model called `context-mode-search` (source: the gateway's own exec records, `[edited: an internal endpoint]`, tool `context-mode-search`, matched to the question by time), how many calls (docid reads and queries apart), and whether the result held the answer: - `found_recorded`: for a docid read, the archived doc itself (read whole with `[edited: an internal endpoint]`) holds the value; for a query, one of the result's recorded top snippets holds it (the record keeps at most 2 snippets of 140 characters, so this is a lower bound); - `found_replay`: the same query, run again after the arm with `[edited: an internal endpoint]` (k = 10), holds the value. An estimate, not the call's own result. Also whether the client stream showed the call. Booleans and counts only: no value, prompt, answer or snippet is written. - **Recall's signed net** from the ledger (`/__recall-stats?period=24h` `saved_usd_recall`: positive = Recall saved, negative = Recall cost more), per gateway arm. - **Gateway without Recall (estimate)** per gateway arm = the arm's $ + Recall's signed net. It comes from the ledger's counterfactual, not from a measured third arm. ## 4. Pairs, caps and budget - 4 pairs, interleaved as before: pair 1 starts plain, pair 2 recall, pair 3 plain, pair 4 recall; the second arm starts 5 s after the first. Seed 424242. One process per pair. At most 1 attempt; one restart only if the harness itself breaks before any pair completes. - On the deployed commit. The deploy identity (main head, [edited: the gateway Worker's build check] build id, Worker version) is read before each pair and after every step; any change stops the pair and makes it invalid. - **Budget: $40 approved, hard cap $45** for everything in this phase (dry run plus pairs), summed in `amendment4/spend.json` and enforced by `--total 45` (an arm stops before a step when the phase total is within $1 of it). - **Cap per arm: $9.00, both arms, all pairs** (an arm stops before a step when its spend is within $0.60 of it). The final A/B's highest arm was $7.29; the final cap of $10.07 does not fit 8 arms in $45. If the dry run projects a plain arm above $8.40, the run stops before pair 1 and this is reported, not tuned. - An arm stopped by a cap makes its pair invalid, so the ship rule fails. ## 5. Ship rule (unchanged) For each of the 4 pairs: recall exact answers >= plain exact answers, recall $ < plain $, task checks equal, and 0 upstream 4xx in the recall arm. Ship = all 4 pairs present, valid and passing. Anything else is FAIL. ## 6. Commands ``` CM_CLAUDE=[edited: the local claude binary] node scripts/e2e/recall-ab/natural.mjs --plan dry4 --search-probes \ --arms plain,recall-b4k --cap-plain 2 --cap-recall 1.5 --total 45 --out docs/verify/recall-ab-2026-10/amendment4 CM_CLAUDE=[edited: the local claude binary] node scripts/e2e/recall-ab/natural.mjs --pair --first --seed 424242 \ --search-probes --cap-plain 9 --cap-recall 9 --total 45 --out docs/verify/recall-ab-2026-10/amendment4 node scripts/e2e/recall-ab/final-summarize.mjs --dir docs/verify/recall-ab-2026-10/amendment4 --pairs 4 ```