{
  "schema": "context-mode/memory-eval/v1",
  "source_commit": "230fdd44",
  "gateway_commit": "9d047fb4",
  "date": "2026-09-29",
  "client": "Claude Code 2.1.283 (claude -p), a fresh config folder per run",
  "model": "claude-opus-5-5",
  "effort": "low",
  "scenario": {
    "session_1": "FYI, some context about our setup. Our production deploy target is Hetzner Cloud, Falkenstein region. Our on-call engineer is Selin Aydin. Our CI runs on CircleCI. Just acknowledge in one short sentence.",
    "session_2": "FYI, an update on our setup. We moved production: our deploy target is now Render, Oregon region. Our on-call engineer is now Baran Kaya. Our CI now runs on Buildkite. Just acknowledge in one short sentence.",
    "probe_suffix": " Answer in one or two short sentences. If you do not know, say you do not know.",
    "probes": [
      {
        "id": "P1",
        "kind": "now",
        "question": "What is our production deploy target now?"
      },
      {
        "id": "P2",
        "kind": "before",
        "question": "What was our production deploy target before we moved it?"
      },
      {
        "id": "P3",
        "kind": "now",
        "question": "Who is our on-call engineer now?"
      },
      {
        "id": "P4",
        "kind": "before",
        "question": "Who was our on-call engineer before the current one?"
      },
      {
        "id": "P5",
        "kind": "now",
        "question": "What do we run CI on now?"
      },
      {
        "id": "P6",
        "kind": "before",
        "question": "What did we run CI on before our current CI system?"
      },
      {
        "id": "P7",
        "kind": "check",
        "question": "Is our production deploy target still Hetzner?"
      },
      {
        "id": "P8",
        "kind": "all",
        "question": "List our current production deploy target, on-call engineer and CI system."
      }
    ]
  },
  "arms": {
    "on": "Context Mode gateway, default flags: memory on.",
    "off": "Context Mode gateway with memory:false. The gateway adds no memory; context-mode-search over stored turns stays available.",
    "none": "Context Mode gateway with memory:false and entitled:false. As 'off'; context-mode-search stays available.",
    "plain": "Claude Code talks to Anthropic direct. No gateway, no memory."
  },
  "grading": "correct: an exact string check on the new value (now, check and all probes) or the old value (before probes). old_value_as_current: read by hand from each answer; every answer that names an old value names it as the past value. client_tool_calls: tool calls the client ran (none in any probe). gateway_search_calls: context-mode-search calls the model made through the gateway. memory_records: records in the memory block the gateway added, per request. usd and secs: as the client reported them.",
  "runs": [
    {
      "id": "C",
      "role": "run of record",
      "arms": [
        "on",
        "off",
        "none",
        "plain"
      ],
      "results": [
        {
          "probe": "P1",
          "arm": "on",
          "correct": true,
          "old_value_as_current": false,
          "answer": "Our production deploy target is now Render, Oregon region. This comes from the newest stored context-mode memory record (edge_id `fe-704f97787c424d`), which replaces the old Hetzner Cloud, Falkenstein target.\n\n⏺ context-mode · memory +1 record about user, production",
          "client_tool_calls": 0,
          "gateway_search_calls": 0,
          "memory_records": 1,
          "usd": 0.02276,
          "secs": 4.7
        },
        {
          "probe": "P1",
          "arm": "off",
          "correct": true,
          "old_value_as_current": false,
          "answer": "Your production deploy target is now Render, in the Oregon region. This change came in an update on 2026-09-29. Before that, the target was Hetzner Cloud, Falkenstein.",
          "client_tool_calls": 0,
          "gateway_search_calls": 1,
          "memory_records": 0,
          "usd": 0.00619,
          "secs": 5.7
        },
        {
          "probe": "P1",
          "arm": "none",
          "correct": true,
          "old_value_as_current": false,
          "answer": "Your production deploy target is now Render, in the Oregon region. On 2026-09-29, you told me that it changed from Hetzner Cloud (Falkenstein) to Render.",
          "client_tool_calls": 0,
          "gateway_search_calls": 1,
          "memory_records": 0,
          "usd": 0.00622,
          "secs": 7.1
        },
        {
          "probe": "P2",
          "arm": "on",
          "correct": true,
          "old_value_as_current": false,
          "answer": "Before the move, the production deploy target was Hetzner Cloud in the Falkenstein region (context-mode memory, edge_id fe-b48f6546798c4a). Now the target is Render in the Oregon region (edge_id fe-d861db41f70046).\n\n⏺ context-mode · memory +1 record about user, production",
          "client_tool_calls": 0,
          "gateway_search_calls": 0,
          "memory_records": 1,
          "usd": 0.02317,
          "secs": 5.4
        },
        {
          "probe": "P2",
          "arm": "off",
          "correct": true,
          "old_value_as_current": false,
          "answer": "The production deploy target before the move was Hetzner Cloud, in the Falkenstein region. On 2026-09-29, you moved it to Render, in the Oregon region.",
          "client_tool_calls": 0,
          "gateway_search_calls": 1,
          "memory_records": 0,
          "usd": 0.00685,
          "secs": 5.5
        },
        {
          "probe": "P2",
          "arm": "none",
          "correct": true,
          "old_value_as_current": false,
          "answer": "Before you moved it, your production deploy target was Hetzner Cloud, in the Falkenstein region. On 2026-09-29, you told me that production now deploys to Render, in the Oregon region.",
          "client_tool_calls": 0,
          "gateway_search_calls": 1,
          "memory_records": 0,
          "usd": 0.00703,
          "secs": 5.4
        },
        {
          "probe": "P3",
          "arm": "on",
          "correct": true,
          "old_value_as_current": false,
          "answer": "Your on-call engineer is now **Baran Kaya**. Your most recent setup update says this, and it replaces Selin Aydin (source: context-mode memory, edge_id=fe-d861db41f70046).\n\n⏺ context-mode · memory +1 record about user, on-call engineer, production",
          "client_tool_calls": 0,
          "gateway_search_calls": 0,
          "memory_records": 1,
          "usd": 0.02275,
          "secs": 4.8
        },
        {
          "probe": "P3",
          "arm": "off",
          "correct": true,
          "old_value_as_current": false,
          "answer": "Your on-call engineer is Baran Kaya. Your update from 2026-09-29 gave this name, and it replaced Selin Aydin.",
          "client_tool_calls": 0,
          "gateway_search_calls": 1,
          "memory_records": 0,
          "usd": 0.0063,
          "secs": 6.4
        },
        {
          "probe": "P3",
          "arm": "none",
          "correct": true,
          "old_value_as_current": false,
          "answer": "The on-call engineer is Baran Kaya. This is from your most recent update on 2029-09-29, which replaced Selin Aydin.",
          "client_tool_calls": 0,
          "gateway_search_calls": 1,
          "memory_records": 0,
          "usd": 0.00635,
          "secs": 6
        },
        {
          "probe": "P4",
          "arm": "on",
          "correct": true,
          "old_value_as_current": false,
          "answer": "The on-call engineer before Baran Kaya was Selin Aydin. You told me this in an earlier message; context-mode memory has it in edge_id fe-b48f6546798c4a, and the change to Baran Kaya is in fe-d861db41f70046.\n\n⏺ context-mode · memory +1 record about user, on-call engineer +2 more",
          "client_tool_calls": 0,
          "gateway_search_calls": 0,
          "memory_records": 1,
          "usd": 0.02351,
          "secs": 4.9
        },
        {
          "probe": "P4",
          "arm": "off",
          "correct": true,
          "old_value_as_current": false,
          "answer": "Your on-call engineer before the current one (Baran Kaya) was Selin Aydin. The update on 2026-09-29 replaced Selin Aydin with Baran Kaya.",
          "client_tool_calls": 0,
          "gateway_search_calls": 1,
          "memory_records": 0,
          "usd": 0.00698,
          "secs": 6.3
        },
        {
          "probe": "P4",
          "arm": "none",
          "correct": true,
          "old_value_as_current": false,
          "answer": "Your on-call engineer before Baran Kaya was Selin Aydin. Your update on 2026-09-29 replaced Selin Aydin with Baran Kaya.",
          "client_tool_calls": 0,
          "gateway_search_calls": 1,
          "memory_records": 0,
          "usd": 0.00687,
          "secs": 5.3
        },
        {
          "probe": "P5",
          "arm": "on",
          "correct": true,
          "old_value_as_current": false,
          "answer": "Your CI now runs on Buildkite. This replaces CircleCI. Source: context-mode memory, edge_id=fe-d861db41f70046, 2026-09-30.\n\n⏺ context-mode · memory +1 record about user",
          "client_tool_calls": 0,
          "gateway_search_calls": 0,
          "memory_records": 1,
          "usd": 0.02185,
          "secs": 4.6
        },
        {
          "probe": "P5",
          "arm": "off",
          "correct": true,
          "old_value_as_current": false,
          "answer": "You run CI on Buildkite now. You moved to it from CircleCI on 2026-09-29.",
          "client_tool_calls": 0,
          "gateway_search_calls": 1,
          "memory_records": 0,
          "usd": 0.00578,
          "secs": 5.5
        },
        {
          "probe": "P5",
          "arm": "none",
          "correct": true,
          "old_value_as_current": false,
          "answer": "Your CI now runs on Buildkite. You told me this in your most recent update on 2026-09-29. Before that, your CI ran on CircleCI.",
          "client_tool_calls": 0,
          "gateway_search_calls": 2,
          "memory_records": 0,
          "usd": 0.01977,
          "secs": 6.6
        },
        {
          "probe": "P6",
          "arm": "on",
          "correct": true,
          "old_value_as_current": false,
          "answer": "Before your current CI system (Buildkite), your CI ran on CircleCI. Source: your earlier setup note (edge_id=fe-b48f6546798c4a), and the later update (edge_id=fe-d861db41f70046).\n\n⏺ context-mode · memory +1 record about user, selin aydin, assistant",
          "client_tool_calls": 0,
          "gateway_search_calls": 0,
          "memory_records": 1,
          "usd": 0.02234,
          "secs": 4.4
        },
        {
          "probe": "P6",
          "arm": "off",
          "correct": true,
          "old_value_as_current": false,
          "answer": "Before Buildkite, you ran CI on CircleCI. I found this only in an earlier answer that I wrote on 2026-09-29, not in a primary record.",
          "client_tool_calls": 0,
          "gateway_search_calls": 1,
          "memory_records": 0,
          "usd": 0.00634,
          "secs": 6
        },
        {
          "probe": "P6",
          "arm": "none",
          "correct": true,
          "old_value_as_current": false,
          "answer": "Before your current CI system (Buildkite), your CI ran on CircleCI. You told me about this change on 2026-09-29.",
          "client_tool_calls": 0,
          "gateway_search_calls": 2,
          "memory_records": 0,
          "usd": 0.02196,
          "secs": 7.8
        },
        {
          "probe": "P7",
          "arm": "on",
          "correct": true,
          "old_value_as_current": false,
          "answer": "No. The production deploy target is now Render, in the Oregon region. It was Hetzner Cloud in Falkenstein before (source: your update in context-mode memory, edge_id=fe-d861db41f70046).\n\n⏺ context-mode · memory +1 record about user, production deploy, production",
          "client_tool_calls": 0,
          "gateway_search_calls": 0,
          "memory_records": 1,
          "usd": 0.02314,
          "secs": 6
        },
        {
          "probe": "P7",
          "arm": "off",
          "correct": true,
          "old_value_as_current": false,
          "answer": "No. On 2026-09-29, you moved the production deploy target from Hetzner Cloud (Falkenstein) to Render (Oregon region), as my notes from past sessions show.",
          "client_tool_calls": 0,
          "gateway_search_calls": 1,
          "memory_records": 0,
          "usd": 0.00687,
          "secs": 6.1
        },
        {
          "probe": "P7",
          "arm": "none",
          "correct": true,
          "old_value_as_current": false,
          "answer": "No. On 2026-09-29, you told me that production moved from Hetzner Cloud (Falkenstein) to Render in the Oregon region.",
          "client_tool_calls": 0,
          "gateway_search_calls": 1,
          "memory_records": 0,
          "usd": 0.00669,
          "secs": 5.2
        },
        {
          "probe": "P8",
          "arm": "on",
          "correct": true,
          "old_value_as_current": false,
          "answer": "Production deploys to Render in the Oregon region, the on-call engineer is Baran Kaya, and CI runs on Buildkite. This is from your latest setup update in context-mode memory (edge_id `fe-d861db41f70046`), which replaces the earlier values: Hetzner Falkenstein, Selin Aydin and CircleCI.\n\n⏺ context-mode · memory +1 record about user, production +2 more",
          "client_tool_calls": 0,
          "gateway_search_calls": 0,
          "memory_records": 1,
          "usd": 0.02414,
          "secs": 5.9
        },
        {
          "probe": "P8",
          "arm": "off",
          "correct": true,
          "old_value_as_current": false,
          "answer": "Your production deploy target is Render (Oregon region), your on-call engineer is Baran Kaya, and your CI system is Buildkite. I found these facts only in my earlier answers from 2026-09-29, not in a primary record, so please do a check of them against your records.",
          "client_tool_calls": 0,
          "gateway_search_calls": 2,
          "memory_records": 0,
          "usd": 0.02346,
          "secs": 6.9
        },
        {
          "probe": "P8",
          "arm": "none",
          "correct": true,
          "old_value_as_current": false,
          "answer": "Your production deploy target is Render (Oregon region), your on-call engineer is Baran Kaya, and your CI system is Buildkite. You gave me this update on 2026-09-29, and it replaced Hetzner Cloud, Selin Aydin and CircleCI.",
          "client_tool_calls": 0,
          "gateway_search_calls": 2,
          "memory_records": 0,
          "usd": 0.02528,
          "secs": 7.3
        },
        {
          "probe": "P1",
          "arm": "plain",
          "correct": false,
          "old_value_as_current": false,
          "answer": "I don't know. I have no saved memory or project info about your production deploy target.",
          "client_tool_calls": 0,
          "gateway_search_calls": 0,
          "memory_records": 0,
          "usd": 0.03027,
          "secs": 0
        },
        {
          "probe": "P2",
          "arm": "plain",
          "correct": false,
          "old_value_as_current": false,
          "answer": "I don't know. Nothing in my memory or this session records where production was deployed before the move.",
          "client_tool_calls": 0,
          "gateway_search_calls": 0,
          "memory_records": 0,
          "usd": 0.01817,
          "secs": 0
        },
        {
          "probe": "P3",
          "arm": "plain",
          "correct": false,
          "old_value_as_current": false,
          "answer": "I don't know who's on call right now. I don't have any on-call schedule or rotation information for your team.",
          "client_tool_calls": 0,
          "gateway_search_calls": 0,
          "memory_records": 0,
          "usd": 0.01796,
          "secs": 0
        },
        {
          "probe": "P4",
          "arm": "plain",
          "correct": false,
          "old_value_as_current": false,
          "answer": "I don't know. I have no saved memory or other information about your on-call rotation.",
          "client_tool_calls": 0,
          "gateway_search_calls": 0,
          "memory_records": 0,
          "usd": 0.02978,
          "secs": 0
        },
        {
          "probe": "P5",
          "arm": "plain",
          "correct": false,
          "old_value_as_current": false,
          "answer": "I don't know. I have no saved memory about your CI setup, and this directory isn't a git repo with CI config I could check.",
          "client_tool_calls": 0,
          "gateway_search_calls": 0,
          "memory_records": 0,
          "usd": 0.01878,
          "secs": 0
        },
        {
          "probe": "P6",
          "arm": "plain",
          "correct": false,
          "old_value_as_current": false,
          "answer": "I don't know. I have no saved memories or project records that say what CI system you used before your current one.",
          "client_tool_calls": 0,
          "gateway_search_calls": 0,
          "memory_records": 0,
          "usd": 0.01823,
          "secs": 0
        },
        {
          "probe": "P7",
          "arm": "plain",
          "correct": false,
          "old_value_as_current": false,
          "answer": "I don't know. I have no saved memory or project info about your production deploy target.",
          "client_tool_calls": 0,
          "gateway_search_calls": 0,
          "memory_records": 0,
          "usd": 0.01808,
          "secs": 0
        },
        {
          "probe": "P8",
          "arm": "plain",
          "correct": false,
          "old_value_as_current": false,
          "answer": "I don't know your production deploy target, on-call engineer or CI system. I have no saved memories about them, and this directory isn't a git repository that would show them.",
          "client_tool_calls": 0,
          "gateway_search_calls": 0,
          "memory_records": 0,
          "usd": 0.01866,
          "secs": 0
        }
      ],
      "store_after_session_2": [
        {
          "edge_id": "fe-a3cad24e286841",
          "subject": "production",
          "predicate": "on_call_engineer",
          "object": "Baran Kaya",
          "kind": "fact",
          "tier": "inferred",
          "source": "ai-extract",
          "valid_at": "2026-09-29T21:15:14.830Z",
          "invalid_at": "",
          "created_at": "2026-09-29T21:15:14.830Z"
        },
        {
          "edge_id": "fe-84dfe9d35b184d",
          "subject": "ci",
          "predicate": "runs_on",
          "object": "Buildkite",
          "kind": "fact",
          "tier": "asserted",
          "source": "mcp-remember",
          "valid_at": "2026-09-29T21:15:09.150Z",
          "invalid_at": "",
          "created_at": "2026-09-29T21:15:09.150Z"
        },
        {
          "edge_id": "fe-108138433bea49",
          "subject": "on-call engineer",
          "predicate": "is",
          "object": "Baran Kaya",
          "kind": "fact",
          "tier": "asserted",
          "source": "mcp-remember",
          "valid_at": "2026-09-29T21:15:09.150Z",
          "invalid_at": "",
          "created_at": "2026-09-29T21:15:09.150Z"
        },
        {
          "edge_id": "fe-704f97787c424d",
          "subject": "production",
          "predicate": "deployed_to",
          "object": "Render, Oregon region",
          "kind": "fact",
          "tier": "asserted",
          "source": "mcp-remember",
          "valid_at": "2026-09-29T21:15:09.150Z",
          "invalid_at": "",
          "created_at": "2026-09-29T21:15:09.150Z"
        },
        {
          "edge_id": "fe-4c27b02899a74b",
          "subject": "ci",
          "predicate": "runs_on",
          "object": "CircleCI",
          "kind": "fact",
          "tier": "inferred",
          "source": "ai-extract",
          "valid_at": "2026-09-29T21:14:38.716Z",
          "invalid_at": "2026-09-29T21:15:09.150Z",
          "created_at": "2026-09-29T21:14:38.716Z"
        },
        {
          "edge_id": "fe-75df046b54ad4a",
          "subject": "on-call engineer",
          "predicate": "is",
          "object": "Selin Aydin",
          "kind": "fact",
          "tier": "inferred",
          "source": "ai-extract",
          "valid_at": "2026-09-29T21:14:38.716Z",
          "invalid_at": "2026-09-29T21:15:09.150Z",
          "created_at": "2026-09-29T21:14:38.716Z"
        },
        {
          "edge_id": "fe-dcc3e01b1be241",
          "subject": "production",
          "predicate": "deployed_to",
          "object": "Hetzner Cloud, Falkenstein region",
          "kind": "fact",
          "tier": "inferred",
          "source": "ai-extract",
          "valid_at": "2026-09-29T21:14:38.716Z",
          "invalid_at": "2026-09-29T21:15:09.150Z",
          "created_at": "2026-09-29T21:14:38.716Z"
        }
      ],
      "summary": {
        "on": {
          "correct": 8,
          "probes": 8,
          "old_value_as_current": 0,
          "gateway_search_calls": 0,
          "usd": 0.18366
        },
        "off": {
          "correct": 8,
          "probes": 8,
          "old_value_as_current": 0,
          "gateway_search_calls": 9,
          "usd": 0.06877
        },
        "none": {
          "correct": 8,
          "probes": 8,
          "old_value_as_current": 0,
          "gateway_search_calls": 11,
          "usd": 0.10017
        },
        "plain": {
          "correct": 0,
          "probes": 8,
          "old_value_as_current": 0,
          "gateway_search_calls": 0,
          "usd": 0.16993
        }
      }
    },
    {
      "id": "B",
      "role": "same design, new accounts",
      "arms": [
        "on",
        "off"
      ],
      "results": [
        {
          "probe": "P1",
          "arm": "on",
          "correct": true,
          "old_value_as_current": false,
          "answer": "Our production deploy target is now **Render, Oregon region**. This comes from your update, \"We moved production\", in context-mode memory (edge_id=fe-df57c2f179a34b). That update replaces the old target, Hetzner Cloud, Falkenstein.\n\n⏺ context-mode · memory +1 record about user, production +2 more",
          "client_tool_calls": 0,
          "gateway_search_calls": 0,
          "memory_records": 1,
          "usd": 0.02354,
          "secs": 7.3
        },
        {
          "probe": "P1",
          "arm": "off",
          "correct": true,
          "old_value_as_current": false,
          "answer": "Our production deploy target is now Render, in the Oregon region. This changed from Hetzner Cloud, Falkenstein, in the update of 2026-09-29.",
          "client_tool_calls": 0,
          "gateway_search_calls": 1,
          "memory_records": 0,
          "usd": 0.0062,
          "secs": 8
        },
        {
          "probe": "P2",
          "arm": "on",
          "correct": true,
          "old_value_as_current": false,
          "answer": "Before the move, the production deploy target was Hetzner Cloud, Falkenstein region. Now it is Render, Oregon region (source: memory records fe-6ab7e50d439f46 and fe-9841ca05f98f44).\n\n⏺ context-mode · memory +1 record about user, production +2 more",
          "client_tool_calls": 0,
          "gateway_search_calls": 0,
          "memory_records": 1,
          "usd": 0.02121,
          "secs": 5.1
        },
        {
          "probe": "P2",
          "arm": "off",
          "correct": true,
          "old_value_as_current": false,
          "answer": "Before the move, the production deploy target was Hetzner Cloud in the Falkenstein region. On 2026-09-29, you moved it to Render in the Oregon region.",
          "client_tool_calls": 0,
          "gateway_search_calls": 1,
          "memory_records": 0,
          "usd": 0.00684,
          "secs": 5.8
        },
        {
          "probe": "P3",
          "arm": "on",
          "correct": true,
          "old_value_as_current": false,
          "answer": "The on-call engineer is Baran Kaya. A later update in memory replaced Selin Aydin with Baran Kaya (edge_id fe-6a4f6d3c40c148).\n\n⏺ context-mode · memory +1 record about user, on-call engineer, project +1 more",
          "client_tool_calls": 0,
          "gateway_search_calls": 0,
          "memory_records": 1,
          "usd": 0.02146,
          "secs": 4.7
        },
        {
          "probe": "P3",
          "arm": "off",
          "correct": true,
          "old_value_as_current": false,
          "answer": "Baran Kaya is the on-call engineer now. This record is from 2026-09-29, and it replaced Selin Aydin.",
          "client_tool_calls": 0,
          "gateway_search_calls": 1,
          "memory_records": 0,
          "usd": 0.00585,
          "secs": 8.2
        },
        {
          "probe": "P4",
          "arm": "on",
          "correct": true,
          "old_value_as_current": false,
          "answer": "The on-call engineer before the current one was Selin Aydin. Baran Kaya replaced Selin (source: context-mode memory, edge_id fe-6ab7e50d439f46 and fe-9841ca05f98f44).\n\n⏺ context-mode · memory +1 record about user, project, production, organization",
          "client_tool_calls": 0,
          "gateway_search_calls": 0,
          "memory_records": 1,
          "usd": 0.02345,
          "secs": 6
        },
        {
          "probe": "P4",
          "arm": "off",
          "correct": true,
          "old_value_as_current": false,
          "answer": "Selin Aydin was the on-call engineer before the current one. Baran Kaya replaced them on 2026-09-29.",
          "client_tool_calls": 0,
          "gateway_search_calls": 1,
          "memory_records": 0,
          "usd": 0.00651,
          "secs": 7
        },
        {
          "probe": "P5",
          "arm": "on",
          "correct": true,
          "old_value_as_current": false,
          "answer": "CI now runs on Buildkite. Before this, it ran on CircleCI. (Source: context-mode memory, edge_id fe-9841ca05f98f44, your newer setup update.)\n\n⏺ context-mode · memory +1 record about user",
          "client_tool_calls": 0,
          "gateway_search_calls": 0,
          "memory_records": 1,
          "usd": 0.02059,
          "secs": 5.1
        },
        {
          "probe": "P5",
          "arm": "off",
          "correct": false,
          "old_value_as_current": false,
          "answer": "I do not know. I searched the knowledge base, and it has no record of your CI platform.",
          "client_tool_calls": 0,
          "gateway_search_calls": 1,
          "memory_records": 0,
          "usd": 0.00386,
          "secs": 5.2
        },
        {
          "probe": "P6",
          "arm": "on",
          "correct": true,
          "old_value_as_current": false,
          "answer": "Before Buildkite, our CI ran on CircleCI (memory record fe-6ab7e50d439f46).\n\n⏺ context-mode · memory +1 record about user, selin aydin",
          "client_tool_calls": 0,
          "gateway_search_calls": 0,
          "memory_records": 1,
          "usd": 0.02122,
          "secs": 9.4
        },
        {
          "probe": "P6",
          "arm": "off",
          "correct": true,
          "old_value_as_current": false,
          "answer": "Before your current CI system (Buildkite), your CI ran on CircleCI. My source is your setup update of 2026-09-29.",
          "client_tool_calls": 0,
          "gateway_search_calls": 2,
          "memory_records": 0,
          "usd": 0.02111,
          "secs": 13.8
        },
        {
          "probe": "P7",
          "arm": "on",
          "correct": true,
          "old_value_as_current": false,
          "answer": "No. Production moved to Render, Oregon region. You told me this in an earlier update (context-mode memory, edge_id=fe-9841ca05f98f44).\n\n⏺ context-mode · memory +1 record about user, production deploy +2 more",
          "client_tool_calls": 0,
          "gateway_search_calls": 0,
          "memory_records": 1,
          "usd": 0.02298,
          "secs": 6.1
        },
        {
          "probe": "P7",
          "arm": "off",
          "correct": true,
          "old_value_as_current": false,
          "answer": "No. On 2026-09-29, you moved the production deploy target from Hetzner Cloud (Falkenstein) to Render (Oregon region).",
          "client_tool_calls": 0,
          "gateway_search_calls": 1,
          "memory_records": 0,
          "usd": 0.00656,
          "secs": 5.8
        },
        {
          "probe": "P8",
          "arm": "on",
          "correct": true,
          "old_value_as_current": false,
          "answer": "Production deploys to Render, Oregon region. The on-call engineer is Baran Kaya, and CI runs on Buildkite. (Source: context-mode memory, your setup update, edge_id fe-9841ca05f98f44. This update replaces the earlier setup of Hetzner Falkenstein, Selin Aydin and CircleCI.)\n\n⏺ context-mode · memory +1 record about user, production, project +1 more",
          "client_tool_calls": 0,
          "gateway_search_calls": 0,
          "memory_records": 1,
          "usd": 0.02424,
          "secs": 5.9
        },
        {
          "probe": "P8",
          "arm": "off",
          "correct": true,
          "old_value_as_current": false,
          "answer": "Past answers in my notes say the production deploy target is Render (Oregon region), the on-call engineer is Baran Kaya, and the CI system is Buildkite, all from a setup update on 2026-09-29. These notes are my earlier answers, not the original records, so please check them against your team's source before you use them.",
          "client_tool_calls": 0,
          "gateway_search_calls": 2,
          "memory_records": 0,
          "usd": 0.02369,
          "secs": 7.8
        }
      ],
      "store_after_session_2": [
        {
          "edge_id": "fe-71705121eb1a43",
          "subject": "ci",
          "predicate": "ci_provider",
          "object": "Buildkite",
          "kind": "fact",
          "tier": "inferred",
          "source": "ai-extract",
          "valid_at": "2026-09-29T21:06:43.153Z",
          "invalid_at": "",
          "created_at": "2026-09-29T21:06:43.153Z"
        },
        {
          "edge_id": "fe-38ae661f45544d",
          "subject": "production",
          "predicate": "on_call_engineer",
          "object": "Baran Kaya",
          "kind": "fact",
          "tier": "inferred",
          "source": "ai-extract",
          "valid_at": "2026-09-29T21:06:43.153Z",
          "invalid_at": "",
          "created_at": "2026-09-29T21:06:43.153Z"
        },
        {
          "edge_id": "fe-5f757c15bf4546",
          "subject": "production",
          "predicate": "production_deploy_target",
          "object": "Render, Oregon region",
          "kind": "fact",
          "tier": "inferred",
          "source": "ai-extract",
          "valid_at": "2026-09-29T21:06:43.153Z",
          "invalid_at": "",
          "created_at": "2026-09-29T21:06:43.153Z"
        },
        {
          "edge_id": "fe-ef2882d3355f41",
          "subject": "project",
          "predicate": "ci_platform",
          "object": "Buildkite",
          "kind": "fact",
          "tier": "asserted",
          "source": "mcp-remember",
          "valid_at": "2026-09-29T21:06:41.010Z",
          "invalid_at": "",
          "created_at": "2026-09-29T21:06:41.010Z"
        },
        {
          "edge_id": "fe-6a4f6d3c40c148",
          "subject": "project",
          "predicate": "on_call_engineer",
          "object": "Baran Kaya",
          "kind": "fact",
          "tier": "asserted",
          "source": "mcp-remember",
          "valid_at": "2026-09-29T21:06:41.010Z",
          "invalid_at": "",
          "created_at": "2026-09-29T21:06:41.010Z"
        },
        {
          "edge_id": "fe-df57c2f179a34b",
          "subject": "project",
          "predicate": "production_deploy_target",
          "object": "Render, Oregon region",
          "kind": "fact",
          "tier": "asserted",
          "source": "mcp-remember",
          "valid_at": "2026-09-29T21:06:41.010Z",
          "invalid_at": "",
          "created_at": "2026-09-29T21:06:41.010Z"
        },
        {
          "edge_id": "fe-bcbe00d81bb649",
          "subject": "ci",
          "predicate": "runs_on",
          "object": "CircleCI",
          "kind": "fact",
          "tier": "inferred",
          "source": "ai-extract",
          "valid_at": "2026-09-29T21:06:11.243Z",
          "invalid_at": "",
          "created_at": "2026-09-29T21:06:11.243Z"
        },
        {
          "edge_id": "fe-440784123c2c46",
          "subject": "on-call engineer",
          "predicate": "name",
          "object": "Selin Aydin",
          "kind": "fact",
          "tier": "inferred",
          "source": "ai-extract",
          "valid_at": "2026-09-29T21:06:11.243Z",
          "invalid_at": "",
          "created_at": "2026-09-29T21:06:11.243Z"
        },
        {
          "edge_id": "fe-611a990e1a1d45",
          "subject": "production deploy",
          "predicate": "target",
          "object": "Hetzner Cloud, Falkenstein region",
          "kind": "fact",
          "tier": "inferred",
          "source": "ai-extract",
          "valid_at": "2026-09-29T21:06:11.243Z",
          "invalid_at": "",
          "created_at": "2026-09-29T21:06:11.243Z"
        },
        {
          "edge_id": "fe-e5f1a38e93e644",
          "subject": "project",
          "predicate": "ci_provider",
          "object": "CircleCI",
          "kind": "fact",
          "tier": "asserted",
          "source": "mcp-remember",
          "valid_at": "2026-09-29T21:06:09.028Z",
          "invalid_at": "",
          "created_at": "2026-09-29T21:06:09.028Z"
        },
        {
          "edge_id": "fe-06b680186cb344",
          "subject": "project",
          "predicate": "on_call_engineer",
          "object": "Selin Aydin",
          "kind": "fact",
          "tier": "asserted",
          "source": "mcp-remember",
          "valid_at": "2026-09-29T21:06:09.028Z",
          "invalid_at": "2026-09-29T21:06:41.010Z",
          "created_at": "2026-09-29T21:06:09.028Z"
        },
        {
          "edge_id": "fe-5d90f71bde9948",
          "subject": "project",
          "predicate": "production_deploy_target",
          "object": "Hetzner Cloud, Falkenstein region",
          "kind": "fact",
          "tier": "asserted",
          "source": "mcp-remember",
          "valid_at": "2026-09-29T21:06:09.028Z",
          "invalid_at": "2026-09-29T21:06:41.010Z",
          "created_at": "2026-09-29T21:06:09.028Z"
        }
      ],
      "summary": {
        "on": {
          "correct": 8,
          "probes": 8,
          "old_value_as_current": 0,
          "gateway_search_calls": 0,
          "usd": 0.17869
        },
        "off": {
          "correct": 7,
          "probes": 8,
          "old_value_as_current": 0,
          "gateway_search_calls": 10,
          "usd": 0.08062
        }
      }
    },
    {
      "id": "A",
      "role": "discarded design: no client tools",
      "arms": [
        "on",
        "off"
      ],
      "results": [],
      "store_after_session_2": [],
      "summary": {
        "on": {
          "correct": 0,
          "probes": 8,
          "old_value_as_current": 0,
          "gateway_search_calls": 0,
          "usd": 0
        },
        "off": {
          "correct": 0,
          "probes": 8,
          "old_value_as_current": 0,
          "gateway_search_calls": 0,
          "usd": 0
        }
      }
    }
  ],
  "limits": [
    "One scenario, three facts, one run of record (run C). A small check, not a benchmark.",
    "Run B used the same design on new accounts. The extractor wrote session 1 and session 2 under different slot names, so only 2 of 6 old rows were closed. No answer gave an old value as current.",
    "Run A used --tools \"\". The gateway treats a Claude Code request with no tools as a side call and adds no memory, so both arms scored 0 of 8. Runs B and C declare one client tool the model never called.",
    "The live memory block bytes were not captured. A dry run returned an empty block for the same questions.",
    "In run C the memory-on probe turns cost more than the memory-off turns, as the client reported. The cause is not found.",
    "Client transcripts are not published: they carry the operator's account e-mail."
  ]
}