# Post-mortem harness: forensics + live A/B for the 1,393-agent run fixes

The scripts that produced every number in tracking issue #103563 and the "Independent review
(round 2)" sections of its 13 PRs. Two halves:

- **`forensics/`** reads a *copy* of a Hermes `state.db` (plus rotated `agent.log*` and git) and
  recomputes the *observed* figures for any run: where the money went, per-call cache behaviour,
  nested-delegate timeouts, batch-join delivery delay, tool friction, `/goal` loop behaviour, and the
  post-open rework inventory. It needs no model calls and no network.
- **`live_ab/`** and **`review_probes/`** exercise the real code paths of a checkout (real
  `AIAgent`, dispatch, judge, scanner, SDK) against local fake providers to show what each fix
  *does*. `run.py` runs them against one or two checkouts and prints PASS/FAIL side by side.
  `review_probes/` are the probes the independent `/review` wrote; each reproduced a defect in the
  first version of a PR and the fixed head must pass it.

Everything is labeled **OBSERVED** (from usage rows / logs / git) or **MODELED** (a reconstruction or
replay). Do not add the modeled figures to the observed ones; see §5 of #103563 for why they overlap.

## Requirements

- A Hermes checkout with its venv (`.venv/bin/python`). NeMo Relay is not required for the forensics;
  it is what the run itself used for the wire captures in `live_ab/cache_prefix_wire.py`.
- For forensics: a **copy** of `~/.hermes/state.db` (never point at the live file; `sqlite3 state.db ".backup copy.db"`
  or `cp` while Hermes is idle) and, optionally, the rotated `~/.hermes/logs/agent.log*`.
- For `--live` probes: real credentials in `HERMES_HOME` (they spend cents per run).

## Forensics: recompute the observed numbers for YOUR run

```bash
cd <hermes-checkout>
P=.venv/bin/python
# 1. cost buckets, depth/duration shares, context-size reconstruction, excess-cache-write proxy, cap replay
$P -m evals.postmortem.forensics.tokens     --db state_copy.db [--root <session_id>] [--cap 200000 --floor 65000]
# 2. per-call cache behaviour from the logs (coverage fraction is printed first; quote nothing without it)
$P -m evals.postmortem.forensics.logcalls   --db state_copy.db --logs "$HOME/.hermes/logs/agent.log*"
# 3. delegation: timeouts, orphaned children, polling, batch-join delay, truncated summaries
$P -m evals.postmortem.forensics.delegation --db state_copy.db
# 4. tool friction: hardline false blocks, foreground refusals, whole-file rewrites, output volume
$P -m evals.postmortem.forensics.tools      --db state_copy.db
# 5. /goal loop: nudges, parked barrier, notification counts
$P -m evals.postmortem.forensics.goal_loop  --db state_copy.db
# 6. rework inventory for a large PR (git only)
$P -m evals.postmortem.forensics.rework     --repo . --base <merge-base> --open <sha-at-open> --head <merged-sha>
```

Each writes `postmortem_out/<lane>.json` and prints a summary. `--root` defaults to the top-level
session with the most descendants; compression-rollover children are excluded from the population so
cost buckets are disjoint. Pricing is fitted from `estimated_cost_usd`, so dollars match what that
Hermes recorded (an estimator, not an invoice).

### Reference output (the #102117 run, `state_copy.db` of 2026-09-04)

| lane | prints |
|---|---|
| tokens | `1394 sessions, $19,302.59`; buckets cache_write $11,159.76 · cache_read $3,587.17 · output $4,555.48; depth-2 65%; >60 min 61%; MODELED excess cache-write proxy ~$9.1k; sawtooth cap 200K: prompt volume ×0.76 (reconstructed sizes) |
| logcalls | coverage 22,489/93,284 (24.1%); median prompt 229,648, p90 408,794, >200K 59%; hit ratio 93.9%; strict plateau 22.5% of uncached input, non-advancing 74.8%; sawtooth cap 200K on real sizes ×0.498 |
| delegation | 266 orchestrators; 332 delegate_task timeouts in 234 sessions (all nested); 93 results carried; 242.6 h sleep after first timeout; those sessions' lifetime $4,034.69 (includes real work); summaries truncated 123/220; batch-join withheld child-hours root 233 / depth-1 300 / depth-2 52 |
| tools | 96,855 tool calls; hardline blocks 579 (566 "malformed" class); foreground timeout refusals 475 (303 asked 900 s); `&` 185, nohup 24; write_file 8,188 calls / 92.8M chars, 661 rewrites of a file read this session >20k; patch 4,623 |
| goal_loop | 5 nudges (2 within 180 s of a "waiting" turn); 34 batch notices; 48 bg-process notices; final barrier `waiting_on_session=proc_…` parked 201 min |
| rework | surface at open: 1,703 names / 341 modules, 951 methods / 156, 126 test defs / 52 files; 125 post-open commits (simplify 55, review-fix 39, fix 19, …) |

The two sawtooth figures differ on purpose: `tokens` replays *reconstructed* per-call sizes for all
93k calls (×0.76); `logcalls` replays *real* per-call sizes for the 24% of calls in the logs (×0.50,
the peak-concurrency window). The tracking issue quotes the second and says so.

## Live A/B: what each fix does

```bash
# offline probes (fake providers, temp HERMES_HOME), main vs a branch or integration checkout:
.venv/bin/python -m evals.postmortem.run --repo /path/to/main --compare /path/to/branch
# add the probes that make real provider calls (cents):
.venv/bin/python -m evals.postmortem.run --repo /path/to/branch --live
# one PR only:
.venv/bin/python -m evals.postmortem.run --repo . --only 103492
```

| probe | PR | expects on the fixed head |
|---|---|---|
| `live_ab/hardline_scanner_matrix.py` | #103492 | 11-case matrix `ALL OK` (546-block class allowed; newline/`;`/`&&`/`|` hidden `reboot` blocked as itself) |
| `review_probes/scanner_bypass_probe.py` | #103492 | public guard `approved: False`, 0 callbacks, harmless Bash witness not executed |
| `live_ab/subagent_context_cap.py` | #103513 | child trigger `200,000` on a 1M model; parent untouched |
| `review_probes/context_cap_probe.py` | #103513 | cap holds through repeated compression + persistence; config validation |
| `live_ab/nested_delegate_deadline.py` | #103486 | 40 s deadline + 75 s leaf: result delivered (main: lost) |
| `review_probes/deadline_probe.py` | #103486 | same through actual dispatch |
| `live_ab/auth_stampede.py <repo> 40` | #103526 | `401s=0` (main: 40) |
| `review_probes/credential_identity_probe.py <repo> pr` | #103526 | explicit account-A key stays A (v1: became B) |
| `live_ab/batch_failure_notice.py` | #103549 | `TASK_FAILURE_NOTICE` at t+0.3 s, `BATCH_FINAL` after |
| `review_probes/notice_delivery_probe.py` | #103549 | gateway receives notice, notice, final; busy-parent final claim succeeds |
| `review_probes/cache_estimator_probe.py` | #103476 | preflight ≈ wire estimate; `should_compress` agrees |
| `live_ab/cache_prefix_wire.py <repo> B` (live) | #103476 | 0 mutated prefixes across 6 calls |
| `live_ab/goal_judge_wait.py <repo> 3` (live) | #103534 | `wait` ×3 on the run's "waiting on workers" response (main: `continue` ×3) |
| `review_probes/goal_scope_probe.py` | #103496/#103534 | judge sees own processes; delegation WAIT lifts on batch return |
| `review_probes/goal_repaste_probe.py` | #103553 | near-whole re-paste → pointer; `ship the API` ≠ `ship the UI` |
| `review_probes/rewrite_hint_probe.py` | #103551 | remote backend: no host-derived hint; FIFO: returns; 460 KB repeated-line file: skipped, not 22 s |
| `review_probes/finalizer_schedule_probe.py` | #103507 | pytest plugin: `-p evals.postmortem.review_probes.finalizer_schedule_probe --finalizer-probe=consumer-first` on `tests/e2e/test_relay_native_openai_stream.py` → 2 passed |

## Cache concurrency probe (`live_ab/cache_concurrency_probe.py`)

The instrument behind #104284 / #104421 and NousResearch/api#227. N concurrent real `AIAgent`
sessions run the same growing tool loop against a route; every call's cache_read / cache_creation,
response id, upstream provider and prefix shas are logged, and consecutive pairs are classified
`ideal` / `stuck` (previous write not visible: routing) / `collapse` (whole context re-written).

```bash
# ~$50 per 20x6 arm on Fable 5.1 at the 5m tier
python -m evals.postmortem.live_ab.cache_concurrency_probe --repo . --provider nous --workers 20 --calls 6 --out /tmp/p.jsonl
python -m evals.postmortem.live_ab.cache_concurrency_probe --repo . --provider nous --wire native --workers 20 --calls 6 --out /tmp/p.jsonl
python -m evals.postmortem.live_ab.cache_concurrency_probe --repo . --provider openrouter --pin anthropic --workers 20 --calls 6 --out /tmp/p.jsonl
```

Reference results (2026-09-05/06): Nous native wire 13.9% stuck (14–20% over 4 runs, unchanged by the
portal's provider pin or a 2 s settle); Nous chat wire 0/320; OpenRouter pinned 0/161; OpenRouter
unpinned 9.8% collapse. Use this to clear a new upstream before flipping
`agent/nous_wire.py::GMI_NATIVE_WIRE_CLEARED`. `--ttl 1h` reproduces the 2× write price
(#104168). The `.summary.json` carries the bad pairs with both response ids for the provider's logs.

## What is NOT here, and why

- **The trajectories themselves.** The run's `state.db` contains 51,956 absolute home paths, 5,341
  e-mail addresses, private IPs, chat/user ids, and real-shaped API keys and JWTs in tool output. It
  is not publishable, and this harness is written so it does not need to be: run it on your own DB.
- **Hand classification.** The root-cause classes of the 72 rework commits (dropped symbol vs semantic
  drift vs compat fallout) were labeled by hand in the original audit; `forensics/rework.py` reproduces
  the mechanical inventory that labeling started from and stops there.
- **A single aggregate saving.** By design. Each lane prints its own number with its own caveat.

## Evidence bundle

The original lane reports, the independent review, and the JSON this harness recomputes on the run's DB
are in a secret gist linked from #103563 (no trajectories; see "What is NOT here").

## Provenance

Forensic lanes: five parallel Hermes subagents (2026-09-04), rewritten here to take `--db`/`--root`
instead of hard-coded paths. `live_ab/`: the primary agent's per-PR A/Bs. `review_probes/`: the
independent `/review` subagent's probes (2026-09-05), adapted to take paths from the command line;
their findings and the fixes are in each PR's "Independent review (round 2)" section.