Day 41 of 43 · Thursday · Learning

Reading the alert ledger as ops historyReview

alert-state.json is your pager: open entries are the infra to-do list, the count field is how loud each is shouting, and a missing entry means auto-resolve did its job.

Catch-up progress
41/43
Why this matters to you

Right now the ledger holds exactly two open incidents — jobhunt-sweep (count 2) and weekly-review-fail (count 1) — and that terse JSON is the most honest status report your system produces. Aug 6's plan-day failure isn't in it because Aug 7 succeeded and auto-resolve closed it. Learning to read this file is learning to take handoff from your own automation.

You shipped the dedup + auto-resolve layer on Jul 21 (PDB d9b37fc) to stop failure spam: first failure opens an incident and fires ONE 🚨 Things task; repeat failures bump `count` and update `last_seen` without new tasks; a success for that job key deletes the entry and closes the loop. The ledger is the layer's memory.

That gives every entry a grammar. `opened_at` = when it broke. `last_seen` = the most recent re-failure — if it's old, the job may have stopped running entirely (a vanish, which the ledger can't see — that's the dead-man's-switch gap from the Jul 29 lesson). `count` = how many times it's re-fired; a rising count with no fix is a standing appointment you keep missing.

Reading it as ops history: jobhunt-sweep opened Jul 27, last_seen Aug 3, count 2 — a weekly job that has failed both chances since opening, root cause known (permission wall, yesterday's lesson). weekly-review-fail opened Aug 2, count 1 — and if tonight's run succeeds, watch it auto-resolve out of the file. An empty ledger isn't 'no information'; it's the system attesting fleet-healthy.

Worked example

Read the pager and verify it against Things:

python3 -c "
import json, datetime
ledger = json.load(open('/Users/tom/Claude/PDB/data/alert-state.json'))
for key, inc in ledger.items():
    print(f\"{key}: open since {inc['opened_at'][:10]}, last failure {inc['last_seen'][:10]}, fired {inc['count']}x\")
"
▶ Do it now
  1. Run the reader above.
  2. Open Things and search 🚨 — match each open incident to its task. If a task was completed but the incident is still open, the underlying job is still broken; completing the task didn't fix anything.
  3. Tomorrow after 18:00, cat the ledger again: if tonight's weekly review succeeded, weekly-review-fail should be gone. Watching an entry auto-resolve once makes the whole layer trustworthy.

Gotchas

One-card takeaway

One JSON file, read weekly, tells you what's broken, how long, and how loudly — that's a pager, and you built it.