Day 34 of 37 · Wednesday · Learning

Dead man's switch — catching the job that vanishedConcept

Your failure alerts only fire when a job runs and crashes. When a job never really starts — or dies before it can report — the cadence just vanishes. The fix is a dead man's switch: a separate checker that asks 'did the artifact show up?' instead of trusting the job to announce its own death.

Catch-up progress
34/37
Why this matters to you

This isn't hypothetical: the weekly review died silently on Jul 5, 12, AND 19 — three ConnectionRefused failures, zero 🚨 tasks, zero notifications. The ledger at planning-history/ shows the receipts: weekly-review-2026-06-28.md, then nothing until 2026-07-26. A month of Sundays where the system's self-improvement loop was dead and nobody — human or machine — noticed.

All your alerting so far is *self-reporting*: run_plan_day.sh catches its own errors and files a 🚨 Things task; the alert ledger (Jul 21) dedups those reports. But self-reporting has a blind spot — the reporter and the failure are the same process. If claude -p can't even reach its API (ConnectionRefused), the code that would have filed the alert dies with the job. The job doesn't fail loudly; it simply leaves no trace.

A dead man's switch inverts the question. Instead of 'did anything report a failure?' it asks 'did the expected *evidence of success* appear on time?' Railway engineers built this into locomotives a century ago: the driver must actively hold a lever; if the hand disappears, the brakes fire. No signal IS the signal. In software the pattern is called a heartbeat or witness check: a second, dumber process that knows only two things — where job X leaves its artifact, and how stale that artifact is allowed to get.

The witness must be *simpler and more reliable* than the thing it watches. Your weekly review depends on Claude, OAuth tokens, and the network — three ways to die. A witness needs only `find -newermt` and the Things URL scheme, which is exactly the deterministic-local-script layer CLAUDE.md already prescribes for verify-and-send. You even have a natural home for it: app.zorc.job-health already runs on a schedule — teaching it to check artifact freshness turns it into the witness for every cadence at once.

Worked example

The whole pattern is one shell test. Run it right now against the real weekly-review ledger:

# Did the weekly review leave its companion record in the last 8 days?
find /Users/tom/Claude/PDB/planning-history \
  -name 'weekly-review-*.md' -newermt '8 days ago' | grep -q . \
  && echo "OK — review ran this week" \
  || echo "MISSING — no weekly-review artifact in 8 days"

# Backdate the check to Jul 20 and the silent month lights up:
find /Users/tom/Claude/PDB/planning-history \
  -name 'weekly-review-*.md' -newermt '2026-07-12' \! -newermt '2026-07-20' | grep -q . \
  || echo "MISSING — this is what Jul 20 would have caught"
▶ Do it now
  1. Run the first command above in Terminal. Today it should print OK — the Jul 26 record is 3 days old.
  2. Now audit your cadences: ls ~/Library/LaunchAgents | grep app.zorc gives you ~20 jobs. For each of the big three — weekly-review, jobhunt-sweep, inventory-digest — name the artifact that proves it ran (companion record, sweep log, digest file).
  3. Pick the one cadence with NO vanish-detection today and write its one-line find test into a note (or hand it to Claude to wire into job-health). You've just designed your first witness — observe that the check passes today, and imagine the 🚨 it would have filed on Jul 12.

Gotchas

Go deeper: healthchecks.io — dead man's switch as a service (the hosted version of this pattern) · Google SRE book — Monitoring Distributed Systems (symptoms vs. causes)
One-card takeaway

A job can't report its own death — pair every cadence with a dumber witness that treats a missing artifact as the alarm.