Day 62 of 62 · Monday · Learning

Trust the side-effect, never the triggerReview

Every incident in this curriculum is one rule wearing different clothes: a job that RAN is not a job that WORKED. Verify the side-effect, never the trigger — and as a weekly operator, make the fleet witness itself.

Catch-up progress
62/62
Why this matters to you

You're no longer watching these jobs daily. The failures that hurt are the silent ones — a runner that exits 0 having done nothing, a green check that never consulted the ledger already screaming. This week's lessons were all the same rule from different angles; today closes the loop by naming it.

Automation-hygiene rule 1: verify the side-effect, not the trigger. `last_fired_at` and `exit 0` only prove the runner STARTED — they say nothing about whether the work landed. The only proof is querying the downstream system: did the file appear, did the cloud doc show up in the listing, did the Things task get created, did the DB row change. Two RemoteTrigger failures in May both trusted `last_fired_at` and were both silently dead; every lesson since has been a variation on that mistake.

See the rule repeated across this week. The reMarkable poller confirms against the CLOUD listing, never against 'the bridge returned' — because a 429-throttled fetch can return without pulling anything (Monday). The JobHunt wrapper ends with `exit $VRC`, the verify leg's code, not the LLM leg's — because a briefing that didn't reach the device is a failed morning no matter how well the model ran (Wednesday). The event feed records what actually happened so the dashboard shows history, not hope (Saturday). And Mission Control reconciles the health check against the alert ledger so a false-green can't hide (the 8-20 lesson). One principle, four surfaces.

The operator's move now is to make the fleet witness itself. Every unattended job needs a SEPARATE check that asks 'did this job's artifact appear in the last N hours?' and fires a 🚨 Things task when it didn't — the dead-man's-switch pattern from 7-29. The job reporting its own success is the thing you can't trust; a second, independent witness checking the side-effect is what turns a silent death into a notification you'll actually read.

Worked example

Audit one job against the rule: what proves it worked, and does anything check that?

# Pick a live job and see what it claims:
launchctl list | grep -i zorc

# Then ask: does the health check look for the ARTIFACT, or just trust the run?
grep -nE 'artifact|fresh|exists|alert-state' /Users/tom/Claude/PDB/scripts/job_health_check.py | head -20
▶ Do it now
  1. Pick one live app.zorc.* job from the launchctl list and name the EXACT downstream artifact that proves it worked — a specific file, a cloud doc, a Things task, a row.
  2. Grep job_health_check.py and confirm whether anything actually checks for THAT artifact within a fresh time window — or whether it just trusts the job ran.
  3. If nothing witnesses it, write the one-line witness rule ('fail if <artifact> is missing or older than N hours, and fire a 🚨') — that sentence is the fix, whether or not you wire it today.

Gotchas

Go deeper: Google SRE: Monitoring Distributed Systems
One-card takeaway

A job that ran is not a job that worked — prove the side-effect landed, and give every unattended job a separate witness that shouts when the artifact doesn't appear.