Watchdogs & timeouts — why a 30-min cap killed a 34-min jobConcept

A watchdog is a wall-clock kill-switch wrapped around a job so a hang can't run forever. But a cap set too tight kills healthy work — on May 30 a 30-min cap killed a plan-day run that legitimately needed 34. Today you'll learn how to set a cap that catches hangs without strangling good runs.

A watchdog is just a timer with a weapon. You wrap your real job in a command that says: 'run this, but if it isn't done in N minutes, kill it.' On macOS the tool is `gtimeout` (from coreutils) or a plain `timeout`. Without one, a job that hangs — waiting on a network call that never returns, a TCC permission prompt nobody clicks — runs forever, holding its slot and never reporting failure. That's the worst outcome: not a crash, but a silent zombie.

The trap is the cap value. Too loose (2 hours) and a hung job wastes two hours before anyone notices. Too tight (30 min) and a job having a slightly slow but *healthy* day gets executed mid-sentence — which is exactly what happened on May 30: plan-day took 34 minutes against a 30-minute watchdog and got killed doing real work. The fix isn't 'remove the watchdog,' it's 'size the cap to the job's real p99 runtime, then add headroom.'

There's a second knob most people miss: `--kill-after`. A normal timeout sends a polite SIGTERM ('please wrap up'). A well-behaved job catches that and exits cleanly. But a truly hung job ignores SIGTERM. `--kill-after=30s` says: 'if it's still alive 30 seconds after I asked nicely, send SIGKILL and end it for real.' Without `--kill-after`, your watchdog can itself hang on a job that refuses to die.

Worked example

This is the actual pattern your plan-day wrapper uses. Run the first block to see your real wrapper; the second is the shape to copy.

# 1. Look at your own watchdog
grep -n -A6 'timeout' ~/Library/LaunchAgents/com.tom.pdb-plan-day.plist 2>/dev/null
cat ~/Claude/PDB/scripts/run_plan_day.sh | grep -n -A4 -i timeout

# 2. The shape of a correct watchdog
gtimeout --kill-after=30s 45m  /usr/bin/python3 my_job.py
#        ^^^^^^^^^^^^^^^^ ^^^                ^^^^^^^^^^^^^^^
#        SIGKILL escape   soft cap           the real work

• Block 1 shows your live configuration — what cap plan-day actually runs under today, so you can see whether May 30's fix (raising it past 34 min) is still in place.
• `gtimeout` is the GNU timeout from coreutils (`brew install coreutils`); plain `timeout` may not exist on stock macOS.
• `45m` is the soft cap: at 45 minutes it sends SIGTERM. Size this to your job's worst *healthy* run plus ~30% headroom — for plan-day that means comfortably above 34 min.
• `--kill-after=30s` is the hard backstop: 30 seconds after the polite SIGTERM, send the un-ignorable SIGKILL. This is the line that prevents the watchdog itself from hanging.

Gotchas

• A watchdog that only sends SIGTERM can hang forever on a job that ignores it — always pair it with `--kill-after`.
• Raising the cap to 'never kill' defeats the purpose. The goal is a cap that's loose enough for healthy runs but tight enough that a true hang surfaces fast.
• `last_fired_at` on a LaunchAgent proves the trigger started, not that the job finished. A timeout-killed job can still look 'fired.' Verify the side-effect (was today.html written?), not the trigger — this is Automation Hygiene rule 1.