The wrong witness - dropzone-sweep red for a week while running every hourHands-on

dropzone-sweep has been red on your day page every morning since 9-28. It has run 77 times with exit 0. The health check watches a log file the job stopped writing to. Fix the witness in ten minutes and watch the alarm clear.

Automation Hygiene Rule 1: verify the side-effect, not the trigger. The health check does that by watching an artifact — a file the job must write every run — and raising a flag when it goes stale. But the check is only as good as its witness. If the job stops writing that particular file, the check cannot tell 'the job died' from 'the job moved'.

How a witness goes stale: a refactor changes where output goes. launchd's StandardOutPath is only touched when the program prints something; the audit fix made the router quiet on stdout and gave it its own log. Nothing failed — the evidence moved. The same bug hit jobhunt-nightly on 8-15 (it watched launchd.out, the wrong file) and was fixed the same way.

Three independent ways to prove a scheduled job ran: launchd's own count and last exit code (launchctl print), the job's log modification time, and the artifact it exists to produce. When they disagree, the odd one out is the lie. Here two say 'running hourly' and one says 'dead since 9-28'.

Worked example

Prove it, fix it, and confirm the alarm clears.

launchctl print gui/501/app.zorc.dropzone-sweep | grep -E "runs|last exit"
ls -l /Users/tom/Claude/JARBUS/logs/dropzone-sweep.out /Users/tom/Claude/JARBUS/logs/dropzone-router.log
grep -n "dropzone-sweep" /Users/tom/Claude/PDB/scripts/job_health_check.py

# the fix: point the witness at the file the router writes now
sed -i '' 's#JARBUS/logs/dropzone-sweep.out#JARBUS/logs/dropzone-router.log#' \
  /Users/tom/Claude/PDB/scripts/job_health_check.py

python3 /Users/tom/Claude/PDB/scripts/job_health_check.py
tail -3 /Users/tom/Claude/PDB/logs/job-health.log

cd /Users/tom/Claude/PDB && git add scripts/job_health_check.py && \
  git commit -m "fix(job-health): dropzone-sweep watches dropzone-router.log (its log since 021aa2d)"

• runs = 77, last exit code = 0: launchd says the sweep fires hourly and succeeds.
• The ls shows the old witness frozen at Sep 28 18:44 and the router log modified today. Both are true.
• The grep shows line 349: the artifact path, a 26-hour tolerance (a slept-through night is fine) and the kickstart recovery string.
• The sed swaps one path; the '' after -i is macOS sed's way of saying 'no backup file'. Nothing else in the file changes.
• The health check then runs the way the 11:00 job does; the log tail should show no dropzone-sweep FAIL line, and tomorrow's day page drops it from Fix-it.
• The commit keeps the nightly git-sweep from folding your fix into an anonymous auto-commit.

Gotchas

• Fix the witness, not the threshold. Widening the 26-hour tolerance would have hidden this and every real outage after it.
• If dropzone-router.log ever stops moving, the alarm is real: the router handles every drop, so a dead router means drops sit unrouted.
• last exit code = 0 matters as much as runs. A job that runs and fails every hour also shows a high run count.
• The router's lock file (dropzone-router.lock) is touched each run too; it is a fine second witness if you ever want one.