Day 52 of 55 · Wednesday · Learning

A timeout you don't catch is just a slower crashHands-on

subprocess timeouts don't return — they RAISE. Set one without a handler and you've traded a hang for a crash, which is what took the pull pipeline down 78 times.

Catch-up progress

52/55

Why this matters to you

rm_pull.py already had a 90-second timeout on its bun-bridge call — responsible engineering, half done. When the cloud was slow, subprocess.run raised TimeoutExpired, nothing caught it, and the device-inventory survey crashlooped 78 times between 8/20 and 8/23 before the 8/23 session added the handler.

In Python, `subprocess.run(cmd, timeout=90)` does not come back with a 'timed out' result. At second 90 it kills the child and raises subprocess.TimeoutExpired. If no try/except is waiting, the exception unwinds your whole program. The timeout parameter is only half a feature; the except block is the other half.

Think of it as the code-level twin of your Automation-hygiene watchdog rule. The rule says every long job needs a watchdog you've PROVED kills. The code corollary: every kill needs a landing pad you've proved catches. An unproved watchdog is decorative; an uncaught one is destructive.

What the good landing pad does is a design decision per call site: return a safe empty value (rm_pull's list_docs now returns an empty inventory), retry once, or surface a 🚨 Things task. What it never does is let one slow network call convert into a crashloop that poisons every later run.

Worked example

Reproduce the raise in miniature, then read the real fix.

python3 - <<'PY'
import subprocess
try:
    subprocess.run(['sleep', '10'], timeout=1)
except subprocess.TimeoutExpired:
    print('caught: child killed at 1s, program continues')
PY
grep -n -A4 "TimeoutExpired" /Users/tom/Claude/PDB/scripts/rm_pull.py | head -20

The inline script asks for a 10-second sleep with a 1-second budget — the except line is the difference between this printout and a traceback.
The grep shows the 8/23 fix in place: where list_docs() catches the raise and what it returns instead of crashing.

▶ Do it now

Run the inline Python above once as-is, then delete the try/except and run it again to watch the uncaught version die — feel the difference you're designing away.
Read the TimeoutExpired handler in rm_pull.py and name its strategy: safe-empty, retry, or alert?
Grep your other pipeline scripts for `timeout=` (`grep -rn 'timeout=' /Users/tom/Claude/PDB/scripts/*.py | grep -v except | head`) and check each hit has a handler somewhere above it.

Gotchas

TimeoutExpired's .stdout/.stderr may hold partial output — log it before discarding; it's often the only clue why the child was slow.
A timeout kills the child process, but if that child spawned its own children (bun spawning network fetches), they can outlive the kill — orphans that hold locks.
Retrying on timeout without backoff against a rate-limited API (the reMarkable cloud) just spends your quota faster — you saw those 429s on 8/18.

Go deeper: subprocess.TimeoutExpired docs

One-card takeaway

Every timeout= you write creates an exception path — build the landing pad in the same commit.

← Prev Index Next →