Day 56 of 62 · Monday · Learning

Rate limits and backoff - the 3,116-error storm and the fixConcept

A 429 is the reMarkable cloud saying 'too many requests, slow down' — and for days in August your poller ignored it, logging 3,116 of them. The fix is backoff, cooldowns, and a timeout that outlasts the backoff.

Catch-up progress
56/62
Why this matters to you

The old rm_pull design fetched a FULL document archive per held doc every 20-minute poll. At 24 held docs on a device carrying 112, that hammered straight past the account's rate limit — 3,116 logged 429s that throttled the account for days. The rebuild you shipped 8-26 is a small master class in living inside someone else's rate limit.

A rate limit is a server's rule: 'no more than N requests per window.' Cross it and you get HTTP 429 Too Many Requests — not an error in your code, a request to back off. The wrong response is to retry immediately (that's more load on an already-overloaded door); the right one is exponential backoff — wait 1s, then 2s, then 4s, growing the gap so the server can drain. The reMarkable bridge does this internally, and a single archive fetch can now sit in a ~10.5-minute backoff before it succeeds.

The rebuilt rm_pull.py attacks the problem from three sides. First, cheap probes: instead of fetching an archive per held doc, one light `tags` call covers ALL due docs in a single bridge invocation — archives are paid for only on an actual release. Second, cooldowns: TAGCHECK_COOLDOWN_S = 1200 (20 min) between probes of the same doc, six hours for REFERENCE docs, so even the light traffic is bounded. Third, a self-expiring quiet window — writing an epoch into rm-quiet-until silences the poller with ZERO API calls while the rolling throttle drains, and it expires on its own so there's no restore step to forget.

The subtlest fix is one number. The bun call's timeout was raised to 780 seconds because it must EXCEED the bridge's ~10.5-minute 429-backoff window. The earlier 300s timeout was killing bun mid-backoff — so a fetch would consume rate-limit budget, get cut off before the backoff paid out, and return nothing, then try again. A timeout shorter than the backoff it's supposed to wait through doesn't cap the work; it just guarantees the work never finishes.

Worked example

The rate-limit defenses are all readable in one file — and the device-count signal is one cat away.

grep -nE 'COOLDOWN|PROBE_CAP|429|quiet|timeout: int' /Users/tom/Claude/PDB/scripts/rm_pull.py
cat '/Users/tom/Claude/PDB/data/rm-device-count.json'
▶ Do it now
  1. Read lines 53–61 and 100–104 of rm_pull.py — the cooldown/probe comment block and the '780s must exceed the backoff window' comment — and say in one sentence why a 300s timeout was worse than no timeout.
  2. Run the cat above and note today's device doc count; anything creeping toward 100 is your early-warning gauge.
  3. Check whether PDB/data/rm-quiet-until exists right now — if it does, the poller is in a deliberate zero-API cooldown, not broken; read the epoch and see when it lifts.

Gotchas

Go deeper: MDN: HTTP 429 Too Many Requests · Exponential backoff and jitter (AWS Architecture)
One-card takeaway

A 429 is an instruction, not a failure — obey it with backoff, bound it with cooldowns, and make every timeout longer than the wait it has to sit through.