Day 55 of 55 · Saturday · Learning

Reconciliation - when two ledgers disagree, one is lying to someoneReview

The outbox holds 62 sent PDFs; the manifest records 26; the pull-state tracks 43. Three ledgers, one reality — reconciliation is the discipline that finds out which one is lying.

Catch-up progress
55/55
Why this matters to you

The 8/20 audit exposed the gap: every automation that reads remarkable-manifest.jsonl — naming lint, retention sweeps, returns detection — is operating on a 42% view of what was actually sent, and none of them knows it. The 8/23 hand-run device survey (32 docs, zero true duplicates) was a reconcile pass done manually; the lesson is making it a job.

A ledger is any file that claims to record what happened: the outbox directory (PDFs that were rendered to send), the manifest (sends the code remembered to log), rm-pull-state.json (documents the pull loop has seen). Each is written by a different code path at a different moment — so each drifts differently. Crashes skip the log line; early versions predate the manifest; retries double-write.

Reconciliation is the cross-check: list what ledger A believes, list what ledger B believes, and diff. It's cheap because it's read-only, and safe because the output is a report, not a mutation. The discipline is running it on a schedule instead of during a crisis — drift found weekly is a backfill; drift found during an incident is an investigation.

This is the 8/20 false-green lesson moved down a layer. Then: a health check that never consults the alert ledger launders a dead job into a pass. Now: a retention sweep that trusts the manifest silently ignores two-thirds of the documents it's supposed to govern. Any system with two records of one truth needs a third thing that compares them.

Worked example

Measure today's gap yourself — five lines, read-only.

python3 - <<'PY'
import json, pathlib
outbox = {p.stem for p in pathlib.Path('/Users/tom/Claude/FOR TOM/PROJECTS/REMARKABLE/outbox').glob('*.pdf')}
manifest = {json.loads(l).get('name','?') for l in open('/Users/tom/Claude/PDB/data/remarkable-manifest.jsonl') if l.strip()}
print(f'outbox {len(outbox)} | manifest {len(manifest)}')
print('sample orphans:', sorted(outbox - manifest)[:5])
PY
▶ Do it now
  1. Run the script above (fix the outbox path if it differs — `ls 'FOR TOM/PROJECTS/REMARKABLE'` to check) and get today's real orphan count.
  2. Eyeball five orphans: are they pre-manifest history (backfill candidates) or post-manifest sends that skipped logging (a live bug)?
  3. Decide the schedule: add a reconcile leg to the weekly retention job that fires a 🚨 Things task when the orphan count GROWS — growth means the leak is live, not historical.

Gotchas

Go deeper: Reconciliation in data pipelines (Google SRE workbook, data integrity)
One-card takeaway

Two records of one truth always drift; the system is only honest if something compares them on a schedule.