Skip to content

FCC crash recovery

Written during the 2026-09-03 recovery, in which a hard crash + WSL reinstall destroyed $HOME and with it FCC's venv, all 8 CI runners, the Claude Code memory, and ~/.fcc/artifacts.db. FCC was the seventh project in the ecosystem to recover and the only one that had no host scaffolding at all — which is why it lost more than its peers.

D-017 (RC, 2026-08-10). Neither cron nor systemd survives a distro reinstall — so durability cannot mean "the scheduler persists". It means the installer is committed, idempotent, and one command.

§0 PROBES THAT LIE

Every probe here reported something true about an instant and false about the state. This table only grows by someone being burned — when you find a probe lying, add a row immediately.

Probe How it lies Check instead
make <anything> Makefile selects .venv/bin/python via $(wildcard). A dangling symlink still matches a wildcard, so after $HOME died every target silently ran a dead interpreter instead of falling back to python3. readlink -f .venv/bin/python — it must resolve to a real file.
make venv Runs python3 -m venv .venv without --clear. On an existing dir it exits 0 and leaves the dangling symlink exactly as it was. mv .venv .venv.broken-<date> first, then rebuild. Never delete — keep it as evidence.
CI run status A self-hosted job with no runner is not reported as failed. It queues and is cancelled at exactly 24h0m0Xs. Reads as "someone cancelled it", means "no runner has existed for a day". gh run list and look at the duration. 24h0m is a starvation signature, not a cancellation.
gh api .../actions/runners Returns registrations, not capacity. Eight offline ghosts look like a pool. They answer nothing. GitHub also ages offline runners out after ~14 days, so the count can silently drop to 0 between checks. Compare online against expected: bash scripts/cadence/runner_watch.sh.
command -v pandoc Was satisfied by anaconda's pandoc, so publications/scripts/build_all.sh never noticed it was looking for Quarto in the wrong place (~/.local/opt/quarto-* vs where publish.yml actually installs it, ~/.local/quarto-*). Removing anaconda turned a silent path mismatch into ERROR: pandoc not found on a host with a working Quarto. Locate the toolchain, don't hardcode a layout: command -v quarto, then <quarto>/bin/tools/x86_64/pandoc. Never depend on another toolchain's incidental contents.
command -v docker-compose On WSL2 the Windows PATH is inherited, so Docker Desktop's /mnt/c/Program Files/Docker/.../docker-compose satisfies it while Ubuntu's docker.io package ships no Compose v2 plugin. docker compose -f ... then fails with unknown shorthand flag: 'f', which reads as a bad compose file. (AOME documented the same shim trap for docker itself.) docker compose version. Fix without sudo: drop the plugin binary into ~/.docker/cli-plugins/docker-compose (where docker-buildx already lives).
The hostname in systemd logs Rocinante is the hostname set inside the WSL distro, not the distro name. Every runner log line reads ... Rocinante systemd[1]: Started ..., which looks like a different machine than the Ubuntu-24.04 you installed onto. echo $WSL_DISTRO_NAME inside, or wsl.exe -l -v from Windows (the * marks the default). Both say Ubuntu-24.04.
A just-started runner reading offline The GitHub API is queried before the listener finishes its first handshake. Two of eight read offline ~1s after svc.sh start and were online moments later — and an offline runner is also what the ghost sweep deletes. Re-query after a few seconds before concluding anything. make runner-status compares online against expected.
A runner's status: offline Does not mean "ghost". svc.sh install needs root, and this host has no passwordless sudo, so registration is necessarily staged — FCC_SKIP_SVC=1 leaves slots registered, real, and offline. A sweep that deletes every offline entry deleted all 8 of ours on 2026-09-03. Cross-check against the local install: a ghost is offline and has no ~/actions-runner-fcc-N/.runner claiming its name.
systemctl --user is-active <unit> Says active for a unit that has done nothing, and for a listener that subscribed to nothing. Judge by effect: artifact freshness (runner_watch.json mtime, newest bundle) and a NEXT time in systemctl --user list-timers.
git bundle verify True about the instant it ran. "Verified at 22:36" says nothing at 09:21, and nothing about whether the bundle holds the current tip. The probe is honest; the tense is not. Also assert git bundle list-heads <b> \| awk '$2=="refs/heads/main"{print $1}' equals git rev-parse main.
mkdir -p on a backup path If the mount is absent it succeeds, building the tree on the rootfs underneath — inside the failure domain the backup exists to outlive. RC found PAOM's nightly bus backup had never once succeeded this way. mountpoint -q and findmnt -no FSTYPE (reject tmpfs/ramfs/devtmpfs).
chmod / cp -p on /mnt/e 9p cannot hold Unix modes. These exit 0 while the mode does not take; a 600 file lands 777 and an explicit chmod afterwards is a no-op. Stage on the rootfs where modes are real, tar from there, then tar -tvf and read the modes back.
git status clean, on a file the crash corrupted Git caches size+mtime. A crash can rewrite a file's content without changing its length, and git then never re-hashes it — so a file full of NUL bytes reports as unmodified. Measured 2026-09-04: frontend/__tests__/App.test.tsx held 245 NUL bytes in 2105, the committed blob held 0 NUL in 2105, and git status showed nothing. Scan content, not status: git ls-files -z + read each text file and test for b"\x00". A full scan of 3,721 tracked text files takes seconds and found exactly one casualty.
git checkout -- <file> restoring it It is a no-op against a stale stat cache — git believes the file already matches, so it does not write. Ran it on the corrupted file above: NUL count stayed 245. rm the file first, then git checkout HEAD -- <file>. Verify by re-reading the bytes, not by re-running git status.
git status clean Says nothing about files that are untracked and unignored. .claude/settings.local.json — the only surviving record of the runner bootstrap — was invisible to it for months. git status --ignored occasionally; name intentional exclusions in .gitignore.
make ecosystem-check Printed "not found; skipping gracefully" and exited 0 for months because ECOSYSTEM_TESTS_PATH was three ../ deep instead of five. A green gate testing nothing. Assert the path resolves before trusting the gate. Fixed 2026-09-03.
pgrep -f <pattern> Matches its own command line, so it always finds itself. Walk /proc/<pid>/cwd — a cwd cannot be spoofed by the text of the query looking for it. Kill by process group, not PID.

§1 Orient — read only, do not act

  1. git -C . fsck --no-dangling — Rule 0: do not run a build until fsck passes. Never git gc/prune/repack before it does.
  2. git status --short --branch and git log --oneline -5.
  3. Read this repo's BACKLOG.md "In flight" table and the newest docs/uat/ hand-off.
  4. Read the Claude memory index (MEMORY.md) — start with crash-recovery-*.

§2 Preserve before mutating

Nothing below is reversible if the tree is lost first.

# any local-only branch is the highest-value thing on the disk
git for-each-ref --format='%(refname:short) %(upstream:short)' refs/heads/ | awk '$2==""{print $1}'
git push -u origin <each-one>

bash scripts/git_backup.sh          # verified, current, off /mnt/d

Also copy .claude/settings.local.json — untracked by design, and the only record of host-specific setup.

Refresh the committed digest anchor afterwards, since /mnt/e grants Windows Authenticated Users Modify and a checksum stored beside its artifact cannot detect tampering:

cd /mnt/e/backups/fcc && sha256sum repo/<newest>.bundle data/fcc_vocab.sqlite \
  > <repo>/docs/runbooks/backup-digests-<date>.txt   # then commit it

§3 Rebuild the venv

mv .venv .venv.broken-<date>        # rename, never delete (see §0)
/usr/bin/python3.12 -m venv .venv   # SYSTEM python; never anaconda
source .venv/bin/activate           # required — venv-check fails without $VIRTUAL_ENV
make install-dev                    # .[dev] + the 6 local plugins

Rebuild the way CI does — never pip install -e into an empty venv. That shortcut drifted AOME by 67/165 packages. If a long-running surface needs an extra, fold it into install-dev rather than hand-patching: RC shipped a fix after a rebuilt environment came back "with the bus client absent and the message watcher mute".

§4 Verify the tree BEFORE granting anything automation rights

make lint
make test                # 98%+ line-coverage gate; a regression is real signal
fcc vocab verify         # Train 7.0 governed vocabulary store
python scripts/registry_sync_check.py

Run these when the box is quiet. Six sibling recoveries put this 24-core host at load ~27 on 2026-09-03; the suite will throw spurious timeouts under that.

§5 Restore the cadence, then the pool

Host tooling the rebuilt distro does not restore on its own (both are user-scoped, no sudo needed):

# Compose v2 — Ubuntu's docker.io ships no compose plugin; the only docker-compose
# on PATH is Docker Desktop's Windows binary, which cannot run here.
mkdir -p ~/.docker/cli-plugins && curl -fsSL -o ~/.docker/cli-plugins/docker-compose \
  https://github.com/docker/compose/releases/latest/download/docker-compose-linux-x86_64
chmod +x ~/.docker/cli-plugins/docker-compose && docker compose version

kind + kubectl are still absent; k8s-smoke.yml stays GitHub-hosted until they are installed (its header carries the one-time commands).

bash scripts/cadence/install.sh     # timers; hard-fails without linger
bash scripts/setup-fcc-runners.sh   # 8 runners, label fcc-local

⛔ Do not add ubuntu-latest to the runner labels. RC's script says to and names FCC as the example; that is correct for RC and wrong here. FCC's 31 jobs request [self-hosted, fcc-local] explicitly, and k8s-smoke.yml is deliberately GitHub-hosted until kind is installed on the runner host. The label would hijack it onto hosts that cannot run it.

§6 Cold-start checklist

Do not report recovery complete with any box unchecked.

  • git fsck --no-dangling clean; main == origin/main
  • every local-only branch pushed
  • newest bundle verifies and its main SHA equals git rev-parse main
  • digest anchor in docs/runbooks/ matches what is on /mnt/e
  • readlink -f .venv/bin/python → /usr/bin/python3.12; fcc --version correct
  • make lint clean; make test green at ≥98% line coverage
  • fcc vocab verify passes
  • systemctl --user list-timers 'fcc-*' shows a NEXT time for both timers
  • scripts/cadence/runner_watch.sh exits 0 (8 online, 0 offline)
  • a previously-cancelled scheduled workflow completes rather than timing out
  • Claude memory index non-empty and every [[wiki-link]] in the repo resolves

§7 Graceful shutdown (the mirror half)

Quiesce the taker first, then serving surfaces, then checkpoint, then verify.

  1. Let any in-flight CI finish; stop the runners (svc.sh stop) so no job is killed mid-run.
  2. bash scripts/git_backup.sh after the last commit, and verify it.
  3. Refresh the digest anchor and commit it.
  4. Leave the systemd timers alone — Persistent=true catches up what was missed.
  5. Leave the tree on main. Units ExecStart scripts from the working tree, so a reboot restarts them from whatever branch happens to be checked out.