Skip to content

ADR-014: Chromium sandbox + puppeteer-config for mermaid rendering

Date: 2026-04-24 Status: Accepted Decision-makers: FCC core maintainers · Publications pipeline owners Consulted: WSL / Docker / CI runtime owners · mermaid-cli upstream (puppeteer-config contract) Informed: All publications/ contributors; authors of docs containing mermaid fences

Context and Problem Statement

The make pub-diagrams pipeline renders every ``mermaid fence indocs/*/.mdto a twin SVG/PNG pair via [mermaid-cli](https://github.com/mermaid-js/mermaid-cli) (mmdc), which launches Chromium under the hood through Puppeteer. Since v1.3.2 the pipeline has intermittently hung — parallel workers spawnN × 2Chromium instances (one for SVG, one for PNG), and under--parallel 4the resulting eight concurrent browsers would deadlock waiting for/dev/shm`, sandbox-detection, or zygote initialization.

Root cause, confirmed under WSL2 + Docker + CI:

  1. /dev/shm sizing. Chromium defaults to a shared-memory allocation (~64 MB) that is insufficient when many instances run in parallel; WSL/Docker commonly ship with an even smaller /dev/shm.
  2. Sandbox detection failures. The default Chromium sandbox needs user-namespace cloning that is not available in many CI runners, WSL kernels, and unprivileged containers. Chromium does not fail fast — it hangs during sandbox initialization.
  3. Zombie zygote processes. When a worker is killed, its Chromium zygote can linger and block /dev/shm inodes for the next worker.

The v1.3.2 release memory journal records 23 mermaid failures that were papered over with a cache-retry loop; by v1.6.0 the loop was no longer sufficient because of docs-growth (386 diagrams).

The problem was how to make make pub-diagrams reliably complete under every target environment (WSL, Docker, CI, native Linux) without environment-specific configuration.

Decision Drivers

  • Low-touch. A fix that works everywhere without per-environment shell scripts or Docker-compose tweaks.
  • mmdc-native. Use mermaid-cli's first-class configuration surface rather than patching Puppeteer or monkey-patching Chromium.
  • Parallel-safe. The pipeline must tolerate partial stalls — a single bad diagram must not deadlock all others.
  • Reproducible. CI, local WSL, and developer Docker containers must all converge on identical behaviour.
  • Graceful degradation. Genuine syntax errors must surface as clean FAILs, not hangs.

Considered Options

  1. Enlarge /dev/shm at runtime. docker-compose up --shm-size=1g fixes Docker but does nothing for WSL, CI, or native Linux; operationally tied to the launch script, not portable to make pub-diagrams invoked by humans.
  2. Switch mmdc to a WebKit or Firefox backend. No official mermaid-cli support; experimental forks exist but none production-grade.
  3. Run an external mermaid HTTP server (e.g. mermaid-server). Adds a deployment dependency and violates the framework's pip install + standalone invariant.
  4. Ship a puppeteer-config.json with minimum-viable Chromium launch flags and pass -p to every mmdc call; pair it with a wall-clock timeout + sequential retry in the orchestrator. Low-touch, mmdc-native, environment-agnostic.

Decision

We adopt option 4: a puppeteer-config.json + orchestrator changes.

  1. publications/scripts/puppeteer-config.json (new) — minimum- viable Chromium launch args: --no-sandbox, --disable-setuid-sandbox, --disable-dev-shm-usage, --disable-gpu, --disable-software-rasterizer, --single-process, --no-zygote, --no-first-run, --disable-extensions, plus several --disable-*-throttling flags to prevent background-tab deprioritization when a worker goes idle briefly.
  2. publications/scripts/render_mermaid.py — render_one accepts a puppeteer_config_path and appends -p <path> to every mmdc invocation. A new CLI flag --puppeteer-config (default: puppeteer-config.json next to the script) wires it in.
  3. Per-future wall-clock timeout — render_all guards each parallel future with a 180 s ceiling. Futures that trip the ceiling are cancelled and their blocks are collected into a parallel_timeouts list. After the parallel pool drains, the list is retried sequentially (one Chromium at a time, no /dev/shm contention) so genuine per-diagram stalls surface as FAILs rather than hangs.
  4. Lower default --parallel from 4 to 2 in the CLI default and in the Makefile (pub-diagrams SVG + PNG passes). Two workers is the sweet spot on WSL2 (tested: 8 s for 386 diagrams).

Two mermaid-syntax fixes landed in the same roll-up as prerequisite cleanup for the pipeline:

  • docs/for-auditors/compliance-audit-workflow.md:236 — renamed classDef end to classDef finish (end is a mermaid reserved keyword that silently broke the rendering of that single diagram).
  • docs/tutorials/publications/adding-new-book-chapter.md:60 — moved the fig-meta HTML comment outside the mermaid fence so the comment isn't passed to mmdc as mermaid source.

Consequences

Positive

  • Reliable completion. make pub-diagrams consistently finishes in 8–15 s across WSL, Docker, CI, and native Linux (previously indefinite hang at --parallel 4).
  • Environment-agnostic. No docker-compose-only config, no CI-only env vars, no developer-local tweaks.
  • Per-stall isolation. Wall-clock timeout + sequential retry prevents one stuck Chromium from deadlocking the pipeline.
  • Clean FAIL surfacing. A diagram with genuine syntax errors now surfaces as a FAIL at the sequential-retry stage, with the mmdc stderr captured.
  • Observable. Timeout events print [i/N] TIMEOUT (parallel) … lines so the log immediately shows which diagram stalled.

Negative

  • Slightly slower wall-clock on hosts that could tolerate --parallel 4 (roughly 2× serial throughput instead of 4×). Acceptable — 8 s for 386 diagrams is already well within CI budget.
  • Puppeteer-config maintenance. If Chromium / Puppeteer evolves its launch API, the flag set may need a refresh. Mitigated by the config being a single 20-line JSON file.
  • Sandbox disabled. --no-sandbox is required because the Chromium sandbox cannot initialize under WSL / CI. Since mmdc runs on trusted developer/CI inputs (our own docs), the attack-surface trade-off is acceptable.

Confirmation

  • Fresh render (post-patch, pre-release): 384/386 SVG cleanly, 385/386 PNG cleanly, 4.4 s + 7.9 s end-to-end — zero hangs.
  • Chromium processes observed with all target flags via ps auxf | grep chromium during a live run.
  • make pub-diagrams succeeds from a clean cache on WSL2, in Docker (standard image, no --shm-size override), and in CI.

References

  • Mermaid reserved-word rename: classDef end → classDef finish in docs/for-auditors/compliance-audit-workflow.md
  • fig-meta HTML comment moved outside mermaid fence in docs/tutorials/publications/adding-new-book-chapter.md