Publication Pipeline Mechanics

Duration: 60 minutes Level: Advanced (Contributor) Module: publications/scripts/build_all.sh, publications/scripts/render_mermaid.py, Makefile pub-* targets

This tutorial explains how make pub-all actually works. You will learn the Quarto + Pandoc + Typst + mermaid-cli orchestration under the hood, the twin processed-markdown tree trick that keeps every output format rendering correctly, how the patent profile and constellation dry-runs plug in, and how to diagnose the three most common failure modes (mermaid render failures, missing fonts, Chromium sandbox issues).

The 10,000-foot view

make pub-all
    |
    v
  pub-diagrams  (pre-render mermaid to SVG + PNG twin trees)
    |
    v
publications/scripts/build_all.sh
    |
    +---> Quarto (reveal.js, PPTX, reference cards, patent profile)
    +---> Pandoc + Typst (PDFs with mermaid diagrams embedded)
    +---> Pandoc (EPUB, DOCX, HTML)
    |
    v
publications/_output/{books,executive,presentations,reference-cards,academic,patent}/

Every make pub-* target is a thin wrapper over one or more invocations of build_all.sh. The script orchestrates four tools:

• Quarto — for reveal.js presentations, PPTX, reference cards (Typst profile), and the patent profile
• Pandoc — for everything else (books, guidebook, executive, academic)
• Typst — Quarto's bundled PDF engine (via pandoc --pdf-engine=typst)
• mermaid-cli (mmdc) — pre-renders fenced mermaid blocks to SVG + PNG

The script lives at publications/scripts/build_all.sh and can be read linearly; there are no hidden side effects.

Why pre-render mermaid?

Pandoc's direct path does not render ```mermaid fenced blocks — they pass through as raw text. That means PDFs and DOCX produced by Pandoc would contain literal mermaid source text instead of diagrams.

render_mermaid.py fixes this by:

1. Scanning every markdown file under docs/ for mermaid fenced blocks
2. Invoking mmdc to render each block to SVG + PNG (twin outputs)
3. Writing processed markdown copies under publications/_build/processed-*/ with image references in place of the fenced blocks
4. Writing a manifest.json of SHA-256 content hashes for incremental caching

Naming convention: {source_file_slug}_{diagram_index}.{svg,png}. Caching skips re-render when the content hash is unchanged, which keeps incremental builds fast.

The twin-tree trick

One markdown source produces four output formats (PDF, EPUB, DOCX, HTML), but those formats have conflicting image needs:

Format	Image format	Why
PDF (Typst engine)	SVG	Vector quality; Typst renders SVG natively
HTML	SVG	Browsers render SVG natively
EPUB	PNG	Most EPUB readers fail on SVG
DOCX	PNG	Pandoc's DOCX writer silently drops SVG

Solution: render_mermaid.py is invoked twice, once for each target family:

# SVG tree for PDF + HTML
python3 scripts/render_mermaid.py \
    --input-dir ../docs \
    --output-dir _output/assets/diagrams \
    --build-dir _build/processed-svg \
    --format pdf --parallel 4

# PNG tree for DOCX + EPUB
python3 scripts/render_mermaid.py \
    --input-dir ../docs \
    --output-dir _output/assets/diagrams \
    --build-dir _build/processed-png \
    --format docx --parallel 4

Each tree writes processed markdown to its own subdirectory. build_all.sh then picks the right tree for each output format when it invokes Pandoc.

Freshness guard

Re-rendering is slow on WSL2 cross-filesystem mounts (~10 minutes). The script skips re-render when both trees exist and are newer than every source markdown file:

needs_render() {
    local tree="$1"
    [ ! -d "${tree}" ] && return 0
    local newest_src
    newest_src=$(find ../docs -name '*.md' -printf '%T@\n' | sort -nr | head -1)
    local newest_proc
    newest_proc=$(find "${tree}" -name '*.md' -printf '%T@\n' | sort -nr | head -1)
    awk -v s="${newest_src}" -v p="${newest_proc}" 'BEGIN { exit !(s > p) }'
}

This is the guard that historically broke in v1.3.5 (the original oldest_proc comparison always fired because the first file written was always older than the most recent source touch) and was fixed in v1.3.5.1.

Book builds — the direct-Pandoc path

Books (1, 2, 3 + Guidebook) use Pandoc directly with the Typst PDF engine. Each book produces four artifacts from the same chapter sources:

pandoc -f markdown-citations --lua-filter=scripts/mermaid-filter.lua \
    --resource-path=.:_output/assets/diagrams:_output \
    "${chapters_svg[@]}" \
    -o "${OUT}/books/book1-understanding-fcc.pdf" \
    --pdf-engine=typst \
    --toc --toc-depth=2 --number-sections \
    --metadata title="Book 1: Understanding FCC" \
    --metadata author="Information Collective, LLC" \
    -V papersize=us-letter -V fontsize=11pt \
    -V margin-top=2cm -V margin-bottom=2cm \
    -V margin-left=2cm -V margin-right=2cm

The --lua-filter is a fallback for any unprocessed mermaid blocks (e.g. blocks that the pre-render missed due to syntax errors). The --resource-path tells Pandoc where to look for image references produced by the twin trees.

Typst sandbox

Typst sandboxes file access by default. Because processed markdown uses absolute paths for diagram assets, the script widens the sandbox root:

export TYPST_ROOT=/

Pandoc's DOCX / HTML / EPUB writers handle absolute paths natively without a sandbox equivalent.

Guidebook, executive, academic packages

Each of these is a Pandoc invocation with a specific chapter list and output path. The pattern is identical to books:

• Guidebook — 24 chapters from docs/guidebook/ch*_*.md → 4 formats
• Executive — 5 chapters from docs/decision-makers/*.md → PDF + DOCX
• Academic — 4 chapters from docs/for-educators/*.md → PDF + EPUB

Presentations — Quarto's reveal.js + PPTX

Presentations are .qmd files rendered by Quarto. Two output formats from one source:

quarto render presentations/executive-overview.qmd --to revealjs
quarto render presentations/executive-overview.qmd --to pptx

The revealjs output is an HTML file with embedded reveal.js; the pptx output is native PowerPoint. Both go under _output/presentations/.

Cleanup step: rm -rf presentations/*_files removes the intermediate directories Quarto creates for media assets.

Reference cards — Quarto Typst profile

The 5 reference cards live under publications/reference-cards/*.qmd. Each card is rendered via Quarto's Typst profile:

for card in reference-cards/*.qmd; do
    base=$(basename "$card" .qmd)
    quarto render "$card" --to typst
    mv "reference-cards/${base}.pdf" "${OUT}/reference-cards/${base}.pdf"
done

The intermediate .tex and .typ files are cleaned up after the move.

Patent profile — gated by docs/patent/

Patent documents render via a dedicated Quarto profile at publications/_quarto-patent.yml. The profile enforces a DRAFT watermark and an attorney-client privilege posture.

quarto render --profile patent --to pdf
quarto render --profile patent --to docx

The script checks for docs/patent/ existence before running — patent is optional and not every deployment has it. Artifacts land under _output/patent/ and are excluded from GitHub release asset whitelists (the patent package is counsel-review only; it is never shipped as a public release asset).

Constellation dry-run

make pub-constellations runs the dry-run scaffold for all 10 constellation vertical projects. The script publications/scripts/dry_run_constellations.py emits a _output/constellations/manifest.json enumerating what would be published per constellation, without actually pushing or calling gh.

Use this in CI to verify the constellation publish pipeline stays green even when no publish is triggered.

Seed-canonical dry-run

make seed-canonical-dry-run is Pillar A's scaffolding target. It writes a publications/_output/seed_canonical_manifest.json file describing what would be downloaded for each vertical (healthcare, finance, legal, insurance, energy) without performing any network I/O.

The real make seed-canonical target is gated behind FCC_ACCEPT_EXTERNAL_LICENSES=1: actual downloads require explicit owner approval because some corpora carry license terms that must be accepted before copying. v1.5.2 shipped the Makefile surface + dry-run; actual fetch machinery lands in a future owner-approved release.

i18n markers

make i18n-check-markers scans publications/**/*.qmd and frontend/src/i18n/locales/**/*.json for {{TRANSLATE_*}} markers. The baseline is permissive (exit 0 even when markers are present) — it surfaces a count so the owner can plan translation campaigns.

v1.5-specific targets

Two targets support incremental v1.5.x diagram work:

Target	Purpose
pub-diagrams-v15	Render only the 10 v1.5.0-aligned diagrams (incremental)
pub-all-v15	Full pipeline refresh anchored to v1.5.x state

pub-diagrams-v15 has a retry loop: first pass at --parallel 4, retry at --parallel 2 on failure. This mitigates Chromium sandbox races that occasionally cause mermaid renders to fail at higher parallelism.

Invoking the pipeline

One-shot full build

make pub-all

Produces 27+ artifacts across books, executive, presentations, academic, reference cards, and (if patent is present) patent. Takes ~10 minutes on a warm cache, ~20 minutes on a cold one.

Targeted builds

make pub-diagrams          # just pre-render mermaid twin trees
make pub-books             # just the 4 books (16 artifacts)
make pub-executive         # just the executive package (2 artifacts)
make pub-academic          # just the academic package (2 artifacts)
make pub-reference         # just reference cards (5 PDFs)
make pub-patent            # just the patent profile (counsel review)

Clean rebuilds

make pub-clean             # nuke _output/ and _build/
make pub-diagrams-clean    # nuke diagram assets + build intermediates only
make pub-all               # rebuild everything

Troubleshooting

Mermaid render failures

Symptom: _build/processed-*/ is missing or has fewer files than expected.

Cause: mmdc choked on a mermaid block. Common triggers:

• Parens + semicolons inside classDiagram member text (see v1.5.2 memory — three diagrams required fixes after Agent A's first pass)
• {{PLACEHOLDER}} markers that Mermaid's parser interprets as interpolations
• <br/> tags in sequenceDiagram Notes (not supported by Mermaid)

Diagnosis: run the render with verbose output:

cd publications
python3 scripts/render_mermaid.py \
    --input-dir ../docs --output-dir _output/assets/diagrams \
    --build-dir _build/processed-svg --format pdf \
    --parallel 1       # serial so errors are attributable

Errors name the source file and block index.

Chromium sandbox / WSL2

Symptom: mmdc exits with a sandbox error on WSL2 or inside a Docker container.

Fix: the pub-diagrams-v15 target already retries at lower parallelism. For persistent failures, export CHROMIUM_FLAGS="--no-sandbox" before the make call, or pin PUPPETEER_ARGS in your mermaid-config.

Missing fonts

Symptom: PDFs render but text uses fallback fonts (usually Noto Sans).

Fix: fc-list should list fonts matching your _quarto-*.yml mainfont/monofont settings. Install missing fonts with your OS package manager. The build_all.sh script warns on missing fonts but does not fail.

Pandoc DOCX image issues

Symptom: DOCX output missing images or shows broken-image placeholders.

Cause: the PNG tree was not pre-rendered (or was stale). Run make pub-diagrams to refresh both trees, then retry.

Typst OOM on large books

Symptom: pandoc ... --pdf-engine=typst fails silently on a large book.

Fix: check _output/.build-errors.log — build_all.sh redirects PDF stderr there. Typst may need a larger memory budget; not currently exposed via the pipeline flags, but you can raise system ulimit before the make call.

Patent profile missing

Symptom: pub-patent target runs but produces no output.

Cause: docs/patent/ does not exist in your checkout. Patent content is owner-gated; if you are not expecting patent artifacts, this is correct.

Verifying a clean build

After make pub-all, check the artifact count:

find publications/_output -type f \
    \( -name "*.pdf" -o -name "*.epub" -o -name "*.docx" \
       -o -name "*.pptx" -o -name "*.html" \) | wc -l

Expected: ≥27. If it is lower, inspect _output/.build-errors.log and the stdout of the failing build function call in build_all.sh.

Summary

In this tutorial you learned:

• How make pub-all delegates to publications/scripts/build_all.sh
• Why mermaid pre-render is required and how the twin SVG/PNG trees keep every format happy
• How the patent profile, reference cards, and presentations plug into Quarto via explicit quarto render calls
• How constellation, seed-canonical, and i18n targets behave as gated dry-runs in the default configuration
• Three common failure modes (mermaid syntax, Chromium sandbox, missing fonts) and the diagnosis path for each

Next steps

• Adding a new book chapter — the contributor workflow for extending the pipeline's content
• Read publications/scripts/render_mermaid.py for the full render pipeline (cache invalidation, parallel rendering, format-aware output selection)
• Experiment with make pub-diagrams-v15 to see how targeted subsets render

Related references

• Makefile — targets pub-*, seed-*, i18n-*
• publications/scripts/build_all.sh — main pipeline orchestration
• publications/scripts/render_mermaid.py — mermaid pre-render
• publications/_quarto.yml — main Quarto configuration
• publications/_quarto-patent.yml — patent profile
• docs/architecture/build-pipeline.md — build architecture diagram (if present)