Skip to content

Chapter 29: The Operator Runbook

New in v1.3.10. Where the architect playbook answers "what shape should this take?", this chapter answers "how do we run it in production?". It is the operator's companion to the framework — monitoring, rollouts, incident response, and capacity planning, all in one place.

The operator's role

An FCC operator is the person who keeps a production FCC instance healthy. That is a different job from building with FCC (the developer journey) or designing with FCC (the architect playbook). Operators care about:

  • Deployment — how does new code reach production?
  • Monitoring — how do we know when something is wrong?
  • Incident response — what do we do when it breaks?
  • Capacity — when do we need to scale?
  • Upgrade — how do we move to the next version safely?
  • Cost — how much does this cost per workflow?

This chapter is the runbook for each.

The production topology

Before getting into runbook procedures, it helps to have a mental model of the production FCC stack. The v1.1.0 Docker stack (see deployment docs) ships four containers by default.

flowchart LR
    U[User / CI pipeline] -->|HTTPS| LB[Load balancer]
    LB --> BE[fcc-backend<br/>WebSocket bridge + HTTP]
    LB --> FE[fcc-frontend<br/>React app]
    BE --> ST[fcc-streamlit<br/>Dashboards]
    BE --> JU[fcc-jupyter<br/>Notebooks]
    BE -.->|events| OB[Observability<br/>OpenTelemetry]
    BE -.->|persist| DB[(State volume)]
    BE -.->|LLM| AP[AI provider<br/>Anthropic / OpenAI / Ollama / LiteLLM / vLLM]

The full Kubernetes deployment (v1.1.1+) extends this with the charts/fcc/ Helm chart which adds horizontal scaling, resource quotas, and a single-tenant namespace isolation pattern. Operators running at scale should read Helm Chart Reference next.

Monitoring — the six SLIs

FCC exposes six service-level indicators out of the box. An operator should alert on all six; each has a conservative default threshold that triggers before customer impact.

SLI Metric Default alert
Availability /health HTTP 200 rate < 99% over 5 min
Simulation latency p95 end-to-end > 30 sec
Event bus backlog subscriber queue depth > 1000 events
LLM error rate 4xx+5xx / total > 2% over 10 min
Quality-gate fail rate failed / total gates > 10% over 1 hour
Compliance audit delay time since last audit > 24 hours

The metrics are emitted via src/fcc/observability/metrics.py and land in whatever observability stack the operator configures — the exporters support Console, JSON file, and OpenTelemetry OTLP out of the box.

Deployment — the three paths

FCC supports three deployment paths with distinct operational characteristics.

Docker Compose. Single-node, dev-mode or small-team. Operators run make docker-up and are done. Failure domain is the host; if the host dies, the whole stack is down until it's rebuilt.

Kubernetes with the Helm chart. Multi-node, production. The chart ships with reasonable resource requests, pod-disruption budgets, and a horizontal pod autoscaler on the backend. Operators running at scale should start here.

Process-direct. pip install fcc and run the CLI directly. This is the library-mode deployment — no containers, no orchestration. Useful for CI integration; not recommended for long-running production workloads.

Deployment runbook

# 0. Pre-flight
make test  # full test suite green
make pub-all  # publication pipeline green
git tag v1.3.10  # tag the release

# 1. Build
make docker-build  # multi-arch images for backend + frontend + streamlit + jupyter

# 2. Smoke
docker-compose up -d  # bring up local stack
curl -f http://localhost:8000/health  # HTTP 200
docker-compose down

# 3. Production
helm upgrade --install fcc charts/fcc/ -f values-prod.yaml  # for k8s
# OR
docker-compose -f docker-compose.prod.yml up -d  # for compose

Incident response

An FCC operator should have a pre-written response playbook for the five most common incident types.

Incident 1 — Simulation engine stalled

Symptom. Simulation latency SLI breaches; no new traces landing.

Response. 1. Check fcc dashboard simulation for in-flight runs. 2. Check the LLM provider's status page (Anthropic, OpenAI, etc.). 3. If LLM is down, fail over to mock provider: export FCC_PROVIDER_DEFAULT=mock. 4. Restart fcc-backend pod.

Incident 2 — Event bus backlog

Symptom. Subscriber queue depth exceeds threshold.

Response. 1. fcc dashboard events --tail 100 to inspect recent events. 2. Identify the slow subscriber from the backlog metric. 3. Temporarily unsubscribe via the admin endpoint: curl -X DELETE /admin/subscribers/<id>. 4. File an issue; re-subscribe once the subscriber is fixed.

Incident 3 — Quality-gate regression

Symptom. Gate fail rate SLI breaches; CI pipeline red.

Response. 1. fcc validate --personas --file <suspect.yaml> to reproduce. 2. Identify which gate failed and which persona. 3. If the regression is a legitimate change, update the constitution tier; if it's a bug, roll back the change.

Incident 4 — Compliance audit stale

Symptom. Compliance audit delay SLI breaches.

Response. 1. fcc compliance audit --all to force a fresh audit. 2. Check src/fcc/compliance/pipeline.py logs for subscriber failures. 3. If the audit is blocked on a specific persona, use the evidence graph at .fcc/compliance/evidence.json to identify the failing requirement.

Incident 5 — Ecosystem federation failure

Symptom. Cross-project entity resolution timeouts.

Response. 1. fcc federation status to inspect namespace health. 2. Identify the failing namespace from the registry. 3. If a peer is down, the EntityResolver automatically demotes that namespace; no operator action is needed beyond confirming the peer's owner is aware.

The incident timeline

The diagram below shows a typical incident timeline — detection, triage, mitigation, resolution, postmortem. Each phase has owners, artifacts, and a default SLA.

sequenceDiagram
    participant M as Monitoring
    participant O as Operator
    participant E as Engineer
    participant S as Stakeholders

    M->>O: Alert (breach of SLI)
    Note over O: Triage (≤5 min)
    O->>E: Page + context
    Note over E: Mitigation (≤30 min)
    E-->>M: Restore SLI
    O->>S: Status update
    Note over E: Resolution (≤4 hr)
    E-->>O: RCA draft
    Note over O,E: Postmortem (≤7 days)
    O->>S: Postmortem published

Capacity planning

Capacity planning is usually driven by three factors.

Persona count. Each persona adds ~50KB to the registry footprint and ~10ms to startup time. The 147-persona default adds ~7MB and ~1.5s. This is negligible.

Scenario concurrency. Each concurrent simulation holds one LLM connection and ~50MB of working memory. Plan for peak concurrency, not average; a 5-engineer team running fcc simulate during a sprint review can hit 15+ concurrent runs.

Event bus throughput. The in-process event bus sustains ~10k events per second on a single backend pod. Beyond that, sharded subscribers and a persistent queue (Redis Streams, Kafka) are needed. The plugin bridge at src/fcc/messaging/plugin_bridge.py makes this swap transparent.

Upgrade procedure

FCC follows semver. Operators upgrade between patch releases (1.3.5.2 → 1.3.5.3) without ceremony; between minor releases (1.3 → 1.4) with a short runbook; between major releases (1.x → 2.x, hypothetical) with a formal migration guide.

For minor releases:

  1. Read the CHANGELOG for the target version.
  2. Review the migration section of Getting Started — Migration.
  3. Bump the version in pyproject.toml / Helm values / compose file.
  4. Run make test against your custom personas / plugins.
  5. Upgrade a canary instance first.
  6. Promote to full production after 24 hours of clean SLIs.

Component health overview

The PlantUML component diagram below shows the production dependency graph operators should keep in mind when triaging incidents. Every box is a component an operator can restart, resize, or inspect independently.

@startuml
skinparam monochrome true
skinparam shadowing false

component "fcc-backend\n(WebSocket + HTTP)" as BE
component "fcc-frontend\n(React)" as FE
component "fcc-streamlit\n(Dashboards)" as ST
component "fcc-jupyter\n(Notebooks)" as JU
component "Load Balancer" as LB
database "State Volume" as DB
cloud "AI Provider" as AI
cloud "OTLP Collector" as OTEL

LB --> BE
LB --> FE
BE --> DB
BE --> AI
BE --> OTEL
FE --> BE
ST --> BE
JU --> BE

note right of BE
  Primary restart target
  on simulation stall.
  Emits all observability.
end note
note right of DB
  Backup daily.
  Loss = audit history loss.
end note
@enduml

Daily health checks

An operator should run four checks each morning:

  1. curl -f https://<host>/health — HTTP 200
  2. fcc dashboard ecosystem — all subsystems green
  3. fcc compliance audit --summary — no new blockers
  4. Backup verification — latest snapshot within 24 hours

A persistent red on any of these is a page-worthy incident; transient reds clear themselves within the SLI alert window and warrant a postmortem only if they recur.

Cost model

A rough cost breakdown for a 20-engineer team running FCC in production:

Component Monthly cost (order of magnitude)
Compute (4 containers on k8s) $150 – $400
State volume + backups $20 – $50
Observability ingest $50 – $200
LLM calls (Anthropic, ~50k/month) $200 – $1,500
CI/CD $20 – $100
Total $440 – $2,250

The LLM line dominates; operators who enable Ollama for low-stakes workflows see LLM costs drop to near zero while keeping Anthropic for the high-stakes ones. See AI providers for the provider matrix.

Sample-prompt cross-references

Knowledge check

  1. (Short) Name the six SLIs an operator should alert on.
  2. (Short) What's the recommended response to an LLM provider outage?
  3. (Short) Which component in the cost table typically dominates?
  4. (Short) What's the first command in the deployment runbook?

Answers

  1. Availability, simulation latency, event bus backlog, LLM error rate, quality-gate fail rate, compliance audit delay.
  2. Fail over to the mock provider via export FCC_PROVIDER_DEFAULT=mock and restart the backend.
  3. LLM calls.
  4. make test — never deploy without a green test suite.

See also