Skip to content

FCC Incident Response

This page exists because FCC is software, software fails, and someone will be paged when it does. The goal: a small, memorable set of commands that converts an incident from "what is broken?" into "here is the failure and the blast radius."

The Three First-Responder Commands

Memorise these. Every FCC incident response starts with at least one of them.

# 1. End-to-end health snapshot (shipped in v1.3.8)
fcc admin health

# 2. Strict persona / quality-gate audit
fcc audit --strict

# 3. Kubernetes/Compose service state
kubectl -n fcc get pods      # K8s
docker compose ps            # Compose

fcc admin health runs the same check battery the readiness probe uses, but with human-readable output: AI provider connectivity, event bus buffer depth, OTel exporter status, and persona / scenario loadability. Use it as the first diagnostic.

fcc audit --strict runs the 58-gate quality registry (see src/fcc/data/governance/quality_gates_extended.yaml) plus R.I.S.C.E.A.R. completeness. Use it when a customer reports "workflow output is wrong" — it catches drift between your persona YAML and the shipped specifications.

Incident Triage Flowchart

flowchart TD
    Start([Page received]) --> Health{`fcc admin health`<br/>green?}
    Health -->|No| Services{Which service<br/>is red?}
    Health -->|Yes| Traffic{Traffic error<br/>rate elevated?}

    Services -->|backend| BRun[Backend runbook]
    Services -->|frontend| FRun[Frontend runbook]
    Services -->|streamlit| SRun[Streamlit runbook]
    Services -->|jupyter| JRun[Jupyter runbook]
    Services -->|otel| ORun[Observability runbook]

    Traffic -->|Yes| Provider{`fcc.provider.failure`<br/>events?}
    Traffic -->|No| Symptoms{Customer<br/>report type?}

    Provider -->|Yes| PRun[Provider runbook]
    Provider -->|No| ARun[Audit runbook]

    Symptoms -->|wrong output| Audit[`fcc audit --strict`]
    Symptoms -->|slow| Perf[Latency runbook]
    Symptoms -->|missing persona| PR[Persona runbook]

    classDef start fill:#e3f2fd,stroke:#0d47a1;
    classDef check fill:#fff9c4,stroke:#f57f17;
    classDef run fill:#c8e6c9,stroke:#2e7d32;
    class Start start;
    class Health,Services,Traffic,Provider,Symptoms check;
    class BRun,FRun,SRun,JRun,ORun,PRun,ARun,Audit,Perf,PR run;

Runbook Library

Runbook 1 — Backend Unhealthy

Symptom: /health returns 503 or connection refused; kubectl get pods shows fcc-backend not Ready.

Checks:

  1. kubectl -n fcc logs fcc-backend -p --tail=200 — look for provider-init errors
  2. kubectl -n fcc describe pod fcc-backend — look for OOMKilled, CrashLoopBackOff, ImagePullBackOff
  3. fcc admin health --json | jq .providers — which provider is refusing connections?

Common causes:

  • ANTHROPIC_API_KEY / OPENAI_API_KEY rotated without updating the Secret
  • OTel collector unreachable; if observability.otel.enabled: true and endpoint is down, some code paths hang
  • Memory limit too low; bump backend.resources.limits.memory in values-prod.yaml

Resolution:

kubectl -n fcc rollout restart deployment/fcc-backend
kubectl -n fcc rollout status deployment/fcc-backend --timeout=5m
curl -sf http://<backend-host>:8765/health && echo OK

Runbook 2 — Provider Failure Storm

Symptom: Alert FCCActionFailureRateHigh firing; fcc.provider.failure events > 10/5min on the bus.

Checks:

  1. Identify which provider: fcc admin health --providers
  2. Check provider status page (Anthropic status, OpenAI status)
  3. Fall-back path: temporarily set FCC_DEFAULT_PROVIDER=mock for degraded-but-available mode

Resolution:

If the upstream is down, document degraded mode on the status page, flip default provider to mock (or litellm router with alternate backend), and wait. If only one persona is affected, override in the scenario setup.ai_config.

Runbook 3 — Audit Finding Storm

Symptom: Customer reports workflows producing unexpected output; fcc audit --strict returns >0 errors.

Checks:

  1. fcc audit personas --strict --report /tmp/audit.json
  2. Grep /tmp/audit.json for severity: tier_1 — these block production
  3. Compare persona YAML hashes against the last-known-good release

Resolution:

# Dry-run diff against main
git diff v1.3.9 -- src/fcc/data/personas/

# If drift detected, checkout clean
git checkout v1.3.9 -- src/fcc/data/personas/
helm upgrade fcc ./charts/fcc --atomic

See For Auditors: R.I.S.C.E.A.R. Completeness for interpretation of findings.

Runbook 4 — Event Bus Overflow

Symptom: Events being dropped; FCC_EVENT_BUS_BUFFER_SIZE exceeded.

Checks:

  1. fcc admin health --events — buffer depth and drop count
  2. Which subscriber is slow? — check subscriber latency in tracing

Resolution:

Increase FCC_EVENT_BUS_BUFFER_SIZE via the chart's backend.env section, or decouple the slow subscriber to an async worker.

Runbook 5 — Frontend 502 / 504

Symptom: Browser gets 502/504 through the ingress.

Checks:

  1. Ingress backend health: kubectl -n fcc describe svc fcc-frontend
  2. Frontend-to-backend connectivity: kubectl -n fcc exec deploy/fcc-frontend -- curl -sf http://fcc-backend:8765/health

Resolution:

Usually a stale Service selector or a probe misconfiguration. helm upgrade with --atomic to reconcile.

Runbook 6 — JupyterLab PVC Full

Symptom: Users can't save notebooks; kubectl describe pvc fcc-jupyter-data shows near 100% usage.

Resolution:

Expand the PVC (if the StorageClass supports it) or archive old notebooks. Set the PVC alert at 80% to prevent recurrence.

Degraded-Mode Checklist

When you declare an incident and shift to degraded mode:

  • Flip FCC_DEFAULT_PROVIDER=mock to keep the UI alive
  • Post the status page update
  • Freeze helm upgrade until post-mortem
  • Turn up event-bus logging to DEBUG
  • Snapshot /tmp/spans.jsonl and fcc-backend logs
  • Start the incident Slack thread with the commands run

Post-Incident Workflow

  1. Attach the fcc admin health --json snapshot to the ticket
  2. Capture the 60-minute window in the OTel backend (Jaeger / Tempo)
  3. File a learnings doc; link it from the runbook that was used
  4. Update this page's Runbook Library if a new class of incident appeared
  5. Add a regression test for the scenario if the root cause was a code defect