Skip to content

FCC for Operators

Welcome to the For Operators section of the FCC Agent Team Framework documentation. This area is written for the people who keep FCC running in production — site reliability engineers, platform operators, on-call engineers, release managers, and anyone responsible for the heartbeat of an FCC deployment after the first helm install.

If you are just learning FCC, start with For Beginners. If you are evaluating deployment strategy, consult For Decision Makers first and then return here once the green-light is given.

Who This Section Is For

Your role

This section assumes you are responsible for one of the following:

  • Running FCC services (backend WebSocket bridge, React frontend, Streamlit apps, JupyterLab) against real traffic
  • Watching dashboards, alerts, and SLOs against deployed FCC instances
  • Coordinating upgrades, patching, and cluster maintenance windows
  • Handling incidents that touch the FCC stack or its upstream persona / model providers
  • Maintaining the helm upgrade, docker compose pull, or kubectl rollout workflow for FCC

If any of those fit, the four guides in this section will take you from "the Docker image built" to "we can sleep through the night with FCC in production."

Overview of Operator-Specific Content

The For Operators section covers the runtime concerns that do not fit neatly into the per-persona or per-module documentation.

Guide Primary Audience When to Read
Deployment Guide Platform engineers, SRE First production rollout
Monitoring SRE, observability leads Hooking up OpenTelemetry + dashboards
Incident Response On-call engineers Building runbooks
Upgrade Guide Release managers, platform engineers Bumping FCC versions safely

Each guide assumes a working Kubernetes cluster or Docker host, basic familiarity with kubectl or docker compose, and that you have read Getting Started once on a laptop.

The FCC Production Surface

An operational FCC deployment has four services plus two optional add-ons. Memorise this diagram — every page in this section is organised around it.

flowchart LR
    User[End User] --> Ingress[Ingress / LB]
    Ingress --> Frontend[React Frontend<br/>:8080]
    Ingress --> Streamlit[Streamlit Apps<br/>:8501]
    Ingress --> Jupyter[JupyterLab<br/>:8888]
    Frontend --> Backend[Backend WS Bridge<br/>:8765]
    Streamlit --> Backend
    Jupyter --> Backend
    Backend --> Providers[(AI Providers<br/>Anthropic / OpenAI / Ollama)]
    Backend --> OTel[(OpenTelemetry<br/>Collector)]
    OTel --> Metrics[(Prometheus / Grafana)]
    OTel --> Traces[(Jaeger / Tempo)]

    classDef edge fill:#e3f2fd,stroke:#0d47a1,stroke-width:2px;
    classDef svc fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px;
    classDef ext fill:#fff9c4,stroke:#f57f17,stroke-width:2px;
    class User,Ingress edge;
    class Frontend,Streamlit,Jupyter,Backend svc;
    class Providers,OTel,Metrics,Traces ext;

Quick-Start for Operators

Bringing FCC from Docker Compose to production Kubernetes is usually a one-week sprint.

Step 1: Pick Your Target Environment

Three supported targets ship out of the box:

  • Docker Compose — single host, appropriate for staging, PoCs, or team previews
  • Helm on Kubernetes — production default. Chart lives at charts/fcc/ (version 0.2.10, appVersion 1.3.9+)
  • Manual — raw docker run or hand-rolled K8s manifests; use only when nothing else is possible

Step 2: Reserve Your Deployment Budget

Minimum production footprint (per the default values.yaml):

Service CPU request Memory request Replicas
backend 250m 512Mi 2
frontend 100m 128Mi 2
streamlit 200m 512Mi 1
jupyter 500m 1Gi 1

Double each value before your first real traffic day and measure with Monitoring.

Step 3: Wire In Your Secrets

FCC reads API keys from environment variables. Use Kubernetes Secret objects (see charts/fcc/templates/secret.yaml) or Docker Compose .env files. At minimum, set either ANTHROPIC_API_KEY or OPENAI_API_KEY; mock mode is for development only.

Step 4: Turn On Observability

The fcc.observability package ships OpenTelemetry-compatible tracing and metrics. Enable the JSON file exporter during bring-up, swap to an OTel collector once your dashboards are ready.

Step 5: Rehearse an Incident Before You Have One

Run through Incident Response against staging before go-live. Known-good runbooks beat heroics at 3am.

Operator Maturity Model

Stage Signals Typical Duration Next Investment
Bring-up Docker Compose works, one-environment deploy 1-2 weeks Helm chart, ingress, TLS
Hardened Helm deploy to prod, alerts wired, basic runbooks 4-8 weeks OTel collector, SLOs
Observable OTel + dashboards, SLOs defined, PagerDuty live 3-6 months Chaos drills, auto-remediation
Self-service Tenant onboarding automated, HPA proven 9+ months Multi-region, blue/green upgrades

Do not skip observability

An FCC deployment without event bus + OTel data is a black box. The FCC event bus alone emits 81 event types — wire them into your dashboards before you have a customer waiting on you to explain an outage.

Common Operator Questions

What is the smallest production footprint?

Two backend replicas, one frontend replica, and an ingress. Streamlit and Jupyter are optional. Minimum cluster: 2 vCPU, 4 GiB RAM worker node with values.yaml defaults; see Deployment Guide for sizing.

How do I roll back a bad FCC release?

helm rollback fcc <previous-revision> — the chart at charts/fcc/ is idempotent and safe to run during traffic. See Upgrade Guide.

What goes into a healthy alerting baseline?

Three alerts at minimum: backend /health failing, provider 5xx rate > 5%, and event bus drop rate > 0. See Monitoring for the full 10-alert starter pack.

Where do secrets live?

The Helm chart templates a Secret per release. For Compose, use .env and docker-compose.prod.yml overrides. Never bake API keys into images.

How does FCC behave if an AI provider is down?

The backend falls back to mock mode for affected personas only; the rest of the workflow continues. Alert on fcc.provider.failure events from the event bus; see Incident Response.

v1.4.x What's New for Operators

The v1.4.x series is the biggest operator-facing change since the v1.1.1 Helm chart. The dual-bus event split (v1.4.2) adds a second physical EventBus (UXBus), a new UX REST gateway on port 8901, per-lane OTel tracers (fcc.paom at 100%, fcc.ux at 10% sampling), and four canonical span attributes (lane, event_type, correlation_id, zachman_cell). The SAFe cadence (v1.4.1) means upgrades now land on PI boundaries rather than ad-hoc. Plan your dashboards, runbooks, and upgrade windows accordingly.

Next Steps

Pick based on immediate need:

  1. Never deployed FCC? → Deployment Guide
  2. Already deployed, dashboards empty? → Monitoring
  3. On-call rotation starts Monday? → Incident Response
  4. Planning a version bump? → Upgrade Guide
  5. Upgrading from v1.3.x to v1.4.x? → Guidebook Ch. 32 migration guide