Skip to content

FCC Monitoring

A running FCC deployment emits three distinct signal streams: traces (span data from the FccTracer), metrics (time-series from FccMetrics), and events (pub/sub from the event bus, 81 event types). This guide walks you through wiring all three into a production observability stack.

The reference module lives at src/fcc/observability/:

  • tracing.py — SpanData, SpanContext, FccTracer, @traced decorator
  • metrics.py — MetricPoint, FccMetrics (7 pre-defined metrics)
  • exporters.py — ConsoleSpanExporter, JsonFileSpanExporter, OTel exporters
  • integration.py — instrument_simulation_engine(), instrument_action_engine()
  • dashboard_bridge.py — bridge from FCC metrics into the terminal dashboards

The Three Streams

flowchart LR
    FCC[FCC Services] --> Traces[FccTracer<br/>span data]
    FCC --> Metrics[FccMetrics<br/>time series]
    FCC --> Events[EventBus<br/>81 event types]
    Traces --> OTel[OTel Collector]
    Metrics --> OTel
    Events --> Bus[Subscriber Plugin]
    Bus --> OTel
    OTel --> Jaeger[(Jaeger / Tempo)]
    OTel --> Prom[(Prometheus)]
    Prom --> Grafana[(Grafana)]
    Jaeger --> Grafana

    classDef src fill:#c8e6c9,stroke:#2e7d32;
    classDef stream fill:#fff9c4,stroke:#f57f17;
    classDef sink fill:#e3f2fd,stroke:#0d47a1;
    class FCC src;
    class Traces,Metrics,Events,Bus stream;
    class OTel,Jaeger,Prom,Grafana sink;

Enabling OpenTelemetry

OpenTelemetry is optional-by-default — the module only imports opentelemetry-api / opentelemetry-sdk when you opt in. Install the extra first:

pip install 'fcc[observability]'
# or the umbrella:
pip install 'fcc[full]'

Programmatic wiring

from fcc.observability import FccTracer, FccMetrics
from fcc.observability.exporters import JsonFileSpanExporter
from fcc.observability.integration import (
    instrument_simulation_engine,
    instrument_action_engine,
)
from fcc.simulation.engine import SimulationEngine

tracer = FccTracer.get()
metrics = FccMetrics.get()

# Bring-up: write to a JSON file first
tracer.add_exporter(JsonFileSpanExporter("/var/log/fcc/spans.jsonl"))

# Instrument the engines
instrument_simulation_engine(SimulationEngine, tracer, metrics)
instrument_action_engine(tracer, metrics)

OTel collector wiring

from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

tracer.add_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="otel-collector:4317"))
)

The Helm chart exposes this via values.yaml:

observability:
  otel:
    enabled: true
    endpoint: http://otel-collector.observability:4317
    service_name: fcc
    resource:
      environment: production
      region: us-east-1

The Seven Pre-defined Metrics

FccMetrics ships seven metrics that are populated automatically once you call instrument_simulation_engine and instrument_action_engine:

Metric Type Unit What it measures
fcc.simulation.duration_ms histogram ms End-to-end simulation run
fcc.simulation.steps counter steps Steps executed
fcc.action.duration_ms histogram ms Action engine wall time
fcc.action.success counter runs Successful action runs
fcc.action.failure counter runs Failed action runs
fcc.persona.invocation counter calls Persona invocations
fcc.provider.latency_ms histogram ms Underlying AI provider latency

Pair each with an alerting rule in Prometheus. A good starter set:

groups:
  - name: fcc
    rules:
      - alert: FCCBackendHealthDown
        expr: up{job="fcc-backend"} == 0
        for: 2m
      - alert: FCCProviderLatencyHigh
        expr: histogram_quantile(0.95, fcc_provider_latency_ms_bucket) > 10000
        for: 5m
      - alert: FCCActionFailureRateHigh
        expr: rate(fcc_action_failure_total[5m]) / rate(fcc_action_success_total[5m]) > 0.05
        for: 5m

Ten-Alert Starter Pack

Copy into your Prometheus rules on day one:

  1. Backend /health endpoint returning non-200
  2. Frontend pod CrashLoopBackOff > 3 restarts/10 min
  3. OTel collector connection failures > 0
  4. fcc.provider.failure events > 10/5 min
  5. P95 provider latency > 10s for 5 min
  6. Action failure rate > 5% for 5 min
  7. Event bus drop rate > 0 (overflow)
  8. JupyterLab PVC usage > 80%
  9. Streamlit pod OOMKilled > 0
  10. Helm release last-deploy timestamp > 30 days (drift)

Event Bus Monitoring

The event bus is a pub/sub channel carrying 81 event types — from simulation.started through compliance.audit.finding to persona.deliverable.ready. Two ways to observe:

1. Subscriber plugin (preferred): write a plugin that implements EventSubscriberPlugin and ship events to your log pipeline.

2. Replay from a file: use EventSerializer + EventReplay to persist events to disk then ingest asynchronously.

from fcc.messaging import EventBus, EventFilter, EventType

bus = EventBus.default()

# Example subscriber wired into structured logging
import structlog
log = structlog.get_logger()

def _forward(event):
    log.info("fcc.event", type=event.type.value, payload=event.payload)

bus.subscribe(_forward, filter=EventFilter(types=[
    EventType.SIMULATION_STARTED,
    EventType.SIMULATION_COMPLETED,
    EventType.PROVIDER_FAILURE,
]))

Dashboard Bridge

The DashboardBridge class feeds live metrics into the terminal dashboards (fcc dashboard ecosystem, fcc dashboard simulation, etc.). Use it during bring-up to get an operator's-eye view without standing up Grafana first:

fcc dashboard simulation --interval 2s

Once Grafana is wired, switch the dashboard bridge to background mode and rely on Grafana for day-to-day visibility.

SLO Recommendations

SLO Target Measurement
Backend availability 99.5% up metric over 30 days
Simulation success 99.0% 1 - failure_rate over 7 days
P95 provider latency < 5s provider_latency_ms histogram
First-byte frontend < 1s nginx access log

Start with these; tighten as you accumulate operational experience.