Skip to content

FCC Upgrade Guide

FCC follows a four-part PEP 440 versioning scheme (e.g. 1.3.9, 1.3.10, 1.3.10.1) and an aggressive forward-only release cadence. This guide documents the upgrade protocol that the project uses internally and recommends for any downstream deployment.

The Coverage Ratchet

The FCC project runs a coverage ratchet on every release: the CI line-coverage gate is raised by one percentage point per patch, and the Release gate matches the CI gate. As of v1.3.5.3 the gate sits at 99% line coverage with a hard --cov-branch gate at 80%.

What this means for you as an operator:

  • Any FCC release tagged v1.3.5.3+ is guaranteed to have ≥99% line coverage and ≥80% branch coverage across the Python codebase. That is stronger than most enterprise software you are running.
  • You can verify a release before you ship it: make test runs the same suite CI ran. If it does not pass locally at 99%, do not promote.
  • The fcc audit --strict command surfaces regressions in YAML data (personas, gates, requirements) that Python coverage alone cannot catch.

See the Testing and Coverage Policy for authoritative policy.

Version Cadence

Release Type Example Cadence Risk
Major 1.0.0 → 2.0.0 ~ annual Breaking API; read release notes cover-to-cover
Minor 1.3.0 → 1.4.0 ~ monthly New features, backward compatible
Patch 1.3.5 → 1.3.5.1 ad hoc (often same day as minor) Hotfix / hardening
Ratchet 1.3.5.3 ad hoc Coverage + CI policy updates

Helm chart versions are independent: Chart.yaml uses its own version field (currently 0.2.10) while appVersion tracks the FCC Python release.

The Upgrade Protocol

Step 0 — Read the Release Notes

Always start here. The project keeps per-release notes in the git tag annotation and in the CHANGELOG. Search for breaking changes, persona YAML drift, and new environment variables.

Step 1 — Stage the New Version

# Pull the new tag locally
git fetch --tags
git checkout v1.3.10

# Build the new Docker images against your registry
make docker-build REGISTRY=ghcr.io/your-org
make docker-push  REGISTRY=ghcr.io/your-org

Step 2 — Run the Full Test Suite Locally

make venv && source .venv/bin/activate
make install-dev
make test          # must hit 99% line / 80% branch
make lint          # ruff check
fcc audit --strict # YAML / persona completeness

If any of these fail, do not promote. Open a ticket upstream instead.

Step 3 — Deploy to Staging

helm upgrade fcc ./charts/fcc \
    --namespace fcc-staging \
    --values values-staging.yaml \
    --atomic \
    --timeout 10m

Watch kubectl -n fcc-staging get pods -w for the full rollout. Confirm /health returns green and that the event bus buffer depth is stable.

Step 4 — Exercise Representative Workloads

Run the scenarios your customers use most — at minimum:

fcc simulate basic_fcc_cycle --mode ai --provider anthropic
fcc compliance-audit --persona all --strict
fcc dashboard simulation

Compare trace outputs against staging-before. Large structural drift is a red flag.

Step 5 — Deploy to Production

Use --atomic and --timeout always:

helm upgrade fcc ./charts/fcc \
    --namespace fcc \
    --values values-prod.yaml \
    --atomic \
    --timeout 10m \
    --cleanup-on-fail

If the rollout fails mid-flight, Helm auto-rolls back; you do not need to intervene.

Step 6 — Post-Deploy Verification

  • fcc admin health is green
  • No fcc.provider.failure events in the first 5 minutes
  • P95 provider latency is within 20% of pre-upgrade baseline
  • No CrashLoopBackOff pods
  • Synthetic probe of a real workflow completes successfully

Breaking-Change Patterns

These patterns have all appeared in past minor bumps. If you see any of them in the release notes, treat the upgrade as high-risk:

Pattern Example Mitigation
Persona schema change v1.2.0 rich schema v2 for vertical packs Regenerate local persona YAML via fcc scaffold
Event type addition New EventType enum values Update subscriber filters
CLI rename fcc benchmark → fcc evaluate Update runbooks and scripts
Plugin protocol addition v1.2.1 VocabularyProviderPlugin Existing plugins continue to work, but newly shipped data may require one

Rollback

helm rollback fcc <previous-revision> --namespace fcc

If the rollback itself fails (rare; the chart is idempotent), re-install the previous revision explicitly:

helm uninstall fcc --namespace fcc
helm install fcc ./charts/fcc \
    --namespace fcc \
    --values values-prod.yaml \
    --version <previous-chart-version>

Pre-Production Testing Matrix

Build this matrix before you promote any release. Save it to the runbook so the next operator can reuse it.

Dimension Coverage Test
Backend health Probe + fcc admin health Must be green
Persona load fcc audit --strict Zero tier-1 findings
AI provider One real call per enabled provider Returns non-empty response
Workflow parity 3 canonical scenarios Trace matches baseline
Event bus 1-minute observed stream Buffer stable, no drops
OTel export 5-minute window Spans visible in backend
Helm dry-run helm upgrade --dry-run No unexpected diffs

Zero-Downtime Upgrades

Both backend and frontend deployments support rolling updates with maxUnavailable: 0, maxSurge: 1. The chart sets this by default in values-prod.yaml. Zero-downtime is guaranteed only if:

  • Backend replica count ≥ 2
  • /health readiness probe is enabled (it is, by default)
  • Sessions are either stateless or persisted through the CollaborationEngine recorder

If any of these are not true, expect a brief window of errored WebSocket connections during rollout.