Custom Quality Gates

This tutorial covers the quality gate YAML format, severity levels, how to add custom validation rules, and how quality gates integrate with the simulation engine.

What Is a Quality Gate?

A quality gate is an automated checkpoint that verifies a persona's output meets defined standards. Each gate specifies:

• What persona it applies to
• What checks must pass
• What threshold score is required
• What severity level applies when checks fail

Quality gates are defined in YAML (data/governance/quality_gates.yaml) and drive both the validation framework and simulation feedback loops.

Gate YAML Format

Here is the structure of a quality gate definition:

quality_gates:
  - id: QG-RC-001
    name: Research Inventory Completeness
    persona_id: RC
    threshold: 1.0
    checks:
      - capability_matrix
      - annotated_references
      - traceability_matrix
      - capability_tags

Field Reference

Field	Type	Required	Description
id	string	Yes	Unique gate identifier (convention: QG-{PERSONA_ID}-{NUMBER})
name	string	Yes	Human-readable gate name
persona_id	string	Yes	ID of the persona this gate applies to (or ALL)
threshold	float	Yes	Minimum pass score (0.0 to 1.0)
checks	list	Yes	List of check names that must be satisfied

Threshold Interpretation

• 1.0: All checks must pass (strict)
• 0.75: 75% of checks must pass (standard)
• 0.50: Half of checks must pass (lenient)

Built-In Quality Gates

The framework includes 25 quality gates:

Gate ID	Name	Persona	Threshold	Checks
QG-RC-001	Research Inventory Completeness	RC	1.0	4 checks
QG-BC-001	Blueprint Completeness	BC	0.75	4 checks
QG-DE-001	Documentation Quality	DE	1.0	3 checks
QG-RB-001	Runbook Readiness	RB	1.0	3 checks
QG-UG-001	User Guide Accessibility	UG	1.0	3 checks
QG-RISCEAR-001	R.I.S.C.E.A.R. Extended Completeness	ALL	1.0	3 checks
QG-CIA-001	Catalog Index Completeness	CIA	1.0	3 checks
QG-UMC-001	UI Mockup Quality	UMC	0.75	4 checks
QG-STE-001	Taxonomy Consistency	STE	1.0	3 checks
QG-TS-001	Traceability Coverage	TS	1.0	3 checks
QG-BV-001	Blueprint Validation Thoroughness	BV	1.0	3 checks
QG-RIC-001	Research Inventory Automation	RIC	1.0	3 checks
QG-GCA-001	Governance Compliance Audit	GCA	1.0	3 checks
QG-DGS-001	Data Governance Integration	DGS	1.0	3 checks
QG-PTE-001	Privacy Classification Coverage	PTE	1.0	3 checks
QG-AMS-001	Anti-fact Mitigation Rigor	AMS	1.0	3 checks
QG-CO-001	Coordination Completeness	CO	1.0	3 checks
QG-SMC-001	Metrics Dashboard Accuracy	SMC	0.75	3 checks
QG-EC-001	Executive Communication Quality	EC	1.0	3 checks
QG-RS-001	Roadmap Synchronization	RS	1.0	3 checks
QG-SCP-001	Content Distribution Coverage	SCP	1.0	3 checks
QG-RCHM-001	Research Orchestration Completeness	RCHM	1.0	3 checks
QG-BCHM-001	Blueprint Orchestration Completeness	BCHM	1.0	3 checks
QG-UGCH-001	User Experience Orchestration Completeness	UGCH	1.0	3 checks
QG-RBCH-001	Operations Orchestration Completeness	RBCH	1.0	3 checks

Creating Custom Quality Gates

Step 1: Define the Gate YAML

Create a new YAML file or extend the existing quality_gates.yaml:

quality_gates:
  # Custom gate for the Security Documentation Specialist
  - id: QG-SDS-001
    name: Security Documentation Completeness
    persona_id: SDS
    threshold: 1.0
    checks:
      - threat_model_present
      - compliance_mapping_complete
      - security_review_passed
      - severity_classifications_applied

  # Stricter version of an existing gate
  - id: QG-BC-002
    name: Blueprint Security Review
    persona_id: BC
    threshold: 1.0
    checks:
      - architecture_diagrams
      - api_specifications
      - data_models
      - workflow_definitions
      - security_considerations_documented

  # Gate that applies to all personas
  - id: QG-CUSTOM-001
    name: Project Naming Convention
    persona_id: ALL
    threshold: 1.0
    checks:
      - naming_convention_followed
      - project_prefix_applied

Step 2: Load Quality Gates

import yaml
from pathlib import Path

def load_quality_gates(path):
    with Path(path).open() as f:
        data = yaml.safe_load(f)
    return data.get("quality_gates", [])

# Load built-in gates
built_in = load_quality_gates("data/governance/quality_gates.yaml")

# Load custom gates
custom = load_quality_gates("my_project/custom_gates.yaml")

# Merge (custom gates can override built-in by ID)
gate_map = {g["id"]: g for g in built_in}
for g in custom:
    gate_map[g["id"]] = g

all_gates = list(gate_map.values())
print(f"Total quality gates: {len(all_gates)}")

Severity Levels

The FCC scenario validation framework uses four severity levels:

from fcc.scenarios.models import ValidationSeverity

# Available severity levels
print(ValidationSeverity.INFO.value)      # "info"
print(ValidationSeverity.WARNING.value)   # "warning"
print(ValidationSeverity.ERROR.value)     # "error"
print(ValidationSeverity.CRITICAL.value)  # "critical"

Severity Level Guidelines

Severity	When to Use	Action Required
INFO	Minor observation, no impact on quality	None, consider for future improvement
WARNING	Noticeable issue, may affect quality	Should fix before final publication
ERROR	Significant failure, affects artifact usability	Must fix before the artifact can proceed
CRITICAL	Fundamental failure, blocks entire workflow	Immediate fix required, all downstream blocked

Adding Custom Validation Rules

Use the ValidationResult dataclass to create custom validation checks:

from fcc.scenarios.models import ValidationResult, ValidationSeverity

def check_security_documentation(artifact):
    """Custom validation rule for security documentation."""
    results = []

    # Check threat model presence
    has_threat_model = "threat_model" in artifact.get("sections", [])
    results.append(ValidationResult(
        rule="threat_model_present",
        passed=has_threat_model,
        severity=ValidationSeverity.ERROR,
        message="Threat model section is required" if not has_threat_model else "Threat model present",
    ))

    # Check compliance mapping
    has_compliance = "compliance_mapping" in artifact.get("sections", [])
    results.append(ValidationResult(
        rule="compliance_mapping_complete",
        passed=has_compliance,
        severity=ValidationSeverity.ERROR,
        message="Compliance mapping is required" if not has_compliance else "Compliance mapping present",
    ))

    # Check severity classifications
    has_severity = artifact.get("severity_classifications", False)
    results.append(ValidationResult(
        rule="severity_classifications_applied",
        passed=has_severity,
        severity=ValidationSeverity.WARNING,
        message="Severity classifications should be applied to all findings",
    ))

    return results

Integrating Gates with Simulation

Quality gates influence simulation behavior through the scenario validation framework:

from fcc.scenarios.models import SimulationResult, ArtifactResult, ValidationResult, ValidationSeverity
from datetime import datetime, timezone

# Create a simulation result with validation
result = SimulationResult(
    scenario_id="CUSTOM-001",
    name="Security Documentation Check",
    started_at=datetime.now(timezone.utc).isoformat(),
)

# Add artifact results
result.artifacts.append(ArtifactResult(
    artifact_type="threat_model",
    generated=True,
    quality_score=0.92,
    sections_present=["attack_vectors", "risk_ratings", "mitigations"],
    sections_missing=["residual_risk"],
))

# Add validation results from quality gate checks
result.validations.append(ValidationResult(
    rule="threat_model_present",
    passed=True,
    severity=ValidationSeverity.INFO,
    message="Threat model generated successfully",
))

result.validations.append(ValidationResult(
    rule="compliance_mapping_complete",
    passed=True,
    severity=ValidationSeverity.INFO,
    message="Compliance mapping covers all required controls",
))

# Check pass rate
print(f"Pass rate: {result.pass_rate:.0%}")
print(f"Result: {result.to_dict()}")

Quality Gate Best Practices

1. Start with high thresholds and lower if needed -- It is easier to relax standards than to tighten them later
2. Use meaningful check names -- Check names should clearly indicate what is being validated
3. Match severity to impact -- Reserve ERROR and CRITICAL for checks that genuinely block quality
4. One gate per persona -- Each persona should have at least one quality gate
5. Use ALL sparingly -- Gates that apply to all personas should check universal requirements only
6. Version your gates -- Increment the gate number (QG-BC-002) when creating stricter or different variants

Next Steps

• Custom Scenarios -- Create validation scenarios that exercise your quality gates
• Custom Personas -- Create personas with matching quality gates
• Governance Overview -- Understand the full governance framework