Custom Validation Scenarios¶

This tutorial covers the scenario JSON format, how to specify personas and workflow graphs in scenarios, adding validation rules, and running custom scenarios through the simulation engine.

What Is a Scenario?¶

A scenario is a predefined simulation configuration that specifies:

What initial input to provide
Which persona starts the workflow
Which personas should participate
What artifacts are expected
What validation rules to apply
What quality thresholds must be met

Scenarios are stored as JSON files in data/scenarios/.

Scenario JSON Format¶

Here is the structure of a scenario definition:

{
  "scenarios": [
    {
      "id": "GEN-001",
      "name": "Full FCC Cycle",
      "type": "typical",
      "description": "Complete Find->Create->Critique cycle through all 5 personas",
      "objectives": [
        "Exercise all 5 personas in sequence",
        "Verify handoff edges carry correct artifacts",
        "Validate end-to-end trace generation"
      ],
      "setup": {
        "initial_input": "Research and document a new microservice API",
        "start_node": "RC",
        "personas_involved": [
          "Research Crafter",
          "Blueprint Crafter",
          "Documentation Evangelist",
          "Runbook Crafter",
          "User Guide Crafter"
        ],
        "expected_artifacts": [
          "research_inventory",
          "blueprint",
          "review_report",
          "runbook",
          "user_guide"
        ],
        "max_steps": 50
      },
      "expected_outcomes": {
        "research_inventory": {"quality_threshold": 0.85},
        "blueprint": {"quality_threshold": 0.85},
        "review_report": {"quality_threshold": 0.80},
        "runbook": {"quality_threshold": 0.80},
        "user_guide": {"quality_threshold": 0.80}
      },
      "validation_rules": [
        {"rule": "artifact_completeness", "threshold": 0.90},
        {"rule": "traceability_chain", "required": true},
        {"rule": "full_fcc_cycle", "required": true}
      ]
    }
  ]
}

Field Reference¶

Field	Type	Description
`id`	string	Unique scenario identifier (convention: `{PREFIX}-{NUMBER}`)
`name`	string	Human-readable name
`type`	string	Scenario type: `typical`, `edge_case`, or `ai`
`description`	string	What the scenario tests
`objectives`	list	Specific goals to validate
`setup.initial_input`	string	The prompt that starts the simulation
`setup.start_node`	string	Persona ID to start from
`setup.personas_involved`	list	Full names of personas expected to participate
`setup.expected_artifacts`	list	Artifact types that should be produced
`setup.max_steps`	integer	Maximum simulation steps
`setup.edge_conditions`	object	Optional edge case conditions
`setup.ai_mode`	boolean	Optional flag for AI-powered simulation
`expected_outcomes`	object	Quality thresholds per artifact
`validation_rules`	list	Validation rules with thresholds or required flags

Built-In Scenarios¶

The framework ships with 6 starter scenarios:

ID	Name	Type	Start	Personas
GEN-001	Full FCC Cycle	typical	RC	5 (all core)
GEN-002	Feedback Loop - Operational Finding	typical	RB	3 (RB, RC, BC)
GEN-003	Minimal Cycle - Research to Docs	typical	RC	3 (RC, BC, DE)
GEN-004	Edge - Incomplete Research Data	edge_case	RC	2 (RC, BC)
GEN-005	Edge - Blueprint Rejection	edge_case	BC	2 (BC, DE)
GEN-006	AI-Powered Full Cycle	ai	RC	5 (all core)

Creating Custom Scenarios¶

Step 1: Define the Scenario JSON¶

Create a new JSON file with your custom scenarios:

{
  "scenarios": [
    {
      "id": "SEC-001",
      "name": "Security Documentation Pipeline",
      "type": "typical",
      "description": "Full pipeline with security documentation specialist producing threat models and compliance mapping",
      "objectives": [
        "Exercise security-focused documentation workflow",
        "Verify threat model generation",
        "Validate compliance mapping completeness",
        "Check security review integration"
      ],
      "setup": {
        "initial_input": "Document security architecture for the payment processing API including threat model, compliance mapping to PCI DSS, and security runbooks",
        "start_node": "RC",
        "personas_involved": [
          "Research Crafter",
          "Blueprint Crafter",
          "Documentation Evangelist",
          "Runbook Crafter",
          "User Guide Crafter"
        ],
        "expected_artifacts": [
          "research_inventory",
          "blueprint",
          "threat_model",
          "compliance_mapping",
          "security_runbook",
          "review_report"
        ],
        "max_steps": 100
      },
      "expected_outcomes": {
        "research_inventory": {"quality_threshold": 0.90},
        "blueprint": {"quality_threshold": 0.90},
        "threat_model": {"quality_threshold": 0.95},
        "compliance_mapping": {"quality_threshold": 0.95},
        "security_runbook": {"quality_threshold": 0.85},
        "review_report": {"quality_threshold": 0.85}
      },
      "validation_rules": [
        {"rule": "artifact_completeness", "threshold": 0.95},
        {"rule": "traceability_chain", "required": true},
        {"rule": "security_review_passed", "required": true},
        {"rule": "compliance_controls_mapped", "threshold": 1.0}
      ]
    },
    {
      "id": "SEC-002",
      "name": "Edge - Missing Threat Data",
      "type": "edge_case",
      "description": "Handle scenario where threat intelligence data is incomplete",
      "objectives": [
        "Verify graceful handling of incomplete threat data",
        "Validate gap identification in threat model"
      ],
      "setup": {
        "initial_input": "Create threat model for legacy API with no existing security documentation",
        "start_node": "RC",
        "personas_involved": [
          "Research Crafter",
          "Blueprint Crafter"
        ],
        "expected_artifacts": [
          "research_inventory",
          "blueprint"
        ],
        "edge_conditions": {
          "incomplete_data": true,
          "missing_fields": ["threat_intelligence", "penetration_test_results"]
        },
        "max_steps": 30
      },
      "expected_outcomes": {
        "research_inventory": {"gap_analysis": true}
      },
      "validation_rules": [
        {"rule": "artifact_completeness", "threshold": 0.50},
        {"rule": "gap_handling", "required": true}
      ]
    }
  ]
}

Step 2: Load Custom Scenarios¶

Use the ScenarioLoader to load your scenarios:

from fcc.scenarios.loader import ScenarioLoader

# Load from a single file
loader = ScenarioLoader("my_project/security_scenarios.json")
print(f"Loaded {len(loader)} scenarios")
print(f"IDs: {loader.ids()}")

# Get a specific scenario
sec001 = loader.get("SEC-001")
print(f"Name: {sec001['name']}")
print(f"Type: {sec001['type']}")

Step 3: Merge with Built-In Scenarios¶

Combine your custom scenarios with the built-in ones:

# Load both
built_in = ScenarioLoader("data/scenarios/starter_scenarios.json")
custom = ScenarioLoader("my_project/security_scenarios.json")

# Merge (built_in takes precedence for duplicate IDs)
merged = built_in.merge(custom)
print(f"Total scenarios: {len(merged)}")
print(f"All IDs: {merged.ids()}")

Or load all scenarios from a directory:

# Load all *.json files from a directory
all_scenarios = ScenarioLoader.from_directory("data/scenarios/")
print(f"Loaded {len(all_scenarios)} scenarios from directory")

Step 4: Filter Scenarios¶

Query scenarios by type:

typical = loader.by_type("typical")
edge_cases = loader.by_type("edge_case")
ai_scenarios = loader.by_type("ai")

print(f"Typical: {len(typical)}")
print(f"Edge cases: {len(edge_cases)}")
print(f"AI: {len(ai_scenarios)}")

Running Custom Scenarios¶

Step 1: Load Workflow and Scenario¶

from fcc.workflow.graph import WorkflowGraph
from fcc.scenarios.loader import ScenarioLoader

graph = WorkflowGraph.from_json("data/workflows/base_sequence.json")
loader = ScenarioLoader("my_project/security_scenarios.json")
scenario = loader.get("SEC-001")

Step 2: Configure and Run Simulation¶

from fcc.simulation.engine import SimulationEngine

engine = SimulationEngine(
    graph=graph,
    max_steps=scenario["setup"]["max_steps"],
)

history = engine.run(
    start_node=scenario["setup"]["start_node"],
    initial_payload=scenario["setup"]["initial_input"],
)

print(f"Events: {len(history.events)}")

Step 3: Validate Results¶

from fcc.simulation.traces import summarize_traces
from fcc.scenarios.models import SimulationResult, ValidationResult, ValidationSeverity
from datetime import datetime, timezone

# Summarize trace output
summary = summarize_traces(history.to_traces_dict())

# Build simulation result
result = SimulationResult(
    scenario_id=scenario["id"],
    name=scenario["name"],
    started_at=datetime.now(timezone.utc).isoformat(),
)

# Check persona coverage
expected_names = set(scenario["setup"]["personas_involved"])
actual_names = set(summary["actor_counts"].keys())
all_participated = expected_names.issubset(actual_names)

result.validations.append(ValidationResult(
    rule="persona_coverage",
    passed=all_participated,
    severity=ValidationSeverity.ERROR,
    message=f"{'All' if all_participated else 'Not all'} expected personas participated",
))

# Check artifact completeness
for rule in scenario.get("validation_rules", []):
    if rule["rule"] == "artifact_completeness":
        coverage = len(actual_names) / len(expected_names) if expected_names else 0
        passed = coverage >= rule.get("threshold", 0.9)
        result.validations.append(ValidationResult(
            rule="artifact_completeness",
            passed=passed,
            severity=ValidationSeverity.ERROR,
            message=f"Coverage: {coverage:.0%} (threshold: {rule.get('threshold', 0.9):.0%})",
        ))

print(f"Pass rate: {result.pass_rate:.0%}")
for v in result.validations:
    status = "PASS" if v.passed else "FAIL"
    print(f"  [{status}] {v.rule}: {v.message}")

Scenario Best Practices¶

Use descriptive IDs -- Prefix with a project or domain code (e.g., SEC-001, API-001)
Set realistic max_steps -- Base workflow: 20-50, Extended: 100-200, Complete: 200+
Include edge cases -- Test incomplete data, rejected artifacts, and feedback loops
Define clear objectives -- Each scenario should test specific, verifiable behaviors
Set appropriate thresholds -- Match quality thresholds to your project's standards
Include AI scenarios -- If using AI-powered simulation, create "type": "ai" scenarios

Next Steps¶

Custom Personas -- Create personas referenced by your scenarios
Custom Quality Gates -- Define gates that your scenarios validate
Reading Results -- Deep-dive into simulation trace analysis