Skip to content

Reading Simulation Results

This tutorial teaches you how to interpret simulation trace output: the JSON structure, message history analysis, persona action tracking, and coverage analysis.

Trace JSON Structure

Every simulation produces a traces dictionary with this structure:

{
  "generated_at": "2024-01-15T10:30:00+00:00",
  "events": [
    {
      "step": 1,
      "timestamp": "2024-01-15T10:30:00.001+00:00",
      "actor": "Research Crafter",
      "payload": "Research and document a new microservice API",
      "origin": "RC",
      "edge_label": "initial",
      "history_len": 1,
      "history": [
        {
          "actor": "Research Crafter",
          "note": "Research Crafter processed payload='...'",
          "timestamp": "2024-01-15T10:30:00.001+00:00"
        }
      ]
    }
  ]
}

Field Reference

Field Type Description
generated_at string (ISO 8601) Timestamp when the trace was created
events array Ordered list of all processing events
events[].step integer Sequential step number (1-based)
events[].timestamp string When this specific event occurred
events[].actor string Full name of the persona that processed this message
events[].payload string Accumulated message payload
events[].origin string ID of the persona that originated this message chain
events[].edge_label string The workflow edge that delivered this message (initial for the first event)
events[].history_len integer Number of processing annotations accumulated
events[].history array Full processing history with actor, note, and timestamp

Loading and Inspecting Traces

Use the trace utilities to load and summarize trace data:

from fcc.simulation.traces import load_traces, summarize_traces

# Load from file
data = load_traces("output/gen001_trace.json")

# Get a quick summary
summary = summarize_traces(data)
print(f"Generated at: {summary['generated_at']}")
print(f"Total events: {summary['event_count']}")
print(f"Unique actors: {summary['unique_actors']}")
print(f"Actor counts: {summary['actor_counts']}")

Example output:

Generated at: 2024-01-15T10:30:00+00:00
Total events: 42
Unique actors: 5
Actor counts: {'Blueprint Crafter': 12, 'Research Crafter': 10, 'Documentation Evangelist': 8, 'Runbook Crafter': 6, 'User Guide Crafter': 6}

Analyzing Message History

Each event carries its full processing history. This tells you the path a message took through the workflow:

data = load_traces("output/gen001_trace.json")

# Find the last event (deepest message chain)
last_event = data["events"][-1]
print(f"Last event at step {last_event['step']}")
print(f"History length: {last_event['history_len']}")
print(f"Path through personas:")
for entry in last_event["history"]:
    print(f"  -> {entry['actor']}")

This reveals the message's journey through the workflow graph. A message that passed through RC -> BC -> DE -> UG -> RC shows a complete Find-Create-Critique-Feedback cycle.

Tracking Persona Actions

To understand what each persona did during the simulation:

from collections import defaultdict

data = load_traces("output/gen001_trace.json")
events = data["events"]

# Group events by actor
by_actor = defaultdict(list)
for event in events:
    by_actor[event["actor"]].append(event)

# Show each persona's activity
for actor, actor_events in by_actor.items():
    print(f"\n{actor}:")
    print(f"  Events: {len(actor_events)}")
    print(f"  First step: {actor_events[0]['step']}")
    print(f"  Last step: {actor_events[-1]['step']}")
    edges = set(e["edge_label"] for e in actor_events)
    print(f"  Edge types received: {edges}")

Coverage Analysis

Coverage analysis answers: "Did every persona in the workflow participate?"

from fcc.workflow.graph import WorkflowGraph
from fcc.simulation.traces import load_traces, summarize_traces

# Load workflow and traces
graph = WorkflowGraph.from_json("data/workflows/base_sequence.json")
data = load_traces("output/gen001_trace.json")
summary = summarize_traces(data)

# Check persona coverage
expected_personas = {node.name for node in graph.nodes}
actual_personas = set(summary["actor_counts"].keys())

covered = expected_personas & actual_personas
missing = expected_personas - actual_personas

print(f"Expected personas: {len(expected_personas)}")
print(f"Covered: {len(covered)} ({', '.join(sorted(covered))})")
if missing:
    print(f"Missing: {len(missing)} ({', '.join(sorted(missing))})")
else:
    print("Full coverage achieved.")

coverage_pct = len(covered) / len(expected_personas) * 100
print(f"Coverage: {coverage_pct:.0f}%")

Edge Coverage Analysis

Beyond persona coverage, check whether all workflow edges were traversed:

data = load_traces("output/gen001_trace.json")
events = data["events"]

# Collect all edge labels that appeared in events
traversed_edges = set(e["edge_label"] for e in events if e["edge_label"] != "initial")

# Compare against workflow graph edges
graph = WorkflowGraph.from_json("data/workflows/base_sequence.json")
expected_edges = set(e.label for e in graph.edges)

covered_edges = traversed_edges & expected_edges
missing_edges = expected_edges - traversed_edges

print(f"Expected edges: {len(expected_edges)}")
print(f"Traversed: {len(covered_edges)}")
if missing_edges:
    print(f"Not traversed: {missing_edges}")

Validating Against Scenario Expectations

Compare trace results against the scenario's expected outcomes:

from fcc.scenarios.loader import ScenarioLoader
from fcc.simulation.traces import load_traces, summarize_traces

# Load scenario and trace
loader = ScenarioLoader("data/scenarios/starter_scenarios.json")
scenario = loader.get("GEN-001")
data = load_traces("output/gen001_trace.json")
summary = summarize_traces(data)

# Check expected personas participated
expected_names = set(scenario["setup"]["personas_involved"])
actual_names = set(summary["actor_counts"].keys())
all_participated = expected_names.issubset(actual_names)

print(f"All expected personas participated: {all_participated}")

# Check event count against max_steps
max_steps = scenario["setup"]["max_steps"]
print(f"Events: {summary['event_count']} (max: {max_steps})")
print(f"Within budget: {summary['event_count'] <= max_steps}")

Understanding Message Propagation

Messages grow as they traverse edges. The payload accumulates edge labels:

Step 1: "Research and document a new microservice API"
Step 2: "Research and document a new microservice API -> research_inventory"
Step 3: "Research and document a new microservice API -> research_inventory -> blueprints_specs"

The history_len field tracks how many personas have processed this message. When history_len exceeds max_history (default: 16), the message stops propagating. This prevents infinite loops in feedback cycles.

# Find the longest message chains
data = load_traces("output/gen001_trace.json")
max_history = max(e["history_len"] for e in data["events"])
print(f"Maximum history length: {max_history}")

# Show messages that hit the history limit
limit_events = [e for e in data["events"] if e["history_len"] >= 16]
print(f"Events at history limit: {len(limit_events)}")

Next Steps