Benchmark Interpretation Guide¶
Advanced Tutorial
Duration: 25-35 minutes | Level: Advanced | Prerequisites: Benchmark Assessment Demo
How to read and interpret KG benchmark reports and object model maturity assessments.
KG Benchmark Report¶
Overall Score (0.0-1.0)¶
The overall score is a weighted composite:
| Component | Weight | What It Measures |
|---|---|---|
| Node type coverage | 35% | Fraction of 9 node types present |
| Edge type coverage | 35% | Fraction of 9 edge types present |
| Orphan penalty | 15% | Penalty for unconnected nodes |
| Connectivity bonus | 15% | Bonus for high edges-per-node ratio |
Score Interpretation¶
| Score | Rating | Meaning |
|---|---|---|
| 0.90-1.00 | Excellent | Full coverage, well-connected graph |
| 0.75-0.89 | Good | Most types covered, minor gaps |
| 0.60-0.74 | Fair | Several missing types or poor connectivity |
| 0.40-0.59 | Needs Work | Significant gaps in coverage |
| 0.00-0.39 | Poor | Minimal graph, many builders missing |
Reading Node Type Coverage¶
node_counts_by_type:
persona: 102 # All personas indexed
category: 20 # All categories
action: 312 # All workflow actions
constitution: 84 # Per-persona constitutions
dimension: 56 # Persona dimensions
ecosystem_project: 11 # Ecosystem projects
deliverable: 15 # Deliverable types
concept: 18 # Ontology concepts
workflow: 5 # Workflow graph nodes
A count of 0 for any type means the corresponding builder was not run or had no data.
Reading Edge Type Coverage¶
edge_counts_by_type:
belongs_to: 102 # Persona→Category assignments
orchestrates: 4 # Champion orchestrations
champions: 4 # Champion-of relationships
interacts_with: 200 # Cross-reference interactions
produces: 312 # Action→Deliverable links
depends_on: 30 # Dependency chains
maps_to: 80 # Concept/dimension mappings
governs: 84 # Constitution→Persona governance
federation_link: 55 # Cross-namespace links
Connectivity Score¶
< 0.5— Very sparse graph, most nodes isolated0.5-1.0— Moderately connected1.0-2.0— Well connected> 2.0— Densely connected (good for knowledge discovery)
Orphan Count¶
Orphan nodes have no edges. Common causes: - Builders running in isolation without cross-references - Missing edge types between node types - Data gaps in source registries
Object Model Assessment¶
Evolution Stages¶
| Stage | Score Range | Characteristics |
|---|---|---|
| Foundational | 0.00-0.39 | Basic models, minimal relationships |
| Structured | 0.40-0.64 | Normalized schemas, some relationships |
| Semantic | 0.65-0.84 | Rich vocabulary, search capability |
| Federated | 0.85-1.00 | Cross-project, full knowledge graph |
Dimension Scores¶
| Dimension | What It Measures |
|---|---|
| Schema normalization | How well models use consistent patterns |
| Relationship richness | Depth of inter-model relationships |
| Vocabulary coverage | Cross-namespace term alignment |
| Cross-project readiness | Federation and entity resolution |
| Governance compliance | Constitution and quality gate coverage |
| Search capability | Semantic search index completeness |
| Knowledge completeness | Knowledge graph builder coverage |
Interpreting Recommendations¶
Recommendations are prioritized by impact:
- Missing node/edge types — Highest priority, add builders
- Orphan nodes — Connect isolated nodes to graph
- Low connectivity — Add cross-reference edges
- Missing vocabulary mappings — Add namespace mappings
- Low dimension scores — Focus on weakest dimensions
CI Integration¶
Baseline Comparison¶
# Generate current report
fcc benchmark kg --output current_kg_report.yaml
# Compare against baseline
python -c "
from fcc.knowledge.benchmark import load_report
baseline = load_report('src/fcc/data/knowledge/kg_benchmark_baseline.yaml')
current = load_report('current_kg_report.yaml')
delta = current.overall_score - baseline.overall_score
print(f'Delta: {delta:+.3f}')
assert delta >= 0, f'Regression: score dropped by {-delta:.3f}'
"