Skip to content

Evaluation Rubrics

Ready-to-use rubrics for instructors and self-assessment. Three rubrics are included: Persona Authoring, Scenario Analysis, and Research Project. Each uses a four-level scale (Emerging / Developing / Proficient / Excellent) across 5-7 criteria.

Rubric 1: Persona Authoring

Applied to one or more persona YAML files submitted by the student.

Criterion Emerging (C) Developing (B-) Proficient (B+) Excellent (A)
Validator compliance Fails validation Passes with warnings Passes cleanly Passes cleanly + well-chosen id, category, risk_category
Role clarity Vague or over-broad Specific but wordy One-sentence, actionable Single sentence + a clear scope-boundary signal
Constraints quality Missing or tautological Present but thin 3-5 specific, removable-matters constraints Every constraint prevents a named failure mode
Responsibilities actionability Restates role Passive / unclear when fired Active, testable during a run Active + includes graceful-failure behaviour
Collaborators realism Absent or invented Listed but not reciprocal Reciprocal + plausible Reciprocal + collaboration depth documented in a short memo
Adoption checklist Empty or token 1-2 steps 3-5 pre-use verification steps Checklist covers provenance, versioning, and failure-recovery

Guidance for self-assessment: if you cannot score yourself "Proficient" or better on every row, revise before submitting.

Rubric 2: Scenario Analysis

Applied to a student-produced analysis of a scenario run + trace.

Criterion Emerging Developing Proficient Excellent
Scenario design Single run 2-3 runs, no variable isolated Clear control + treatment Multiple seeds + proper statistical framing
Trace reading Surface claims Cites specific event IDs Quantifies with trace data Quantifies + visualizes patterns
Interpretation Restates trace Draws one conclusion Explains mechanism Explains mechanism + falsifiable prediction
Reproducibility No provided artifacts Artifacts but no command One-line reproduction make replicate + archived traces + versions pinned
Limitations Unstated Acknowledged briefly Specific + reasonable Specific + proposes a follow-up design

Rubric 3: Research Project

Applied to a term-paper scale project using FCC as instrument. Uses the four-level scale across seven criteria.

Criterion 1: Question. The research question is operational and FCC-answerable. Excellent: question + hypothesis + falsifying condition.

Criterion 2: Methods. Exactly-reproducible protocol. Excellent: pre-registered, seed set, every hyperparameter documented.

Criterion 3: Execution. All planned runs executed, results stable across re-runs. Excellent: robustness checks included.

Criterion 4: Analysis. Results summarized quantitatively with visualizations. Excellent: confidence intervals or equivalent, not just point estimates.

Criterion 5: Discussion. Interpretation + threats to validity. Excellent: threats are specific and mitigations proposed.

Criterion 6: Writing. Clear structure, correct citation, minimal jargon. Excellent: readable by a peer who has not taken the course.

Criterion 7: Reproducibility. Companion repo. Excellent: single-command replication, archived at a permanent URL.

Self-assessment checklist

Before submitting anything:

  • My persona YAMLs pass fcc personas validate.
  • My scenario JSONs pass fcc scenarios validate.
  • My report cites the FCC version I used.
  • I used --provider mock unless I explicitly justified otherwise.
  • My traces are archived, not just screenshots.
  • My claims reference specific IDs (persona IDs, finding IDs, event IDs).
  • My analysis script runs on a fresh clone.
  • I have attributed FCC in the references.

Peer-review template

When a peer asks you to review their FCC-based work, focus on three questions:

  1. Could I reproduce this? Clone, install, run. If any step fails, fix that first.
  2. Do the quantitative claims trace to evidence? Pick three claims at random. For each, find the trace event, finding ID, or test output that supports it.
  3. What is the most load-bearing assumption? Articulate it. Ask the author if it is justified.

Peer reviews that tackle these three questions are more useful to the author than any amount of line-level editing.

Self-reflection prompts

Write a paragraph for each after completing your project:

  1. What would you do differently if you had another two weeks?
  2. Which persona decision turned out to be most consequential?
  3. Which FCC feature was most surprising in practice?
  4. What is your strongest result? Your weakest?
  5. If your analysis is replicated in 12 months with FCC v2.x, what might change?

These prompts are low-effort to write and high-value for your next project.

Grading philosophy

Well-designed FCC-based assignments reward:

  • Reproducibility (explicit, testable).
  • Specificity (claims tied to IDs).
  • Honesty about limitations (promotes scientific maturity).
  • Transfer (can the student apply the same pattern to a new domain).

They de-emphasize:

  • Volume (more personas ≠ better work).
  • Model-chasing (using a bigger LLM ≠ better work).
  • Polish over correctness (nice figures ≠ correct findings).

Students who internalize these values do well on later research in any field.

Where next

Return to Research Project Workflow to plan the project these rubrics grade, or to the track index for the full reading list.