Skip to content

Reproducibility Primer

Reproducibility is journalism's second-most important concept (after sourcing). In AI coverage the word is used loosely. This page gives you a precise definition usable in copy, a list of what FCC makes reproducible and what it does not, and a short checklist for evaluating a reproducibility claim.

Three definitions worth distinguishing

  • Repeatability. Same team, same equipment, same data, same answer.
  • Reproducibility. Different team, new run, same answer within tolerance.
  • Replicability. Different study, different data, same scientific conclusion.

AI coverage often uses these interchangeably. They are not. FCC is strong on the first two and silent on the third (as any framework is).

What FCC makes repeatable

By design, FCC offers bit-identical repeatability in mock mode. Given the same:

  • FCC version
  • Scenario JSON
  • Persona YAMLs
  • Random seed

the trace JSON will be byte-identical across runs. This is a strong property and relatively rare in AI systems.

In a hosted-LLM mode, repeatability drops because the upstream model is non-deterministic. However, FCC still records every prompt and response so the provenance chain is preserved.

What FCC makes reproducible

Cross-team reproducibility is a stronger test. FCC makes it practical because:

  • The framework is open source (MIT). Anyone can install it.
  • Scenarios and personas are plain-text YAML and JSON. No proprietary binary.
  • Mock mode removes the LLM variable.
  • Evidence graphs are JSON-LD, a widely supported linked-data format.

In practice, a reproducibility audit proceeds like this:

  1. Obtain the scenario JSON, persona YAMLs, FCC version, and (if used) hosted-LLM provider+model+date.
  2. pip install fcc==<version>.
  3. fcc run <scenario> --provider mock --seed <same seed>.
  4. diff the resulting trace JSON against the published one.
  5. If different, investigate which step diverges.

A journalist who wants to corroborate a claim can do this in 15 minutes on a laptop.

What FCC does not make reproducible

Be honest about the ceiling:

  • Real-world behaviour. A deployment's mock-mode trace is not the same as its production behaviour. If a vendor claims "reproduced in production," ask for hosted-LLM artifacts with provider, model, and date.
  • Downstream data. FCC is about the reasoning workflow. If the deployment integrates user data, weather feeds, or medical records, those are out of FCC's reproducibility scope.
  • Scientific generalization. FCC does not do experimental design for you. If a paper says "we tested three scenarios and concluded X," whether X replicates depends on whether those scenarios were representative.

A journalist's reproducibility checklist

When evaluating a reproducibility claim in an FCC-based paper, report, or product announcement:

  • Is the FCC version cited?
  • Are scenario JSONs attached or linked?
  • Are persona YAMLs attached or linked (or identified by ID)?
  • Is the provider stated (mock, anthropic, openai, ollama, litellm, azure)?
  • If hosted, is the model name and date stated?
  • Is the random seed stated?
  • Is the analysis script available?
  • Is there a single-command replication (e.g., make replicate)?

Count the checks. Report the count. A paper with 8/8 is exemplary; one with 2/8 is thin. If a vendor pushes back, point to this list.

Why mock mode matters in journalism

Mock mode is peculiar because no LLM is called. A purist might ask: is a mock-mode demo really "the system"? Two responses:

  • For architecture stories — stories about how the system is built — mock mode is the most faithful demonstration. You see the workflow, the personas, the traces, uncontaminated by LLM noise.
  • For performance stories — how well does it work on real cases — mock mode is insufficient. You need hosted-LLM results.

Good AI coverage makes this distinction explicit.

A small worked example

Suppose a vendor press release claims: "Crater v0.2.0 delivers 97% accuracy on records-triage with full reproducibility."

Questions to ask:

  1. 97% on what test set? Is the set available?
  2. Was "accuracy" measured in mock mode or with a hosted LLM? (If mock, 97% is suspiciously precise and likely reflects a deterministic scoring rule.)
  3. Can you (the journalist) run the reproduction? With what version?
  4. Show me the trace of a representative failure. What did the Critique persona say?

If the vendor answers all four, the story is likely solid. If they wave their hands, that is the story.

Reproducibility and research fraud

Multi-agent AI is young enough that norms are still forming. A few patterns to watch for:

  • Cherry-picking. Reporting one of ten runs without disclosing the others.
  • Version shopping. Running on an old version where performance was better and not disclosing.
  • Silent seed fixing. Running with seeds that happen to look good without disclosing search.
  • Mock-as-production. Showing mock-mode results as if they represented production.

FCC's open structure makes each of these catchable if the raw artifacts are shared. If they are not shared, that absence is the finding.

Language for copy

Usable sentences:

  • "The deployment is reproducible in the technical sense: a separate team, using the published artifacts, produced identical traces on first run."
  • "The vendor's reproducibility claim is partially substantiated: mock-mode results match, but production-mode artifacts were not provided."
  • "The researchers pinned FCC v1.7.0 and provided all scenarios, enabling independent verification."

Each sentence is specific, verifiable, and charitable-until-evidence.

Where next

Now that you can evaluate a reproducibility claim, learn to explain FCC-based systems accessibly at Explaining FCC to the Public. Or prepare for interviews at Interview Personas.