Skip to content

Peer Review — Methodology

A methodology peer review provides structured, transparent feedback on the method of a proposed experiment, model, dataset, or architectural artifact — before results are generated. The methodology review is paired with a results review (OPEN-SCI-005b) once the experiment completes; separating the two prevents result-tinted evaluation of the method. Produce this artifact during the Critique phase of any FCC cycle that submits a method for sibling-project or internal review.

Template

Section 1: Review Metadata

Instructions: Record the review type, reviewer identity, and date. FCC Critique reviews are traceable by design — anonymous review is not permitted in this workflow.

Field Value
Review ID [FILL — e.g. REV-MTH-2026-001]
Reviewer [FILL — name / role]
Date [FILL]
Review type [FCC Critique / Partnership Review / IP Review / Technical Review]

Section 2: Artifact Under Review

Instructions: Identify the artifact precisely (name, version, repository, author set). Declare whether the artifact itself used LLM assistance — this is a first-class transparency signal per Frontiers 2025.

Field Value
Artifact name [FILL]
Artifact type [Experiment / Model / Dataset / ADR / Template / Code]
Version [FILL]
Authors [FILL]
LLM usage in artifact [Yes / No / Unknown — models + components]

Section 3: Methodological Rigor

Instructions: Tick only what the artifact demonstrably meets. Each unchecked item becomes a Concern in §5.

  • Hypothesis stated unambiguously (if applicable)
  • Study design matches hypothesis class
  • Preregistration filed before data collection (link: OPEN-SCI-001)
  • Sample size + stopping rule declared with power rationale
  • Analysis plan locked; multiple-comparison posture explicit
  • Assumptions explicit and falsifiable

Section 4: Reproducibility, Power, and Bias

Instructions: These four dimensions define the methodology bar. Mark each Pass / Partial / Fail and support with one sentence of evidence.

  • Reproducibility (environment, code, seeds pinned): [Pass / Partial / Fail — evidence]
  • Statistical power (α, β, effect-size justification): [Pass / Partial / Fail — evidence]
  • Bias controls (sampling, measurement, analytical): [Pass / Partial / Fail — evidence]
  • FAIR compliance (link: OPEN-SCI-002): [Pass / Partial / Fail — evidence]

Section 5: Findings

Instructions: Separate strengths, concerns (with severity and blocking flag), required changes (must address before acceptance), and suggestions (non-blocking).

  • Strengths: [FILL]
  • Concerns (severity + blocking?): [FILL]
  • Required changes: [FILL]
  • Suggestions: [FILL]

Section 6: Reviewer LLM Declaration & Decision

Instructions: The reviewer declares their own AI assistance — transparency cuts both ways. Record decision, confidence, and any follow-up review date.

  • Reviewer LLM usage: [Yes / No — models + tasks]
  • Decision: [Accept / Revise / Reject]
  • Confidence: [High / Medium / Low]
  • Follow-up review needed: [Yes — by date / No]

Adoption Checklist

  • All required sections completed
  • Artifact peer-reviewed by at least one R.I.S.C.E.A.R. peer
  • Stored in the project's designated docs location
  • Linked from README or equivalent index
  • Versioned + date-stamped, paired with a results review (OPEN-SCI-005b) when the experiment completes

References

  • PHOENIX v4.0.0 — docs/resources/templates/open-science/peer-review.md
  • ICML 2025 — Towards Transparent Peer Review, Position Paper
  • Frontiers in Research Metrics (2025) — Peer Reviewers and AI
  • Ross-Hellauer, T. (2017) — What is open peer review?, F1000Research 6
  • Center for Open Science — Open Science Lifecycle

FCC integration

This template is referenced from the Forensic Auditor persona (src/fcc/data/personas/forensic_auditor.yaml) as part of the Critique-phase evidence set — the methodology review is evidence that the FCC Critique gate was exercised meaningfully (not just ticked). The auditor pairs each methodology review with its OPEN-SCI-005b results review and flags orphaned reviews as P2 findings. See also src/fcc/data/governance/critique_protocol.yaml and src/fcc/data/governance/open_science_gates.yaml.