Methodology Archive
How the JRS Evidence Development Program collects, governs, and reports evidence on whether structured pre-finalization review can identify Decision Reconstruction Risk and support decision defensibility. This work is preliminary and observational. This archive is versioned: methodology changes are recorded in the Evidence Ledger.
Measurement instrument
All review and analysis reference the JRS Codebook, which defines the five review conditions, detection criteria, examples, risks, and severity guidance. No review occurs outside the codebook's governed definitions. Conditions carry a maturity level (Experimental → Emerging → Stable → Validated); all are currently Experimental.
Data sources
- One-Minute Challenge — one constructed record, one question, optional reviewer metadata. Aggregated counts only are shown publicly.
- Extended Review — three constructed records with structured per-record questions.
- Pilot contact — practitioner inquiries (not a study input; operational).
Records used in exercises are constructed and fictional. Participation is voluntary and may be anonymous.
What is measured
- Agreement — whether reviewers converge on the same assessment of a record.
- Accuracy — agreement with an expert benchmark mapping (separate from reviewer agreement; requires benchmark datasets not yet established).
- Condition performance — per-condition agreement and dispute patterns.
Reproducibility, accuracy, and validation are distinct and are never used interchangeably. Reproducibility (consistency of repeated assessment) does not imply accuracy (correctness against a benchmark), and neither alone constitutes validation of the framework.
- Records are constructed, not sampled from real organizational populations.
- Self-selected participants; not a representative sample.
- Early sample sizes; public aggregates are withheld until a minimum threshold.
- Author-applied reviews are illustrative, not independent validation.