Methodology

This page is the SwarmBench measurement contract. It is published as a defensive publication: the rules are fixed before the runs, the harness is published with the scores, and any future change is a new version, not a quiet edit.

1. Frozen splits

Every run divides the corpus into practice and held-out items under an anti-Goodhart salt (csoai-flywheel-v1). Fuel — anything a submitter may learn from — is exported from practice items only; exporting from held-out raises an error in the harness. The salt re-splits between run generations, so a score earned by split leakage does not survive the next salt.

2. Three-outcome honesty

Every item ends in exactly one of three outcomes: measured, unparseable, or UNMEASURED. Coverage is a first-class field: coverage = measured / (measured + unparseable + UNMEASURED), reported alongside every score with unparseable_rate and unmeasured_rate. A benchmark that cannot parse a model's answer says so; it does not silently drop the cell.

3. n_eff accounting (the Illusion Index rule)

A group of N models is reported with its effective ensemble size n_eff — the number of independent voters its agreement pattern actually implies. Correlated members voting is one model with extra steps. The canonical case: a 33-agent council measured n_eff 1.21 of 3 and is retracted on the CSOAI refutation ledger. Any group submitted to SwarmBench receives the same accounting, published on the Illusion Index.

4. Panel aggregation rules

When several models score the same artifact (a panel — arena provisions, axis batteries), the median decides and the geometric mean is the robust centre; the arithmetic mean is reported as a transparency statistic only. Means over independent runs (per-model composites, per-site scores) remain means — they are descriptive statistics, not panel verdicts.

5. Cross-substrate requirement

A score is substrate-qualified, always. Models are joined across substrates by weights, not names — a same-named model on two substrates is not the same model unless the weights join (the v7 lesson: sov33-v7 measured 0.581 held-out on Kaggle T4 and 0.290 on M4). Where a weights-verified join exists, SwarmBench publishes the delta and flags |Δ| > 0.15 as divergent. Observed across 57 joined models: mean signed Δ 0.096, median 0.097, 17 flagged divergent.

6. Salt-split reproducibility

Reproducibility is proven at the anchor level. Every result file carries a corpus anchor (Merkle root of the provision corpus, provision count, normaliser version, anchor timestamp) written at run time. Two runs with the same anchor measured the same corpus; the run name carries the salt generation (v1 control, v2 baseline, and so on). Anchored artifacts for the current leaderboard are listed with sha256 on the leaderboard provenance table.

7. Lost-weights honesty

When weights are lost, the model is declared LOST and removed from measurement — never silently substituted by a same-named rebuild. The register: 2026-08-02_LOST-WEIGHTS.md (8 custom-weight models, blob-store wipe). A model NAME is not a model.

8. Reporting vocabulary

SwarmBench reports are measured and carry a signed attestation (via os.meok.ai/api/sign). SwarmBench does not certify, accredit, or crown. Orderings are descriptive and ship with their confidence intervals.