# Calibration runs (group b) against MiniMax's released scores

Released-score recomputation (all 285 sessions per model, published weights) vs published leaderboard: Spearman ρ = 0.955 before the wrong-language rule, 0.955 after it.

MiniMax pairs separated by the reference rule (scenario-paired sign-flip test, Holm over 55 pairs, α = .05; tools/experiments/anchor_check.py method, MC 20000, seed 20261026): 41 of 55 on the released scores before the wrong-language rule, 34 of 55 after it (out/minimax_separated_pairs.csv).

Per calibration run (sessions = released MiniMax sessions re-scored by the run's judges; MiniMax side = released session scores before the wrong-language rule; model mean over that run's 1–2 sessions per model):

| run | protocol | panel | judges | sessions | models | session r vs MiniMax | mean bias (ours − MM) | MAE | ρ model vs MM same sessions | ρ vs MM recomputation (285) | ρ vs published | separated pairs (41) direction kept, model means | …on shared seeds | judge A vs B: session r / MAD |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| b3de5443dd | v1.2 · 2026-10-24 | official | judge_a=claude-opus-5.5(high)×0.5 + judge_b=gpt-6.1-sol(medium)×0.5 | 22 | 11 | 0.563 | +0.23 | 6.05 | 0.891 | 0.918 | 0.855 | 39/41 | 39/41 | 0.923 / 4.03 (n=22) |
| 830a56611d | v2.0 · 2026-10-24 | official | judge_a=claude-opus-5.5(high)×0.5 + judge_b=gpt-6.1-sol(medium)×0.5 | 22 | 11 | 0.675 | -3.10 | 6.23 | 0.809 | 0.927 | 0.882 | 39/41 | 39/41 | 0.810 / 8.23 (n=22) |
| 18c2e5aca5 | v2.1 · 2026-10-24 | official | judge_a=claude-opus-5.5(high)×0.5 + judge_b=gpt-6.1-sol(medium)×0.5 | 11 | 11 | 0.616 | +0.87 | 5.92 | 0.645 | 0.600 | 0.664 | 32/41 | 16/21 | 0.944 / 2.99 (n=11) |
| 89fdb6ae78 | v2.2 · 2026-10-24 | official | judge_a=claude-opus-5.5(high)×0.5 + judge_b=gpt-6.1-sol(medium)×0.5 | 11 | 11 | 0.540 | -0.18 | 6.86 | 0.482 | 0.436 | 0.464 | 30/41 | 15/21 | 0.951 / 2.89 (n=11) |
| 0a6dbf7a25 | v2.3 · 2026-10-24 | official | judge_a=claude-opus-5.5(high)×0.5 + judge_b=gpt-6.1-sol(medium)×0.5 | 11 | 11 | 0.591 | -0.21 | 6.33 | 0.564 | 0.545 | 0.518 | 32/41 | 16/21 | 0.968 / 2.77 (n=11) |
| bb0b0999ca | v2.4 · 2026-10-24 | official | judge_a=claude-opus-5.5(high)×0.5 + judge_b=gpt-6.1-sol(medium)×0.5 | 11 | 11 | 0.562 | +1.48 | 6.91 | 0.564 | 0.564 | 0.609 | 33/41 | 17/21 | 0.970 / 2.39 (n=11) |
| 967e455f80 | v3.0 · 2026-10-24 | other | judge_a=claude-sonnet-5.5(high)×0.5 + judge_b=gpt-6-luna(high)×0.5 | 11 | 11 | 0.500 | +1.79 | 6.28 | 0.455 | 0.682 | 0.800 | 36/41 | 18/21 | 0.729 / 5.44 (n=11) |
| cbfe9afde2 | v4.0 · 2026-10-25 | other | judge_a=gpt-6.1-sol(low)×0.5 + judge_b=jev-1.14-20261014×0.5 | 11 | 11 | 0.637 | +2.26 | 6.21 | 0.727 | 0.664 | 0.600 | 33/41 | 17/21 | 0.790 / 4.97 (n=11) |
| 59d3d9cb3d | v4.1 · 2026-10-25 | other | judge_a=gpt-6.1-sol(low)×0.5 + judge_b=jev-1.14-20261014×0.5 | 11 | 11 | 0.599 | +0.34 | 7.20 | 0.591 | 0.582 | 0.573 | 31/41 | 17/21 | 0.802 / 6.21 (n=11) |
| 026e02768d | v5.0 · 2026-10-25 | official | judge_a=claude-opus-5.5(high)×0.5 + judge_b=gpt-6.1-sol(medium)×0.5 | 0 |  |  |  |  |  |  |  |  |  |

None of these calibration runs uses the judge prompt in force for v6.0: the official judge_system prompt of runs 1547acce31/44e958def4/cd542f9e50 (sha256 prefix 9f6fd89243, raw/runs_prompts.json) first appears in calibration run cbfe9afde2 (v4.0, Sol-low + Jev panel). The official-panel calibration runs used the v1.2–v2.4 prompts (four different judge_system texts) or, for v5.0 (026e02768d, paused, 0 of 2 sessions completed; judge_a batch never returned), the current prompt. Their model-level numbers rest on 1 session per model (11-session runs) or 2 (22-session runs); none of the separated pairs can be tested for significance on them.

## B2. Inter-judge agreement, disputes and rankings inside the official-panel calibration runs

Pooled over the six completed official-panel calibration runs (the same released dialogues re-scored under six prompt versions, so sessions are not independent across runs):

| set | level | dimension | n | mean A | mean B | Pearson | Spearman | MAD | ICC(2,1) | κ | % agree |
|---|---|---|---|---|---|---|---|---|---|---|---|
| official calibration runs pooled (6 runs, v1.2–v2.4 prompts) | session | basics | 88 | 63.64 | 77.95 | 0.806 | 0.823 | 14.32 | 0.71 |  |  |
| official calibration runs pooled (6 runs, v1.2–v2.4 prompts) | session | logic | 88 | 56.82 | 63.18 | 0.783 | 0.757 | 15.0 | 0.771 |  |  |
| official calibration runs pooled (6 runs, v1.2–v2.4 prompts) | session | knowledge | 88 | 99.82 | 99.93 | 0.14 | 0.174 | 0.2 | 0.118 |  |  |
| official calibration runs pooled (6 runs, v1.2–v2.4 prompts) | session | diversity | 88 | 69.97 | 67.73 | 0.935 | 0.917 | 4.76 | 0.925 |  |  |
| official calibration runs pooled (6 runs, v1.2–v2.4 prompts) | session | content_logic | 88 | 94.58 | 96.57 | 0.801 | 0.596 | 2.97 | 0.761 |  |  |
| official calibration runs pooled (6 runs, v1.2–v2.4 prompts) | session | interaction | 88 | 92.32 | 92.62 | 0.953 | 0.955 | 1.2 | 0.952 |  |  |
| official calibration runs pooled (6 runs, v1.2–v2.4 prompts) | session | overall | 88 | 79.34 | 82.67 | 0.876 | 0.776 | 4.44 | 0.835 |  |  |
| official calibration runs pooled (6 runs) | segment | basics | 440 | 63.64 | 77.95 | 0.703 | 0.703 | 14.32 | 0.663 | 0.662 | 0.857 |
| official calibration runs pooled (6 runs) | segment | logic | 440 | 56.82 | 63.18 | 0.581 | 0.581 | 20.45 | 0.576 | 0.576 | 0.795 |
| official calibration runs pooled (6 runs) | segment | knowledge | 440 | 99.82 | 99.93 | 0.174 | 0.21 | 0.2 | 0.147 |  |  |
| official calibration runs pooled (6 runs) | segment | diversity | 440 | 69.97 | 67.73 | 0.9 | 0.894 | 7.27 | 0.895 |  |  |
| official calibration runs pooled (6 runs) | segment | content_logic | 440 | 94.58 | 96.57 | 0.742 | 0.561 | 3.53 | 0.72 |  |  |
| official calibration runs pooled (6 runs) | segment | interaction | 440 | 92.32 | 92.62 | 0.924 | 0.9 | 1.49 | 0.924 |  |  |
| official calibration runs pooled (6 runs) | segment | overall | 440 | 79.34 | 82.67 | 0.777 | 0.727 | 6.2 | 0.756 |  |  |

Per-run rows: out/reliability_b_agreement.csv.

| run | protocol | sessions | models | Kendall τ judge_a vs judge_b (model means) | Spearman | range of model means | disputed sessions | segment disputes | τ zh vs en (models); Kendall W |
|---|---|---|---|---|---|---|---|---|---|
| b3de5443dd | v1.2 · 2026-10-24 | 22 | 11 | +0.82 | +0.95 | 31.9 | 6 (27%) | 29 (0.26/segment) | +0.44; W(2 scenarios) = 0.79 |
| 830a56611d | v2.0 · 2026-10-24 | 22 | 11 | +0.60 | +0.74 | 36.5 | 17 (77%) | 76 (0.69/segment) | +0.45; W(2 scenarios) = 0.77 |
| 18c2e5aca5 | v2.1 · 2026-10-24 | 11 | 11 | +0.60 | +0.71 | 38.3 | 3 (27%) | 12 (0.22/segment) | n/a (1 session per model) |
| 89fdb6ae78 | v2.2 · 2026-10-24 | 11 | 11 | +0.55 | +0.65 | 40.2 | 4 (36%) | 16 (0.29/segment) | n/a (1 session per model) |
| 0a6dbf7a25 | v2.3 · 2026-10-24 | 11 | 11 | +0.64 | +0.75 | 41.0 | 5 (45%) | 24 (0.44/segment) | n/a (1 session per model) |
| bb0b0999ca | v2.4 · 2026-10-24 | 11 | 11 | +0.82 | +0.89 | 39.2 | 3 (27%) | 13 (0.24/segment) | n/a (1 session per model) |

Discrimination and ranking stability cannot be estimated inside the calibration runs: each model has 1 session (11-session runs) or 2 (one zh, one en; 22-session runs), so there is nothing to resample within a model.