$ python3 -I <work>/audit/scripts/reliability.py
# started 2026-10-28T16:43:11+00:00  ended 2026-10-28T16:43:33+00:00  exit 0

# Judge reliability — official panel (Claude Opus 5.5 high + GPT-6.1 Sol medium)

Source: production D1 export (scripts/export.py). Session scores: `result_json` (`judges.judge_a/judge_b.dims` = each judge's session mean over 5 segments; `combined` = 0.5/0.5 average, set to 0 by the wrong-language rule). Segment scores: `judgments.scores_json.dims`. Overall = published weights (Worlds 0.5, Stories 0.25 with Diversity 4 : Content logic 2, Preferences 0.25).

## A. Official-panel full-protocol platform runs (group a)

| run | model | status | scored sessions | seeds | sim assignment | penalised | substituted |
|---|---|---|---|---|---|---|---|
| 1547acce31 | Doubao-Seed-Character | completed | 285 | 95 | hash | 4 | 2 |
| 44e958def4 | deepseek-v4-1-flash-261017 | completed | 285 | 95 | slot | 0 | 0 |
| cd542f9e50 | qwen-plus-character | running | 201 | 95 | slot | 0 | 0 |

Only 2 official-panel full-protocol runs are complete (two models); a third (qwen-plus-character) is still running with 201 of 285 sessions scored. Model-level statistics below therefore compare 2 models (3 with the partial run, marked †). With 2–3 models, rank correlations can only take the values ±1 (2 models) or ±1, ±1/3 (3 models) and carry almost no information about a 10+-model leaderboard.

### A1. Inter-judge agreement (judge_a = Claude Opus 5.5, judge_b = GPT-6.1 Sol; sessions with substituted segments excluded)

| set | level | dimension | n | mean A | mean B | SD A | SD B | Pearson | Spearman | MAD | ICC(2,1) | κ (pass/fail) | % agree |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| complete runs (2) | session | basics | 568 | 88.24 | 97.08 | 19.13 | 8.99 | 0.635 | 0.533 | 8.91 | 0.417 |  |  |
| complete runs (2) | session | logic | 568 | 73.1 | 80.63 | 23.56 | 21.55 | 0.623 | 0.594 | 14.86 | 0.588 |  |  |
| complete runs (2) | session | knowledge | 568 | 99.76 | 99.41 | 1.26 | 4.22 | 0.712 | 0.609 | 0.46 | 0.388 |  |  |
| complete runs (2) | session | diversity | 568 | 77.42 | 77.1 | 8.85 | 10.83 | 0.873 | 0.855 | 4.29 | 0.855 |  |  |
| complete runs (2) | session | content_logic | 568 | 94.48 | 96.51 | 9.41 | 8.84 | 0.724 | 0.608 | 3.62 | 0.705 |  |  |
| complete runs (2) | session | interaction | 568 | 98.86 | 98.34 | 1.85 | 2.23 | 0.799 | 0.639 | 0.85 | 0.761 |  |  |
| complete runs (2) | session | overall | 568 | 89.01 | 91.66 | 5.92 | 4.61 | 0.684 | 0.675 | 3.82 | 0.59 |  |  |
| complete + partial (3)† | session | basics | 769 | 81.69 | 94.33 | 24.0 | 12.82 | 0.658 | 0.608 | 12.8 | 0.45 |  |  |
| complete + partial (3)† | session | logic | 769 | 62.89 | 72.33 | 30.91 | 27.84 | 0.764 | 0.729 | 16.1 | 0.723 |  |  |
| complete + partial (3)† | session | knowledge | 769 | 99.53 | 99.11 | 1.61 | 4.16 | 0.632 | 0.62 | 0.69 | 0.423 |  |  |
| complete + partial (3)† | session | diversity | 769 | 74.78 | 74.5 | 10.6 | 12.42 | 0.896 | 0.883 | 4.44 | 0.885 |  |  |
| complete + partial (3)† | session | content_logic | 769 | 93.98 | 95.92 | 9.75 | 9.17 | 0.741 | 0.62 | 3.8 | 0.724 |  |  |
| complete + partial (3)† | session | interaction | 769 | 98.63 | 97.64 | 1.88 | 2.83 | 0.762 | 0.714 | 1.26 | 0.646 |  |  |
| complete + partial (3)† | session | overall | 769 | 85.64 | 89.11 | 8.71 | 7.0 | 0.834 | 0.806 | 4.5 | 0.743 |  |  |
| complete runs (2) | segment | basics | 2840 | 88.24 | 97.08 | 32.21 | 16.84 | 0.469 | 0.469 | 8.91 | 0.364 | 0.363 | 0.911 |
| complete runs (2) | segment | logic | 2840 | 73.1 | 80.63 | 44.34 | 39.52 | 0.523 | 0.523 | 17.54 | 0.511 | 0.511 | 0.825 |
| complete runs (2) | segment | knowledge | 2840 | 99.76 | 99.41 | 2.1 | 5.41 | 0.471 | 0.506 | 0.51 | 0.317 |  |  |
| complete runs (2) | segment | diversity | 2840 | 77.42 | 77.1 | 13.01 | 15.47 | 0.817 | 0.784 | 6.91 | 0.805 |  |  |
| complete runs (2) | segment | content_logic | 2840 | 94.48 | 96.51 | 12.46 | 11.32 | 0.617 | 0.504 | 4.32 | 0.605 |  |  |
| complete runs (2) | segment | interaction | 2840 | 98.86 | 98.34 | 2.7 | 3.18 | 0.694 | 0.575 | 1.08 | 0.674 |  |  |
| complete runs (2) | segment | overall | 2840 | 89.01 | 91.66 | 10.02 | 7.96 | 0.572 | 0.627 | 5.27 | 0.534 |  |  |
| complete + partial (3)† | segment | basics | 3845 | 81.69 | 94.33 | 38.67 | 23.13 | 0.503 | 0.503 | 12.9 | 0.411 | 0.411 | 0.871 |
| complete + partial (3)† | segment | logic | 3845 | 62.89 | 72.33 | 48.31 | 44.74 | 0.58 | 0.58 | 19.17 | 0.567 | 0.567 | 0.808 |
| complete + partial (3)† | segment | knowledge | 3845 | 99.53 | 99.11 | 2.81 | 5.82 | 0.438 | 0.525 | 0.79 | 0.342 |  |  |
| complete + partial (3)† | segment | diversity | 3845 | 74.78 | 74.5 | 14.53 | 16.86 | 0.842 | 0.815 | 7.04 | 0.832 |  |  |
| complete + partial (3)† | segment | content_logic | 3845 | 93.98 | 95.92 | 13.17 | 11.91 | 0.628 | 0.51 | 4.66 | 0.618 |  |  |
| complete + partial (3)† | segment | interaction | 3845 | 98.63 | 97.64 | 2.83 | 3.88 | 0.661 | 0.599 | 1.49 | 0.604 |  |  |
| complete + partial (3)† | segment | overall | 3845 | 85.64 | 89.11 | 12.19 | 10.15 | 0.668 | 0.695 | 6.12 | 0.627 |  |  |

Per-run rows: out/reliability_a_agreement.csv. Basics and Logic are pass/fail per segment (100 or 0); κ treats them as binary. Knowledge, Content logic and Interaction have little variance (most segments 100), so correlations on them rest on few non-ceiling cases.

### A1b. Model level: overall by each judge alone (wrong-language rule applied to both)

| model | sessions | judge_a only | judge_b only | A − B | official (combined) |
|---|---|---|---|---|---|
| Doubao-Seed-Character | 285 | 87.63 | 90.0 | -2.37 | 88.81 |
| deepseek-v4-1-flash-261017 | 285 | 89.14 | 92.07 | -2.92 | 90.6 |
| qwen-plus-character† | 201 | 76.11 | 81.91 | -5.79 | 79.01 |

Kendall τ between the judge_a-only and judge_b-only rankings: +1.00 (2 complete models), +1.00 (3 models incl. partial†).

### A1c. Dispute rates as recorded (result_json.disputed / disputes / chunkDisputes)

| model | sessions | disputed sessions | session disputes by dimension | segments with both judges | segment disputes | by dimension | thresholds (snapshot) |
|---|---|---|---|---|---|---|---|
| Doubao-Seed-Character | 285 | 96 (33.7%) | logic 51, basics 31, content_logic 27, knowledge 4, diversity 2 | 1419 | 489 (0.34/segment) | logic 244, basics 140, content_logic 92, knowledge 9, diversity 4 | session ≥15, segment ≥30 |
| deepseek-v4-1-flash-261017 | 285 | 77 (27.0%) | logic 53, basics 23, content_logic 9 | 1425 | 402 (0.28/segment) | logic 254, basics 113, content_logic 25, diversity 5, knowledge 5 | session ≥15, segment ≥30 |
| qwen-plus-character | 201 | 120 (59.7%) | logic 73, basics 70, content_logic 16, diversity 3, knowledge 2, interaction 2 | 1005 | 551 (0.55/segment) | basics 243, logic 239, content_logic 50, knowledge 13, diversity 5, interaction 1 | session ≥15, segment ≥30 |

A pass/fail dimension (Basics, Logic) is a session dispute only when the judges split on ≥ 2 segments (engine/results.ts); every split pass/fail segment is a segment dispute (gap 100 ≥ 30).

### A2. Discrimination: overall with 95% bootstrap CIs (B = 2000)

| model | sessions | seeds | overall | CI over sessions | CI over scenarios (cluster by seed) | width ratio cluster/session |
|---|---|---|---|---|---|---|
| Doubao-Seed-Character | 285 | 95 | 88.81 | [87.41, 89.98] | [87.03, 90.24] | 1.25 |
| deepseek-v4-1-flash-261017 | 285 | 95 | 90.60 | [90.00, 91.14] | [89.80, 91.40] | 1.39 |
| qwen-plus-character† | 201 | 95 | 79.01 | [77.96, 80.03] | [77.64, 80.39] | 1.32 |

Pairs (paired bootstrap over shared scenarios, seed-level means; sign-flip p, Monte Carlo 20000):

| pair | shared seeds | mean diff | paired 95% CI | sign-flip p | cluster CIs | separated (paired CI excludes 0) |
|---|---|---|---|---|---|---|
| Doubao-Seed-Character − deepseek-v4-1-flash-261017 | 95 | -1.79 | [-3.83, -0.29] | 0.0331 | overlap | yes |
| Doubao-Seed-Character − qwen-plus-character † | 95 | +9.59 | [+7.18, +11.88] | 0.0000 | separated | yes |
| deepseek-v4-1-flash-261017 − qwen-plus-character † | 95 | +11.39 | [+9.78, +13.02] | 0.0000 | separated | yes |

### A3. Ranking stability

Rank distribution under scenario bootstrap (B = 2000; resampling the shared seeds, seed-level means): 2 complete models on 95 seeds; 3 models† on 95 seeds.

| model | set | P(rank 1) | P(rank 2) | P(rank 3) |
|---|---|---|---|---|
| Doubao-Seed-Character | 2 models | 0.009 | 0.991 |  |
| deepseek-v4-1-flash-261017 | 2 models | 0.991 | 0.009 |  |
| Doubao-Seed-Character | 3 models† | 0.013 | 0.987 | 0.000 |
| deepseek-v4-1-flash-261017 | 3 models† | 0.987 | 0.013 | 0.000 |
| qwen-plus-character | 3 models† | 0.000 | 0.000 | 1.000 |

| model | zh only | en only | judge_a only | judge_b only |
|---|---|---|---|---|
| Doubao-Seed-Character | 90.92 | 86.48 | 87.63 | 90.00 |
| deepseek-v4-1-flash-261017 | 93.04 | 87.89 | 89.14 | 92.07 |
| qwen-plus-character† | 81.41 | 77.83 | 76.11 | 81.91 |

Kendall τ zh-only vs en-only: +1.00 (2 models), +1.00 (3†). judge_a-only vs judge_b-only: +1.00 (2), +1.00 (3†).

Split-half (random halves of the shared scenarios, 2000 repetitions): τ distribution, 2 models: τ=-1.00: 3.2%, τ=+1.00: 96.8%; 3 models†: τ=+0.33: 3.5%, τ=+1.00: 96.5%.

### A4. Generalisation across scenarios

Kendall's W over scenarios (each shared seed ranks the models by its 3-sample mean; tie-corrected): W = 0.025 for the 2 complete models over 95 seeds; W = 0.466 for 3 models† over 95 seeds.
2 models: the overall leader (deepseek-v4-1-flash-261017) is ahead on 55 of 95 seeds, behind on 40, tied on 0. 3 models†: 42 of 95 seeds reproduce the overall order.

Strata (the seed data carry no scenario category field — worker/src/benchmarks/minimax-rp/data/seeds.*.jsonl have id, names, settings, prologue, first user input only — so language and simulator model are the only available strata):

| stratum | Doubao-Seed-Character | deepseek-v4-1-flash-261017 | qwen-plus-character† | order |
|---|---|---|---|---|
| language zh | 90.92 | 93.04 | 81.41 | deepseek-v4-1- > Doubao-Seed-Ch > qwen-plus-char |
| language en | 86.48 | 87.89 | 77.83 | deepseek-v4-1- > Doubao-Seed-Ch > qwen-plus-char |
| simulator claude-sonnet-5.5 | 90.09 | 90.66 | 79.15 | deepseek-v4-1- > Doubao-Seed-Ch > qwen-plus-char |
| simulator gpt-6-luna | 87.63 | 90.55 | 78.86 | deepseek-v4-1- > Doubao-Seed-Ch > qwen-plus-char |
| all | 88.81 | 90.60 | 79.01 | deepseek-v4-1- > Doubao-Seed-Ch > qwen-plus-char |

Simulator strata: 1547acce31 used hash assignment, so its simulator strata are not the same scenario/sample slots as the slot runs.

# Calibration runs (group b) against MiniMax's released scores

Released-score recomputation (all 285 sessions per model, published weights) vs published leaderboard: Spearman ρ = 0.955 before the wrong-language rule, 0.955 after it.

MiniMax pairs separated by the reference rule (scenario-paired sign-flip test, Holm over 55 pairs, α = .05; tools/experiments/anchor_check.py method, MC 20000, seed 20261026): 41 of 55 on the released scores before the wrong-language rule, 34 of 55 after it (out/minimax_separated_pairs.csv).

Per calibration run (sessions = released MiniMax sessions re-scored by the run's judges; MiniMax side = released session scores before the wrong-language rule; model mean over that run's 1–2 sessions per model):

| run | protocol | panel | judges | sessions | models | session r vs MiniMax | mean bias (ours − MM) | MAE | ρ model vs MM same sessions | ρ vs MM recomputation (285) | ρ vs published | separated pairs (41) direction kept, model means | …on shared seeds | judge A vs B: session r / MAD |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| b3de5443dd | v1.2 · 2026-10-24 | official | judge_a=claude-opus-5.5(high)×0.5 + judge_b=gpt-6.1-sol(medium)×0.5 | 22 | 11 | 0.563 | +0.23 | 6.05 | 0.891 | 0.918 | 0.855 | 39/41 | 39/41 | 0.923 / 4.03 (n=22) |
| 830a56611d | v2.0 · 2026-10-24 | official | judge_a=claude-opus-5.5(high)×0.5 + judge_b=gpt-6.1-sol(medium)×0.5 | 22 | 11 | 0.675 | -3.10 | 6.23 | 0.809 | 0.927 | 0.882 | 39/41 | 39/41 | 0.810 / 8.23 (n=22) |
| 18c2e5aca5 | v2.1 · 2026-10-24 | official | judge_a=claude-opus-5.5(high)×0.5 + judge_b=gpt-6.1-sol(medium)×0.5 | 11 | 11 | 0.616 | +0.87 | 5.92 | 0.645 | 0.600 | 0.664 | 32/41 | 16/21 | 0.944 / 2.99 (n=11) |
| 89fdb6ae78 | v2.2 · 2026-10-24 | official | judge_a=claude-opus-5.5(high)×0.5 + judge_b=gpt-6.1-sol(medium)×0.5 | 11 | 11 | 0.540 | -0.18 | 6.86 | 0.482 | 0.436 | 0.464 | 30/41 | 15/21 | 0.951 / 2.89 (n=11) |
| 0a6dbf7a25 | v2.3 · 2026-10-24 | official | judge_a=claude-opus-5.5(high)×0.5 + judge_b=gpt-6.1-sol(medium)×0.5 | 11 | 11 | 0.591 | -0.21 | 6.33 | 0.564 | 0.545 | 0.518 | 32/41 | 16/21 | 0.968 / 2.77 (n=11) |
| bb0b0999ca | v2.4 · 2026-10-24 | official | judge_a=claude-opus-5.5(high)×0.5 + judge_b=gpt-6.1-sol(medium)×0.5 | 11 | 11 | 0.562 | +1.48 | 6.91 | 0.564 | 0.564 | 0.609 | 33/41 | 17/21 | 0.970 / 2.39 (n=11) |
| 967e455f80 | v3.0 · 2026-10-24 | other | judge_a=claude-sonnet-5.5(high)×0.5 + judge_b=gpt-6-luna(high)×0.5 | 11 | 11 | 0.500 | +1.79 | 6.28 | 0.455 | 0.682 | 0.800 | 36/41 | 18/21 | 0.729 / 5.44 (n=11) |
| cbfe9afde2 | v4.0 · 2026-10-25 | other | judge_a=gpt-6.1-sol(low)×0.5 + judge_b=jev-1.14-20261014×0.5 | 11 | 11 | 0.637 | +2.26 | 6.21 | 0.727 | 0.664 | 0.600 | 33/41 | 17/21 | 0.790 / 4.97 (n=11) |
| 59d3d9cb3d | v4.1 · 2026-10-25 | other | judge_a=gpt-6.1-sol(low)×0.5 + judge_b=jev-1.14-20261014×0.5 | 11 | 11 | 0.599 | +0.34 | 7.20 | 0.591 | 0.582 | 0.573 | 31/41 | 17/21 | 0.802 / 6.21 (n=11) |
| 026e02768d | v5.0 · 2026-10-25 | official | judge_a=claude-opus-5.5(high)×0.5 + judge_b=gpt-6.1-sol(medium)×0.5 | 0 |  |  |  |  |  |  |  |  |  |

None of these calibration runs uses the judge prompt in force for v6.0: the official judge_system prompt of runs 1547acce31/44e958def4/cd542f9e50 (sha256 prefix 9f6fd89243, raw/runs_prompts.json) first appears in calibration run cbfe9afde2 (v4.0, Sol-low + Jev panel). The official-panel calibration runs used the v1.2–v2.4 prompts (four different judge_system texts) or, for v5.0 (026e02768d, paused, 0 of 2 sessions completed; judge_a batch never returned), the current prompt. Their model-level numbers rest on 1 session per model (11-session runs) or 2 (22-session runs); none of the separated pairs can be tested for significance on them.


## B2. Inter-judge agreement, disputes and rankings inside the official-panel calibration runs

Pooled over the six completed official-panel calibration runs (the same released dialogues re-scored under six prompt versions, so sessions are not independent across runs):

| set | level | dimension | n | mean A | mean B | Pearson | Spearman | MAD | ICC(2,1) | κ | % agree |
|---|---|---|---|---|---|---|---|---|---|---|---|
| official calibration runs pooled (6 runs, v1.2–v2.4 prompts) | session | basics | 88 | 63.64 | 77.95 | 0.806 | 0.823 | 14.32 | 0.71 |  |  |
| official calibration runs pooled (6 runs, v1.2–v2.4 prompts) | session | logic | 88 | 56.82 | 63.18 | 0.783 | 0.757 | 15.0 | 0.771 |  |  |
| official calibration runs pooled (6 runs, v1.2–v2.4 prompts) | session | knowledge | 88 | 99.82 | 99.93 | 0.14 | 0.174 | 0.2 | 0.118 |  |  |
| official calibration runs pooled (6 runs, v1.2–v2.4 prompts) | session | diversity | 88 | 69.97 | 67.73 | 0.935 | 0.917 | 4.76 | 0.925 |  |  |
| official calibration runs pooled (6 runs, v1.2–v2.4 prompts) | session | content_logic | 88 | 94.58 | 96.57 | 0.801 | 0.596 | 2.97 | 0.761 |  |  |
| official calibration runs pooled (6 runs, v1.2–v2.4 prompts) | session | interaction | 88 | 92.32 | 92.62 | 0.953 | 0.955 | 1.2 | 0.952 |  |  |
| official calibration runs pooled (6 runs, v1.2–v2.4 prompts) | session | overall | 88 | 79.34 | 82.67 | 0.876 | 0.776 | 4.44 | 0.835 |  |  |
| official calibration runs pooled (6 runs) | segment | basics | 440 | 63.64 | 77.95 | 0.703 | 0.703 | 14.32 | 0.663 | 0.662 | 0.857 |
| official calibration runs pooled (6 runs) | segment | logic | 440 | 56.82 | 63.18 | 0.581 | 0.581 | 20.45 | 0.576 | 0.576 | 0.795 |
| official calibration runs pooled (6 runs) | segment | knowledge | 440 | 99.82 | 99.93 | 0.174 | 0.21 | 0.2 | 0.147 |  |  |
| official calibration runs pooled (6 runs) | segment | diversity | 440 | 69.97 | 67.73 | 0.9 | 0.894 | 7.27 | 0.895 |  |  |
| official calibration runs pooled (6 runs) | segment | content_logic | 440 | 94.58 | 96.57 | 0.742 | 0.561 | 3.53 | 0.72 |  |  |
| official calibration runs pooled (6 runs) | segment | interaction | 440 | 92.32 | 92.62 | 0.924 | 0.9 | 1.49 | 0.924 |  |  |
| official calibration runs pooled (6 runs) | segment | overall | 440 | 79.34 | 82.67 | 0.777 | 0.727 | 6.2 | 0.756 |  |  |

Per-run rows: out/reliability_b_agreement.csv.

| run | protocol | sessions | models | Kendall τ judge_a vs judge_b (model means) | Spearman | range of model means | disputed sessions | segment disputes | τ zh vs en (models); Kendall W |
|---|---|---|---|---|---|---|---|---|---|
| b3de5443dd | v1.2 · 2026-10-24 | 22 | 11 | +0.82 | +0.95 | 31.9 | 6 (27%) | 29 (0.26/segment) | +0.44; W(2 scenarios) = 0.79 |
| 830a56611d | v2.0 · 2026-10-24 | 22 | 11 | +0.60 | +0.74 | 36.5 | 17 (77%) | 76 (0.69/segment) | +0.45; W(2 scenarios) = 0.77 |
| 18c2e5aca5 | v2.1 · 2026-10-24 | 11 | 11 | +0.60 | +0.71 | 38.3 | 3 (27%) | 12 (0.22/segment) | n/a (1 session per model) |
| 89fdb6ae78 | v2.2 · 2026-10-24 | 11 | 11 | +0.55 | +0.65 | 40.2 | 4 (36%) | 16 (0.29/segment) | n/a (1 session per model) |
| 0a6dbf7a25 | v2.3 · 2026-10-24 | 11 | 11 | +0.64 | +0.75 | 41.0 | 5 (45%) | 24 (0.44/segment) | n/a (1 session per model) |
| bb0b0999ca | v2.4 · 2026-10-24 | 11 | 11 | +0.82 | +0.89 | 39.2 | 3 (27%) | 13 (0.24/segment) | n/a (1 session per model) |

Discrimination and ranking stability cannot be estimated inside the calibration runs: each model has 1 session (11-session runs) or 2 (one zh, one en; 22-session runs), so there is nothing to resample within a model.


outputs recomputed from frozen/ against the published results/: {'minimax_separated_pairs.csv': True, 'reliability_a.md': True, 'reliability_a_agreement.csv': True, 'reliability_b.md': True, 'reliability_b_agreement.csv': True}