# Judge reliability — official panel (Claude Opus 5.5 high + GPT-6.1 Sol medium)

Source: production D1 export (scripts/export.py). Session scores: `result_json` (`judges.judge_a/judge_b.dims` = each judge's session mean over 5 segments; `combined` = 0.5/0.5 average, set to 0 by the wrong-language rule). Segment scores: `judgments.scores_json.dims`. Overall = published weights (Worlds 0.5, Stories 0.25 with Diversity 4 : Content logic 2, Preferences 0.25).

## A. Official-panel full-protocol platform runs (group a)

| run | model | status | scored sessions | seeds | sim assignment | penalised | substituted |
|---|---|---|---|---|---|---|---|
| 1547acce31 | Doubao-Seed-Character | completed | 285 | 95 | hash | 4 | 2 |
| 44e958def4 | deepseek-v4-1-flash-261017 | completed | 285 | 95 | slot | 0 | 0 |
| cd542f9e50 | qwen-plus-character | running | 201 | 95 | slot | 0 | 0 |

Only 2 official-panel full-protocol runs are complete (two models); a third (qwen-plus-character) is still running with 201 of 285 sessions scored. Model-level statistics below therefore compare 2 models (3 with the partial run, marked †). With 2–3 models, rank correlations can only take the values ±1 (2 models) or ±1, ±1/3 (3 models) and carry almost no information about a 10+-model leaderboard.

### A1. Inter-judge agreement (judge_a = Claude Opus 5.5, judge_b = GPT-6.1 Sol; sessions with substituted segments excluded)

| set | level | dimension | n | mean A | mean B | SD A | SD B | Pearson | Spearman | MAD | ICC(2,1) | κ (pass/fail) | % agree |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| complete runs (2) | session | basics | 568 | 88.24 | 97.08 | 19.13 | 8.99 | 0.635 | 0.533 | 8.91 | 0.417 |  |  |
| complete runs (2) | session | logic | 568 | 73.1 | 80.63 | 23.56 | 21.55 | 0.623 | 0.594 | 14.86 | 0.588 |  |  |
| complete runs (2) | session | knowledge | 568 | 99.76 | 99.41 | 1.26 | 4.22 | 0.712 | 0.609 | 0.46 | 0.388 |  |  |
| complete runs (2) | session | diversity | 568 | 77.42 | 77.1 | 8.85 | 10.83 | 0.873 | 0.855 | 4.29 | 0.855 |  |  |
| complete runs (2) | session | content_logic | 568 | 94.48 | 96.51 | 9.41 | 8.84 | 0.724 | 0.608 | 3.62 | 0.705 |  |  |
| complete runs (2) | session | interaction | 568 | 98.86 | 98.34 | 1.85 | 2.23 | 0.799 | 0.639 | 0.85 | 0.761 |  |  |
| complete runs (2) | session | overall | 568 | 89.01 | 91.66 | 5.92 | 4.61 | 0.684 | 0.675 | 3.82 | 0.59 |  |  |
| complete + partial (3)† | session | basics | 769 | 81.69 | 94.33 | 24.0 | 12.82 | 0.658 | 0.608 | 12.8 | 0.45 |  |  |
| complete + partial (3)† | session | logic | 769 | 62.89 | 72.33 | 30.91 | 27.84 | 0.764 | 0.729 | 16.1 | 0.723 |  |  |
| complete + partial (3)† | session | knowledge | 769 | 99.53 | 99.11 | 1.61 | 4.16 | 0.632 | 0.62 | 0.69 | 0.423 |  |  |
| complete + partial (3)† | session | diversity | 769 | 74.78 | 74.5 | 10.6 | 12.42 | 0.896 | 0.883 | 4.44 | 0.885 |  |  |
| complete + partial (3)† | session | content_logic | 769 | 93.98 | 95.92 | 9.75 | 9.17 | 0.741 | 0.62 | 3.8 | 0.724 |  |  |
| complete + partial (3)† | session | interaction | 769 | 98.63 | 97.64 | 1.88 | 2.83 | 0.762 | 0.714 | 1.26 | 0.646 |  |  |
| complete + partial (3)† | session | overall | 769 | 85.64 | 89.11 | 8.71 | 7.0 | 0.834 | 0.806 | 4.5 | 0.743 |  |  |
| complete runs (2) | segment | basics | 2840 | 88.24 | 97.08 | 32.21 | 16.84 | 0.469 | 0.469 | 8.91 | 0.364 | 0.363 | 0.911 |
| complete runs (2) | segment | logic | 2840 | 73.1 | 80.63 | 44.34 | 39.52 | 0.523 | 0.523 | 17.54 | 0.511 | 0.511 | 0.825 |
| complete runs (2) | segment | knowledge | 2840 | 99.76 | 99.41 | 2.1 | 5.41 | 0.471 | 0.506 | 0.51 | 0.317 |  |  |
| complete runs (2) | segment | diversity | 2840 | 77.42 | 77.1 | 13.01 | 15.47 | 0.817 | 0.784 | 6.91 | 0.805 |  |  |
| complete runs (2) | segment | content_logic | 2840 | 94.48 | 96.51 | 12.46 | 11.32 | 0.617 | 0.504 | 4.32 | 0.605 |  |  |
| complete runs (2) | segment | interaction | 2840 | 98.86 | 98.34 | 2.7 | 3.18 | 0.694 | 0.575 | 1.08 | 0.674 |  |  |
| complete runs (2) | segment | overall | 2840 | 89.01 | 91.66 | 10.02 | 7.96 | 0.572 | 0.627 | 5.27 | 0.534 |  |  |
| complete + partial (3)† | segment | basics | 3845 | 81.69 | 94.33 | 38.67 | 23.13 | 0.503 | 0.503 | 12.9 | 0.411 | 0.411 | 0.871 |
| complete + partial (3)† | segment | logic | 3845 | 62.89 | 72.33 | 48.31 | 44.74 | 0.58 | 0.58 | 19.17 | 0.567 | 0.567 | 0.808 |
| complete + partial (3)† | segment | knowledge | 3845 | 99.53 | 99.11 | 2.81 | 5.82 | 0.438 | 0.525 | 0.79 | 0.342 |  |  |
| complete + partial (3)† | segment | diversity | 3845 | 74.78 | 74.5 | 14.53 | 16.86 | 0.842 | 0.815 | 7.04 | 0.832 |  |  |
| complete + partial (3)† | segment | content_logic | 3845 | 93.98 | 95.92 | 13.17 | 11.91 | 0.628 | 0.51 | 4.66 | 0.618 |  |  |
| complete + partial (3)† | segment | interaction | 3845 | 98.63 | 97.64 | 2.83 | 3.88 | 0.661 | 0.599 | 1.49 | 0.604 |  |  |
| complete + partial (3)† | segment | overall | 3845 | 85.64 | 89.11 | 12.19 | 10.15 | 0.668 | 0.695 | 6.12 | 0.627 |  |  |

Per-run rows: out/reliability_a_agreement.csv. Basics and Logic are pass/fail per segment (100 or 0); κ treats them as binary. Knowledge, Content logic and Interaction have little variance (most segments 100), so correlations on them rest on few non-ceiling cases.

### A1b. Model level: overall by each judge alone (wrong-language rule applied to both)

| model | sessions | judge_a only | judge_b only | A − B | official (combined) |
|---|---|---|---|---|---|
| Doubao-Seed-Character | 285 | 87.63 | 90.0 | -2.37 | 88.81 |
| deepseek-v4-1-flash-261017 | 285 | 89.14 | 92.07 | -2.92 | 90.6 |
| qwen-plus-character† | 201 | 76.11 | 81.91 | -5.79 | 79.01 |

Kendall τ between the judge_a-only and judge_b-only rankings: +1.00 (2 complete models), +1.00 (3 models incl. partial†).

### A1c. Dispute rates as recorded (result_json.disputed / disputes / chunkDisputes)

| model | sessions | disputed sessions | session disputes by dimension | segments with both judges | segment disputes | by dimension | thresholds (snapshot) |
|---|---|---|---|---|---|---|---|
| Doubao-Seed-Character | 285 | 96 (33.7%) | logic 51, basics 31, content_logic 27, knowledge 4, diversity 2 | 1419 | 489 (0.34/segment) | logic 244, basics 140, content_logic 92, knowledge 9, diversity 4 | session ≥15, segment ≥30 |
| deepseek-v4-1-flash-261017 | 285 | 77 (27.0%) | logic 53, basics 23, content_logic 9 | 1425 | 402 (0.28/segment) | logic 254, basics 113, content_logic 25, diversity 5, knowledge 5 | session ≥15, segment ≥30 |
| qwen-plus-character | 201 | 120 (59.7%) | logic 73, basics 70, content_logic 16, diversity 3, knowledge 2, interaction 2 | 1005 | 551 (0.55/segment) | basics 243, logic 239, content_logic 50, knowledge 13, diversity 5, interaction 1 | session ≥15, segment ≥30 |

A pass/fail dimension (Basics, Logic) is a session dispute only when the judges split on ≥ 2 segments (engine/results.ts); every split pass/fail segment is a segment dispute (gap 100 ≥ 30).

### A2. Discrimination: overall with 95% bootstrap CIs (B = 2000)

| model | sessions | seeds | overall | CI over sessions | CI over scenarios (cluster by seed) | width ratio cluster/session |
|---|---|---|---|---|---|---|
| Doubao-Seed-Character | 285 | 95 | 88.81 | [87.41, 89.98] | [87.03, 90.24] | 1.25 |
| deepseek-v4-1-flash-261017 | 285 | 95 | 90.60 | [90.00, 91.14] | [89.80, 91.40] | 1.39 |
| qwen-plus-character† | 201 | 95 | 79.01 | [77.96, 80.03] | [77.64, 80.39] | 1.32 |

Pairs (paired bootstrap over shared scenarios, seed-level means; sign-flip p, Monte Carlo 20000):

| pair | shared seeds | mean diff | paired 95% CI | sign-flip p | cluster CIs | separated (paired CI excludes 0) |
|---|---|---|---|---|---|---|
| Doubao-Seed-Character − deepseek-v4-1-flash-261017 | 95 | -1.79 | [-3.83, -0.29] | 0.0331 | overlap | yes |
| Doubao-Seed-Character − qwen-plus-character † | 95 | +9.59 | [+7.18, +11.88] | 0.0000 | separated | yes |
| deepseek-v4-1-flash-261017 − qwen-plus-character † | 95 | +11.39 | [+9.78, +13.02] | 0.0000 | separated | yes |

### A3. Ranking stability

Rank distribution under scenario bootstrap (B = 2000; resampling the shared seeds, seed-level means): 2 complete models on 95 seeds; 3 models† on 95 seeds.

| model | set | P(rank 1) | P(rank 2) | P(rank 3) |
|---|---|---|---|---|
| Doubao-Seed-Character | 2 models | 0.009 | 0.991 |  |
| deepseek-v4-1-flash-261017 | 2 models | 0.991 | 0.009 |  |
| Doubao-Seed-Character | 3 models† | 0.013 | 0.987 | 0.000 |
| deepseek-v4-1-flash-261017 | 3 models† | 0.987 | 0.013 | 0.000 |
| qwen-plus-character | 3 models† | 0.000 | 0.000 | 1.000 |

| model | zh only | en only | judge_a only | judge_b only |
|---|---|---|---|---|
| Doubao-Seed-Character | 90.92 | 86.48 | 87.63 | 90.00 |
| deepseek-v4-1-flash-261017 | 93.04 | 87.89 | 89.14 | 92.07 |
| qwen-plus-character† | 81.41 | 77.83 | 76.11 | 81.91 |

Kendall τ zh-only vs en-only: +1.00 (2 models), +1.00 (3†). judge_a-only vs judge_b-only: +1.00 (2), +1.00 (3†).

Split-half (random halves of the shared scenarios, 2000 repetitions): τ distribution, 2 models: τ=-1.00: 3.2%, τ=+1.00: 96.8%; 3 models†: τ=+0.33: 3.5%, τ=+1.00: 96.5%.

### A4. Generalisation across scenarios

Kendall's W over scenarios (each shared seed ranks the models by its 3-sample mean; tie-corrected): W = 0.025 for the 2 complete models over 95 seeds; W = 0.466 for 3 models† over 95 seeds.
2 models: the overall leader (deepseek-v4-1-flash-261017) is ahead on 55 of 95 seeds, behind on 40, tied on 0. 3 models†: 42 of 95 seeds reproduce the overall order.

Strata (the seed data carry no scenario category field — worker/src/benchmarks/minimax-rp/data/seeds.*.jsonl have id, names, settings, prologue, first user input only — so language and simulator model are the only available strata):

| stratum | Doubao-Seed-Character | deepseek-v4-1-flash-261017 | qwen-plus-character† | order |
|---|---|---|---|---|
| language zh | 90.92 | 93.04 | 81.41 | deepseek-v4-1- > Doubao-Seed-Ch > qwen-plus-char |
| language en | 86.48 | 87.89 | 77.83 | deepseek-v4-1- > Doubao-Seed-Ch > qwen-plus-char |
| simulator claude-sonnet-5.5 | 90.09 | 90.66 | 79.15 | deepseek-v4-1- > Doubao-Seed-Ch > qwen-plus-char |
| simulator gpt-6-luna | 87.63 | 90.55 | 78.86 | deepseek-v4-1- > Doubao-Seed-Ch > qwen-plus-char |
| all | 88.81 | 90.60 | 79.01 | deepseek-v4-1- > Doubao-Seed-Ch > qwen-plus-char |

Simulator strata: 1547acce31 used hash assignment, so its simulator strata are not the same scenario/sample slots as the slot runs.