# v6.0 changes: timeline and effect in the data

All times UTC. Versions: Cloudflare Workers API version list of `rp-bench-api` (62 versions, raw/cf_versions_p*.json); markers: each version's own worker.js fetched with `GET .../workers/workers/rp-bench-api/versions/{id}?include=modules` (scripts/version_markers.py -> out/version_markers.csv). Deployment records (`wrangler deployments list`, raw/deployments_list.txt; API raw/cf_deployments.json) exist only for the 10 most recent deployments (versions 53–62); for earlier versions the version upload time is used (every version shows source `wrangler`, annotation `version_upload`; the deployments themselves are no longer listed).

## 1. Timeline: first Worker version containing each change

| change | claimed in | first version | version created (UTC) | previous version (UTC) | deployment record | can it change a completed run's scores? |
|---|---|---|---|---|---|---|
| v6.0 protocol (realistic simulator, sampled lengths, fixed personas, target max_tokens 8192, finish reasons) | protocol comment v6.0 | 4 | 2026-10-24 19:59:48 | 2026-10-24 18:51:11 | not retained (only last 10 deployments listed) | No (generation-time; new runs) |
| Single-host leaderboard eligibility rule | report: "later additions without score changes" | 11 | 2026-10-25 13:19:33 | 2026-10-25 09:52:43 | not retained (only last 10 deployments listed) | Eligibility only (recomputed with results) |
| Experience-quality sub-score | report: "later additions" | 12 | 2026-10-25 14:28:53 | 2026-10-25 13:19:33 | not retained (only last 10 deployments listed) | No (not in overall) |
| Simulator turn blocked by moderation written by the other simulator model | change log 26 Oct | 24 | 2026-10-26 11:41:25 | 2026-10-26 10:00:36 | not retained (only last 10 deployments listed) | No (generation-time; affects sessions generated/resumed after it) |
| Wrong-language session rule (score 0) + hourly re-application to all stored results | change log 26 Oct; protocol: sessionRuleVersion | 26 | 2026-10-26 13:19:51 | 2026-10-26 12:56:45 | not retained (only last 10 deployments listed) | YES: re-scores stored sessions of minimax/platform/calibration runs and recomputes run results |
| Simulator model fixed by scenario and sample (assignment "slot") | change log 26 Oct | 26 | 2026-10-26 13:19:51 | 2026-10-26 12:56:45 | not retained (only last 10 deployments listed) | No (snapshot-level; runs created after it) |
| Judge output limits 32000 -> 16000 (Opus) / 8000 (Sol) | change log 26 Oct | 27 | 2026-10-26 13:36:33 | 2026-10-26 13:19:51 | not retained (only last 10 deployments listed) | No (snapshot-level; runs created after it) |
| Re-judge a truncated reply with twice the limit | change log 26 Oct | 27 | 2026-10-26 13:36:33 | 2026-10-26 13:19:51 | not retained (only last 10 deployments listed) | No (judging-time) |
| Batches submitted only as far as the balance covers | change log 26 Oct | 27 | 2026-10-26 13:36:33 | 2026-10-26 13:19:51 | not retained (only last 10 deployments listed) | No (scheduling only) |
| Expired batch: segments judged directly (sync API) | change log 27 Oct | 51 | 2026-10-27 11:30:25 | 2026-10-27 11:28:08 | not retained (only last 10 deployments listed) | No for completed runs; changes judging mode of runs in flight |
| Moderation-refused segment scored by the other official judge (substituteBlocked) | change log 27 Oct | 52 | 2026-10-27 11:48:37 | 2026-10-27 11:30:25 | not retained (only last 10 deployments listed) | No for completed sessions (result computed at session completion); lets previously stuck sessions complete |
| Direct judging: a second unusable output is final | not in change log | 57 | 2026-10-27 13:36:13 | 2026-10-27 13:06:43 | 2026-10-27T13:36:15.349557Z | No (judging-time) |

Protocol version strings carry local (UTC+8) dates: "v6.0 · 2026-10-25" first appears in version 4 created 2026-10-24 19:59 UTC (= 25 Oct 03:59 UTC+8); the "26 Oct" changes are in versions created 26 Oct 11:41–13:36 UTC; the "27 Oct" changes in versions created 27 Oct 11:30–11:48 UTC. No version string changed for any of them: every bundle from version 4 to 62 carries "v6.0 · 2026-10-25" (out/version_markers.csv, column protocol_versions_in_bundle).

Wrong-language rule pass: settings `session_rule:minimax-rp` = "wrong-language 0.5 · 2026-10-26" written 2026-10-26 16:19:18 UTC (raw/settings_rules.json), i.e. the hourly cron finished re-scoring all stored results 2026-10-26 16:19:18 — after version 26 (2026-10-26 13:19:51).

## 2. Runs created before / after each 26–27 Oct change (platform + calibration runs; imported runs are not created by the engine)

Only runs from 25 Oct on are shown in full in out/runs.md; every run created before 2026-10-26 13:19:51 UTC is "before" the slot/rule/limit changes. Runs that matter:

| run | name | mode | protocol | created | finished/status | slot assignment (v26) | judge limits 16k/8k (v27) | expired→direct (v51) | substitution (v52) |
|---|---|---|---|---|---|---|---|---|---|
| 52e61f3834 | qwen-plus-character | quick | v6.0 · 2026-10-25 | 2026-10-25 14:35:55 | 2026-10-25 14:39:26 | before | before | before (finished) | before (finished) |
| 7c79ab9425 | qwen-plus-character | standard | v6.0 · 2026-10-25 | 2026-10-25 14:43:15 | 2026-10-25 15:09:22 | before | before | before (finished) | before (finished) |
| 6d6b0367ce | doubao-seed-character-260628 | quick | v6.0 · 2026-10-25 | 2026-10-25 14:47:53 | 2026-10-25 14:50:41 | before | before | before (finished) | before (finished) |
| 06c68fc86a | ~openai/gpt-luna-latest | quick | v6.0 · 2026-10-25 | 2026-10-25 14:57:17 | 2026-10-25 15:00:06 | before | before | before (finished) | before (finished) |
| 1547acce31 | Doubao-Seed-Character | full | v6.0 · 2026-10-25 | 2026-10-26 09:20:32 | 2026-10-27 13:44:38 | before | before | after | spans |
| cd542f9e50 | qwen-plus-character | full | v6.0 · 2026-10-25 | 2026-10-27 13:37:38 | running | after | after | after | after |
| 44e958def4 | deepseek-v4-1-flash-261017 | full | v6.0 · 2026-10-25 | 2026-10-27 13:38:04 | 2026-10-27 15:22:41 | after | after | after | after |

1547acce31 (Doubao-Seed-Character) was created 2026-10-26 09:20 UTC, before every 26/27 Oct change, and finished 2026-10-27 13:44 UTC, after all of them: its generation used hash assignment and 32000-token judge limits (snapshot), but its judging was completed under the 27 Oct code (direct judging, substitution). cd542f9e50 and 44e958def4 were created after all changes.

## 3. Wrong-language rule: penalised sessions and score change per run

| run | name | penalised / scored | overall all: without → with rule (Δ) | zh | en | sessions |
|---|---|---|---|---|---|---|
| mm-doubao-1.5-pro | doubao-1.5-pro | 4/285 | 87.11 → 85.9 (-1.21) | 85.06 → 85.06 (+0.00) | 89.39 → 86.84 (-2.55) | en_dialogue_022#2 (32/50 条回复未使用英文, judges' overall 89.5); en_dialogue_041#1 (50/50 条回复未使用英文, judges' overall 89.8); en_dialogue_041#2 (50/50 条回复未使用英文, judges' overall 85.6); en_dialogue_041#3 (50/50 条回复未使用英文, judges' overall 78.9) |
| 6d6b0367ce4a4edf | doubao-seed-character-260628 | 1/10 | 94.08 → 84.33 (-9.75) | 92.58 → 92.58 (+0.00) | 95.58 → 76.08 (-19.50) | en_dialogue_031#1 (19/20 条回复未使用英文, judges' overall 97.5) |
| 1547acce317f4fc3 | Doubao-Seed-Character | 4/285 | 90.02 → 88.81 (-1.21) | 90.92 → 90.92 (+0.00) | 89.03 → 86.48 (-2.56) | en_dialogue_024#1 (50/50 条回复未使用英文, judges' overall 92.3); en_dialogue_023#2 (50/50 条回复未使用英文, judges' overall 94.1); en_dialogue_031#3 (32/50 条回复未使用英文, judges' overall 71.2); en_dialogue_023#1 (50/50 条回复未使用英文, judges' overall 87.8) |

9 sessions in 3 runs carry a penalty; all other stored sessions (official, screening, calibration, imported) have none (out/v6_penalties.csv).

## 4. Simulator assignment (hash vs slot)

| run | name | mode | created | assignment (snapshot) | sessions per language × simulator (majority model of user turns) | sessions matching the assignment rule | sessions with >1 simulator model |
|---|---|---|---|---|---|---|---|
| c7dd699c40 | MiniMax-M2-her | quick | 10-24 15:10 | hash | en claude-opus-5.5: 3; en gpt-6.1-sol: 2; zh claude-opus-5.5: 3; zh gpt-6.1-sol: 2 | 10/10 | 0 |
| c744e020df | minimax/minimax-m2-her | quick | 10-24 15:58 | hash | en claude-opus-5.5: 3; en gpt-6.1-sol: 2; zh claude-opus-5.5: 3; zh gpt-6.1-sol: 2 | 10/10 | 0 |
| 51a38906af | DeepSeek | quick | 10-24 16:06 | hash | en claude-opus-5.5: 3; en gpt-6.1-sol: 2; zh claude-opus-5.5: 3; zh gpt-6.1-sol: 2 | 10/10 | 0 |
| 87510dfe3a | deepseek-v4-1-flash-261017 | quick | 10-24 16:19 | hash | en claude-opus-5.5: 2; en gpt-6.1-sol: 3; zh claude-opus-5.5: 2; zh gpt-6.1-sol: 3 | 10/10 | 0 |
| f909cbadda | doubao-seed-character-260628 | quick | 10-24 16:26 | hash | en claude-opus-5.5: 4; en gpt-6.1-sol: 1; zh claude-opus-5.5: 2; zh gpt-6.1-sol: 3 | 10/10 | 0 |
| 88696d6adf | deepseek-v4.1-flash-uncensor | quick | 10-24 16:35 | hash | en claude-opus-5.5: 3; en gpt-6.1-sol: 2; zh claude-opus-5.5: 2; zh gpt-6.1-sol: 3 | 10/10 | 0 |
| 3feb0cae4b | qwen-plus-character | quick | 10-24 16:38 | hash | en claude-opus-5.5: 2; en gpt-6.1-sol: 3; zh claude-opus-5.5: 4; zh gpt-6.1-sol: 1 | 10/10 | 0 |
| 8c159707c1 | MiniMax-M2-her（v3.0 试测） | quick | 10-24 16:48 | hash | en claude-sonnet-5.5: 2; en gpt-6-luna: 3; zh claude-sonnet-5.5: 5 | 10/10 | 0 |
| 371cf5d44d | MiniMax-M2-her（v3.2 试测） | quick | 10-24 17:00 | hash | en claude-sonnet-5.5: 1; en gpt-6-luna: 4; zh claude-sonnet-5.5: 1; zh gpt-6-luna: 4 | 10/10 | 0 |
| ebebdd6609 | MiniMax-M2-her（v5.0 快筛） | quick | 10-24 17:52 | hash | en claude-sonnet-5.5: 4; en gpt-6-luna: 1; zh claude-sonnet-5.5: 2; zh gpt-6-luna: 3 | 10/10 | 0 |
| 6ca7b7f2b7 | qwen-plus-character | quick | 10-24 17:56 | hash | en claude-sonnet-5.5: 4; en gpt-6-luna: 1; zh claude-sonnet-5.5: 4; zh gpt-6-luna: 1 | 10/10 | 0 |
| 808242363e | doubao-seed-character-260628 | quick | 10-24 18:14 | hash | en claude-sonnet-5.5: 3; en gpt-6-luna: 2; zh claude-sonnet-5.5: 4; zh gpt-6-luna: 1 | 10/10 | 0 |
| 2454e023b5 | openai/gpt-6-sol | quick | 10-24 18:17 | hash | en claude-sonnet-5.5: 2; en gpt-6-luna: 3; zh claude-sonnet-5.5: 3; zh gpt-6-luna: 2 | 10/10 | 0 |
| 452c5c36e4 | anthropic/claude-opus-5.5 | quick | 10-24 18:24 | hash | en claude-sonnet-5.5: 3; en gpt-6-luna: 2; zh claude-sonnet-5.5: 1; zh gpt-6-luna: 4 | 10/10 | 0 |
| 52e61f3834 | qwen-plus-character | quick | 10-25 14:35 | hash | en claude-sonnet-5.5: 3; en gpt-6-luna: 2; zh claude-sonnet-5.5: 3; zh gpt-6-luna: 2 | 10/10 | 0 |
| 7c79ab9425 | qwen-plus-character | standard | 10-25 14:43 | hash | en claude-sonnet-5.5: 9; en gpt-6-luna: 6; zh claude-sonnet-5.5: 7; zh gpt-6-luna: 8 | 30/30 | 0 |
| 6d6b0367ce | doubao-seed-character-260628 | quick | 10-25 14:47 | hash | en claude-sonnet-5.5: 3; en gpt-6-luna: 2; zh claude-sonnet-5.5: 1; zh gpt-6-luna: 4 | 10/10 | 0 |
| 06c68fc86a | ~openai/gpt-luna-latest | quick | 10-25 14:57 | hash | en claude-sonnet-5.5: 3; en gpt-6-luna: 2; zh claude-sonnet-5.5: 3; zh gpt-6-luna: 2 | 10/10 | 0 |
| 1547acce31 | Doubao-Seed-Character | full | 10-26 09:20 | hash | en claude-sonnet-5.5: 72; en gpt-6-luna: 63; zh claude-sonnet-5.5: 65; zh gpt-6-luna: 85 | 285/285 | 1 |
| cd542f9e50 | qwen-plus-character | full | 10-27 13:37 | slot | en claude-sonnet-5.5: 68; en gpt-6-luna: 67; zh claude-sonnet-5.5: 35; zh gpt-6-luna: 35 | 205/205 | 0 |
| 44e958def4 | deepseek-v4-1-flash-261017 | full | 10-27 13:38 | slot | en claude-sonnet-5.5: 68; en gpt-6-luna: 67; zh claude-sonnet-5.5: 75; zh gpt-6-luna: 75 | 285/285 | 0 |

Assignment rule check: hash = sha1("usersim:"+session id)[0] mod 2 (engine/runner.ts simModelFor), slot = (position of seed among its language's seeds in id order × samples + sample) mod 2.

## 5. Judge substitution (substituteBlocked)

| run | session | segments substituted | detail | session overall (stored) | judge_b alone | judge_a incl. substituted segments | note |
|---|---|---|---|---|---|---|---|
| 1547acce31 | en_dialogue_017#1 | 4 | judge_a seg2←judge_b, judge_a seg3←judge_b, judge_a seg4←judge_b, judge_a seg5←judge_b | 87.19 | 87.00 | 87.38 |  |
| 1547acce31 | en_dialogue_017#2 | 2 | judge_a seg1←judge_b, judge_a seg2←judge_b | 81.46 | 82.38 | 80.54 |  |

Judgements recorded as failed with a moderation error: 1547acce31 judge_a: 4.

Effect on 1547acce31: overall 88.81 with the 2 substituted sessions, 88.84 without them (-0.03). Without substitution these sessions would have stayed incomplete and the run could not have completed (engine: a session completes only when every segment is settled).

## 6. Expired batches and direct judging

| run | batch | judge | status (DB) | items | provider status | error | cost $ | reserved $ | created | updated |
|---|---|---|---|---|---|---|---|---|---|---|
| 026e02768d | batch-1792864368-7zyVUGrzQB5 | judge_b | done | 10 |  |  | 0.18 |  | 2026-10-24 17:52:49 | 2026-10-24 18:25:00 |
| 026e02768d | batch-1792864605-JN9Xq54ey53 | judge_b | done | 10 |  |  | 0.15 |  | 2026-10-24 17:56:47 | 2026-10-24 18:44:32 |
| 026e02768d | batch-1792864605-3LfkvUDGuIN | judge_a | submitted | 10 |  |  | 0.00 |  | 2026-10-24 17:56:47 | 2026-10-24 17:56:47 |
| 1547acce31 | batch-1793012897-Svln97rtjW7 | judge_a | done | 200 | completed 200/200 |  | 7.68 |  | 2026-10-26 11:08:20 | 2026-10-27 08:15:49 |
| 1547acce31 | batch-1793012902-NgFiEMV9n1Z | judge_a | done | 200 | completed 200/200 |  | 8.07 |  | 2026-10-26 11:08:25 | 2026-10-27 08:16:00 |
| 1547acce31 | batch-1793012906-l1djq9TRYKY | judge_a | done | 200 | completed 200/200 |  | 7.94 |  | 2026-10-26 11:08:29 | 2026-10-26 14:58:51 |
| 1547acce31 | batch-1793012909-nsM5e85n6kC | judge_a | done | 200 | completed 200/200 |  | 7.97 |  | 2026-10-26 11:08:32 | 2026-10-26 14:59:00 |
| 1547acce31 | batch-1793012913-gCBywx7jouz | judge_a | done | 200 | completed 200/200 |  | 8.39 |  | 2026-10-26 11:08:36 | 2026-10-26 14:53:22 |
| 1547acce31 | batch-1793012916-UE15GSz8xQD | judge_a | done | 200 | completed 200/200 |  | 7.60 |  | 2026-10-26 11:08:39 | 2026-10-27 08:16:09 |
| 1547acce31 | batch-1793012920-0nihSIIu2fb | judge_a | done | 200 | completed 200/200 |  | 8.86 |  | 2026-10-26 11:08:42 | 2026-10-27 08:16:18 |
| 1547acce31 | batch-1793012922-tGtppOqm461 | judge_a | done | 20 | completed 20/20 |  | 0.95 |  | 2026-10-26 11:08:44 | 2026-10-26 14:53:26 |
| 1547acce31 | batch-1793012925-7iQQltQ807z | judge_b | done | 200 | completed 200/200 |  | 2.51 |  | 2026-10-26 11:08:48 | 2026-10-27 08:16:28 |
| 1547acce31 | batch-1793012928-Y1kWDM59wqQ | judge_b | done | 200 | expired 198/200 | The provider ended this batch as "expire | 0.00 |  | 2026-10-26 11:08:51 | 2026-10-27 11:31:25 |
| 1547acce31 | batch-1793012931-NZ21oAIz84z | judge_b | done | 200 | expired 199/200 | The provider ended this batch as "expire | 0.00 |  | 2026-10-26 11:08:55 | 2026-10-27 11:24:59 |
| 1547acce31 | batch-1793012935-8mzb217MHTl | judge_b | done | 200 | expired 198/200 | The provider ended this batch as "expire | 0.00 |  | 2026-10-26 11:08:58 | 2026-10-27 11:25:14 |
| 1547acce31 | batch-1793012939-0YfBzaLbbdP | judge_b | done | 200 | expired 197/200 | The provider ended this batch as "expire | 0.00 |  | 2026-10-26 11:09:02 | 2026-10-27 11:25:31 |
| 1547acce31 | batch-1793012942-YmLSQFLSIZ0 | judge_b | done | 200 | completed 200/200 |  | 2.46 |  | 2026-10-26 11:09:05 | 2026-10-27 08:16:40 |
| 1547acce31 | batch-1793012946-aRW9X7ybWw2 | judge_b | done | 200 | completed 200/200 |  | 2.41 |  | 2026-10-26 11:09:08 | 2026-10-27 08:16:49 |
| 1547acce31 | batch-1793012949-bbEqZcPE98d | judge_b | done | 20 | completed 20/20 |  | 0.25 |  | 2026-10-26 11:09:11 | 2026-10-27 08:16:51 |
| 1547acce31 | batch-1793015184-ZS5veOyYFLB | judge_a | done | 5 | completed 5/5 |  | 0.25 |  | 2026-10-26 11:46:25 | 2026-10-27 08:16:53 |
| 1547acce31 | batch-1793015185-EtvXHwWXJCJ | judge_b | done | 5 | completed 5/5 |  | 0.05 |  | 2026-10-26 11:46:27 | 2026-10-27 08:16:55 |
| 1547acce31 | batch-1793026414-YVCAu2GjB9N | judge_a | done | 4 | completed 4/4 |  | 0.11 | 1.34 | 2026-10-26 14:53:36 | 2026-10-27 08:16:57 |
| 1547acce31 | batch-1793089023-zl3G30HNuQa | judge_a | done | 9 | completed 9/9 |  | 0.19 | 3.05 | 2026-10-27 08:17:05 | 2026-10-27 08:32:50 |
| 1547acce31 | batch-1793089997-llwiZi542SL | judge_a | submitted | 5 | in_progress 0/5 |  | 0.00 | 1.71 | 2026-10-27 08:33:19 | 2026-10-27 08:33:19 |
| 1547acce31 | batch-1793100343-sFCQHqXUA2m | judge_b | submitted | 200 | in_progress 0/200 |  | 0.00 | 33.18 | 2026-10-27 11:25:47 | 2026-10-27 11:25:47 |
| 1547acce31 | batch-1793100347-nFLpsRX7SP0 | judge_b | submitted | 200 | in_progress 0/200 |  | 0.00 | 33.14 | 2026-10-27 11:25:50 | 2026-10-27 11:25:50 |
| 1547acce31 | batch-1793100351-9K5sKkg5cqk | judge_b | submitted | 200 | in_progress 0/200 |  | 0.00 | 33.45 | 2026-10-27 11:25:54 | 2026-10-27 11:25:54 |

1547acce31: 4 judge_b (GPT-6.1 Sol) batches expired after ~24 h (800 items, provider status 'expired 197–199/200', no results returned; first expiry logged 2026-10-27 11:24:47). Direct judging was then switched on for the whole run by an administrator at 2026-10-27 11:30:35 UTC (config.directJudging.at; run log '管理员将剩余未出分的分段改为直接调用裁判评分'), 10 s after version 51 (automatic expired-batch path) was created. Logs record 12 direct-judging rounds covering 810 segment requests; 0 rounds are labelled '所在批处理已过期' (the automatic expired-batch path), the rest '本评测已改为直接评分' (the admin switch).
Scored judgements stored before the switch: {'judge_a': 1418, 'judge_b': 625}; after: {'judge_b': 800, 'judge_a': 1} (judgments.created_at vs directJudging.at). Non-done judgements: {('judge_a', 'empty completion (finish_reason=content_filter)'): 4}.
5 batches are still 'submitted' in judge_batches for runs that are completed or paused (026e02768d judge_a 10 items, 1547acce31 judge_a 5 items, 1547acce31 judge_b 200 items, 1547acce31 judge_b 200 items, 1547acce31 judge_b 200 items); reserved total $101.48 (where recorded). Their late results would be ignored for segments already judged (pollBatches skips them).
44e958def4 and cd542f9e50 use config.fastJudging (official judges called synchronously, batch=false): no batches at all. So of the two completed official runs, one was judged partly by batch and partly directly (1547acce31), the other entirely synchronously (44e958def4).

## 7. Judge output limits and truncation

| run | judge | maxTokens (snapshot) | done judgements | median completion tokens | max | > 8000 | > 16000 | stored with truncation error |
|---|---|---|---|---|---|---|---|---|
| 1547acce31 | judge_a | 32000 | 1419 | 2782 | 7764 | 0 | 0 | 0 |
| 1547acce31 | judge_b | 32000 | 1425 | 1643 | 3774 | 0 | 0 | 0 |
| 44e958def4 | judge_a | 16000 | 1425 | 2494 | 6676 | 0 | 0 | 0 |
| 44e958def4 | judge_b | 8000 | 1425 | 1321 | 4737 | 0 | 0 | 0 |
| cd542f9e50 | judge_a | 16000 | 1005 | 4681 | 13290 | 21 | 0 | 0 |
| cd542f9e50 | judge_b | 8000 | 1005 | 2237 | 5729 | 0 | 0 | 0 |

completion_tokens of a judgement include reasoning tokens as reported by the provider.

## 8. Simulator turn blocked by moderation

Sessions whose user turns come from more than one simulator model: 1 1547acce31 38281dfd94ce4f8e: claude-sonnet-5.5×48, gpt-6-luna×2.
Log lines "本轮改由另一个用户模拟器": 2 (1547acce31 2026-10-26 11:42:54; 1547acce31 2026-10-26 11:43:39).