command: /python -I tools/experiments/coser_table_checks.py --json /results/table_checks.json started: 2026-11-01T14:18:19+00:00 ended: 2026-11-01T14:18:19+00:00 exit status: 0 INFO VERSIONS 11 of 18 source files identical in v1, v2, v3; differing: ['Sections/4_Methodology.tex', 'Sections/5_Experiments.tex', 'Sections/Appendix.tex', 'Tables/exp_diff_judges.tex', 'Tables/human_eval.tex', 'Tables/judge_alignment.tex', 'Tables/prompts_eval_2.tex'] PASS T01 main table: 21 rows, identical in v1, v2, v3 PASS T02 worker and server published.ts equal the LaTeX table: 294 means and 294 ± values, 0 differ PASS T03 published.ts records the SHA-256 of Tables/main_exp.tex (99c3cfda63aa…) and the two copies are the same file INFO AVG Average vs the mean of the four printed dimensions: largest gap 0.0050 (Gemini Pro); within 0.005 (rounding of one value) 17/21, within 0.01 21/21 INFO MARKS bold (best overall) and underlined (best open) marks against the values: 0 mismatches INFO SPREAD printed ± on the 21 rows: sc 0.13-2.46 (median 1.12), an 0.16-2.35 (median 1.09), cf 0.13-2.95 (median 1.43), sq 0.14-1.75 (median 0.88), avg 0.22-1.35 (median 0.63), bleu 0.00-0.33 (median 0.10), rouge 0.02-0.56 (median 0.09); the paper states 'averaged across three runs' and does not define ± INFO SPREAD2 rows whose ± on Average exceeds the ± of all four dimensions: 0 INFO IDOOD table exp_idood: 21 rows; mean of in-domain and out-of-domain against the main table (equal sizes, so it must equal it if it is the same runs): Average differs by up to 0.450, mean signed -0.016; any dimension up to 0.710 (GPT-4o sq); cells within 0.01: 95/105; beyond: [('GPT-3.5', 'sc', 0.415), ('GPT-3.5', 'an', -0.085), ('GPT-3.5', 'cf', 0.195), ('GPT-3.5', 'sq', -0.13), ('GPT-3.5', 'avg', 0.1), ('GPT-4o', 'sc', -0.505), ('GPT-4o', 'an', -0.135), ('GPT-4o', 'cf', -0.45), ('GPT-4o', 'sq', -0.71), ('GPT-4o', 'avg', -0.45)] INFO JUDGES table exp_diff_judges: the GPT-4o column equals the in-domain Average of exp_idood for 7/7 models, the full-test Average for 0/7; the caption does not say so INFO ABL ablation table: Standard equals the main table's Average for 6/6 models; removing motivations raises the score for ['CoSER-8B'] although the text says improvements are 'consistent across all models' INFO CLAIMS Average above 57%: ['Claude-3.5-Sonnet', 'CoSER-70B', 'Doubao-pro', 'GPT-4o', 'GPT-4o Mini', 'Gemini Pro', 'Qwen-2-72B', 'Step-2'] (the text lists GPT-4o, Gemini Pro, Claude-3.5-Sonnet, Doubao-pro, Step-2, Qwen-2-72B, CoSER-70B); GPT-4o Mini at 57.93 also qualifies; Claim '58% above the second-best on BLEU': CoSER-70B 10.1 / best other Doubao-pro 6.38 = 0.583; Qwen-2 72B-7B gap 11.53 INFO FORMULA initial score in Methodology v3: True; in Appendix v3 (length correction): False (still 100 − Σv); regression quoted: True; in v1 Methodology: False INFO EX-100-5S+1.5R released evaluation of 3 scenes × 4 dimensions: the stored score equals clip(100-5S+1.5R) for 12/12 INFO EX-100-S+1.5R released evaluation of 3 scenes × 4 dimensions: the stored score equals clip(100-S+1.5R) for 0/12 INFO EX-100-5S released evaluation of 3 scenes × 4 dimensions: the stored score equals clip(100-5S) for 0/12 INFO TEST test set: SHA-256 eb175779d1d5adf99c893ff1f82de9b18ada912d3f385b3f5419195afdfb374a; 200 scenes, 100 id, 100 ood; 101 books (69 id, 32 ood); speaking characters per scene 1-5 (mean 2.40, 479 in all), 451 with a motivation; original lines per scene 9-32 (median 12.0), 7 scenes longer than the 20-line limit; major characters per scene 1-5; Environment listed as a speaker in 130; speakers without any profile text 13 INFO EXP-FILES released result files under exp/: ['exp/evaluation/test_set_gpt-4o.json', 'exp/simulation/test_set_gpt-4o-mini_rag=expr3_conv1.json', 'exp/simulation/test_set_gpt-4o.json', 'exp/simulation/test_set_gpt-4o_rag=expr3_conv1.json'] no failures