CoSER: An Independent Reproduction Report

1 The original benchmark

1.1 Sources

The evaluation code changed in steps that matter here. On 2025-02-18 (0e11992, “typo fixes”) a ceiling of 100 was added to the dimension score. On 2025-02-24 (297ffff) the script became main.py: the next-speaker model became a parameter with default GPT-4o-mini, and motivations were read from a field the test set does not have. On 2025-12-25 (958f962, whose message reads “add support for thinking models ( especially for nsp)”, then d8489fa and 4d6ed0f) the next-speaker prompt gained an Output Format section, its reply gained a new parser, and the speaker prefix was removed from the record. The main results table has not changed in any arXiv version, so it predates all of these.

1.2 Data

The dataset covers 17,966 characters from 771 books. CoSER Test holds 200 English scenes held out from the final 10% of each book, 100 from books used to train the CoSER models (in-domain) and 100 from books not used (out-of-domain). We count, from the released file: 101 books (69 and 32); 1 to 5 speaking characters per scene, 479 in all, of whom 451 have a motivation; 9 to 32 original lines per scene (median 12), 7 scenes exceeding the simulation's limit of 20 lines. Environment is listed as a speaker in 130 scenes and appended in the other 70. 13 speaking characters have no profile text. The paper says scenes were drawn by a weighted sampling from the top half of conversations ranked by the characters' prominence. Neither that sampling nor the “final 10%” can be verified from what was released: the held-out plot file covers 15 books, one of which appears in the test set (4 scenes).

1.3 Given-circumstance acting

1. Simulation. The model under test plays every character, with the scenario, the profiles of the characters involved and the character's motivation, and is instructed to write speech, action and inner thought. The paper's main results use GPT-4o as the environment and CoSER-70B as the next-speaker predictor (NSP), with no original lines given in advance (k = 0). The simulation stops at an end signal from the NSP or after 20 lines. Motivations and inner thoughts are hidden from the other characters.
2. Judging. GPT-4o receives the plot summary, the scenario, the profiles and the original conversation, and reads the simulation without inner thoughts. In a separate request per dimension it lists flaws by detailed rubrics, each with a severity from 1 to 5.
3. Score. A dimension scores 100 − 5 × the sum of severities + λ|M̄|, with λ = 1.5 as a length correction [4]. The overall score is the mean of storyline consistency, anthropomorphism, character fidelity and storyline quality. Scores are averaged across three runs. BLEU and ROUGE-L against the original dialogue are reported for analysis and are not in the overall score.

1.4 What the sources leave open

1. Temperature, output limit, seeds, retries and failure handling: none is in the paper; a search of its source for these terms finds nothing about evaluation.
2. The 60-word instruction, the start of the conversation and the per-model handling of prompts and replies are in the code, not in the paper's prompt tables (Appendix A).
3. The meaning of the ± in the tables.
4. The judge's version: the footnote with version 240806 belongs to the evaluated GPT-4o; the README states that reproduction must use gpt-4o-24-08-06 as the judge.
5. What |M̄| counts. The formula reads as the whole simulation; the released examples show that only lines not spoken by the environment count (Section 2.2).

1.5 Related work

Most role-play benchmarks score a single reply given a fixed context, with a trained reward model, a fine-tuned judge or a prompted LLM. CoSER differs in two ways that shape its reproduction: the generation is a multi-agent simulation, so the result depends on the environment and next-speaker models as well as on the model under test; and the score counts flaws against a reference with a prompted LLM, so it depends on one API model's behaviour. The paper's own appendix shows how much that matters: GPT-3.5 scores 52.8 with GPT-4o (a column that holds the in-domain subset, Table 2), 35.9 with DeepSeek-R1 and 40.5 with DeepSeek-V3. Later papers with other configurations report very different values for the same models (Section 9).

2 Published results and their provenance

2.1 The published table

The paper's main table holds 21 models (Appendix B). The platform read it with a script (tools/experiments/coser_paper.py) that records the source's SHA-256 and writes the same data module for the server and the Worker; the audit bundle re-reads the table and confirms the 294 printed values (means and ± of 7 columns for 21 models, in each of the two modules) are unchanged. Tables/main_exp.tex is byte-identical in v1, v2 and v3. Every printed Average is within 0.005 of the mean of its four dimensions. Cells print a mean with a ±; the paper says results are “averaged across three runs” and does not define the ±. Model versions are as in the paper's footnotes (GPT-4o 240806, Claude-3.5-Sonnet 20240620, Doubao-pro 241215). CoSER-8B and CoSER-70B are the authors' models trained on this dataset.

19 rows of the main table equal the mean of the in-domain and out-of-domain table, as eval_reproduce.py computes it; the rows of GPT-4o and GPT-3.5 do not (Table 2). That is why this report calls that script the most likely source, not the source, of the numbers.

2.2 Inconsistencies in the sources

The reviews (Section 6) and a mechanical check of the paper's own numbers (coser_table_checks.py, in the audit bundle) found the following. Line numbers refer to arXiv v3.

3 Sources of the protocol

3.1 Every setting and its source

Table 3 names the source of each setting of protocol 1.1. The full matrix with file and line references is published as docs/复现规范/CoSER-核查矩阵.md, in the audit bundle.

3.2 Deviations from the paper

1. Next-speaker model. The paper uses CoSER-70B; the platform uses GPT-4o-mini (2024-07-18), the default of the official code since February 2025. The project owner chose this when protocol 1.0 was set up in October 2026, to avoid deploying a 141 GB model for one role; the README recommends CoSER-70B or another self-deployed model for better cost-efficiency. The weights are public [3], so this is a choice, not a necessity. The NSP decides who speaks and when the scene ends, and each extra line not spoken by the environment adds 1.5 to every dimension, so this deviation can move every dimension. Its size is unknown, and because CoSER-70B was not deployed the anchor cannot isolate it (Section 7.3).
2. Random choices. Drawn from seeded streams instead of the official stream, which depends on earlier scenes and on Python's per-process hash seed. Every model under test sees the same choices. Single scenes cannot be aligned with an official run; averages should not be affected.
3. Thinking. A leading think block is removed before anything else; the official code keeps it in the record and the judge's input. Only models that write their reasoning into the reply are affected, and the direction favours them.
4. Content refusals. Recorded as empty lines rather than retried forever.
5. ROUGE-L. Not reported.

In protocol 1.1 the prompts, the speaking rules and the record are those of the official main.py at d7cee90 (2025-02-25), whose default NSP is GPT-4o-mini, with the prompts of the paper's tables. The differences from that commit are the motivations, the absent ceiling, the random stream, the failure paths, the scope of the name-prefix rule (LLaMA-3.1-Instruct 8B and 70B, as in the paper-era code, where that commit applies it to all names starting with llama), the removal of a leading think block and of surrounding whitespace, the tag rules of remove_inner_thoughts, and the retry count and judge limit of the current code. No official commit equals protocol 1.1, and no published number was produced by it.

3.3 Where the current official code departs from the paper

Running 624efc5 as released would give different scores from the paper for reasons unrelated to the model under test: characters get no motivation; the simulation the judge reads has no speaker names, while the original conversation it compares with has them; the next-speaker prompt and parser differ; dimension scores have a ceiling; and the second and third runs repeat the first. The platform follows the paper, or the code of its time, on each. The authors' own recommendation for reproducing the paper is to use gpt-4o-24-08-06 as the judge; the README says nothing about the other differences.

4 The protocol on this platform

Protocol coser-gca-1.1 · 2026-11-01, as it stood when the leaderboard was withdrawn. It is no longer offered in the evaluation lab; the code stays in the repository. Every run stores this version with the prompts and settings it used.

4.1 Simulation

A session is one simulation of one scene. The engine asks the protocol module who acts next, sends the exact messages that agent would hold, stores the reply and moves on; a session can therefore stop and resume at any step. The model under test plays every character. GPT-4o (2024-08-06) plays the environment and GPT-4o-mini (2024-07-18) predicts the next speaker, each pinned to OpenAI with no fallback. Requests start at temperature 0, with a limit of 512 output tokens. The first speaker is the scene's first listed speaker. After every line the NSP names the next speaker; an unknown or repeated name is replaced by a seeded draw among the main characters and the environment, an end signal counts from the sixth line on, and after 20 lines the simulation ends.

Per-model handling follows the official code through a fixed table of model families (gca/family.ts): Claude and o1 receive the system prompt as a user message; only LLaMA-3.1-Instruct 8B and 70B and Step models get the “start your response with” line and have their reply cut to their own lines; models other than GPT and Claude have repeated text removed from their replies; CoSER models get no output examples. New model families fall under “other”.

4.2 Judging

Each session is judged in four independent requests to GPT-4o (2024-08-06), one per dimension, with the template and rubrics of the paper in the official replacement order. The system message holds the materials; the user message holds the simulation with inner thoughts removed and speakers named. A reply is read as the official code reads it: a JSON value is extracted, and if none is found the judge is asked once to repair it; every key must be a dimension and hold flaws; a missing severity counts as 1. A failed reading is requested again at temperature 1, six generations in all.

4.3 Scores

A dimension scores max(0, 100 − 5S + 1.5R): S sums the severities that are JSON integers (as Python's isinstance(x, int), so 3.0 and "3" do not count and true counts 1), R counts the lines not spoken by the environment, empty lines included. The variant with a ceiling of 100, which the official code has had since 2025-02-18, is stored beside it for each dimension of each session, so the effect of the choice can be computed from any run; a single session's dimension can exceed 100. A session's overall score is the mean of its four dimensions, and a run's dimension is the mean over its sessions. Nothing is rounded before display. BLEU is computed as nltk computes it on the lower-cased texts [5], stored ×100, and is not part of the overall score.

4.4 Failures

An empty reply is requested again at temperature 1, six tries in all, and the sixth result is recorded even if empty, as the official code does; it still counts in R. A content refusal by the provider takes the same path and is marked as refused; an account error or a gateway's 403 without a moderation marker is not treated as a refusal. Network errors, server errors and rate limits are retried at the same temperature with backoff, then fail the session, which can be retried from where it stopped; they never produce a score. If the judge cannot be read after six generations, or returns another dimension, the session fails rather than scoring zero. A scene that keeps failing therefore leaves the run incomplete; the official code would retry forever at the same point.

4.5 Run configurations and eligibility

A run would have qualified for a ranking if it covered all 200 scenes at least three times under the current protocol, every session was complete, and the model under test was served by a single host with fallbacks disabled. The protocol also carried a flag that kept every run report-only until the anchor was judged. No run was ever ranked. Reasoning settings may be used and are marked in the model's name. The protocol's temperature, output limit and similar parameters cannot be overridden by connection settings. An address other than OpenRouter counts as one host.

4.6 Uncertainty and what each run records

The interval given with a run is a 95% percentile bootstrap (1,000 resamples) over the 200 scenes, resampling the three sessions of a scene together, because their scores are correlated. The result of each run also holds: the mean of each of the three runs and their sample standard deviation; the means of the in-domain and out-of-domain scenes; and, per role, the number of lines, of empty lines, of replies cut at the limit, of refusals, of calls that needed a retry, and for the NSP the fallbacks and end signals. The paper's ± is undefined, so the platform's standard deviation is not presented as the same quantity. The three runs are not independent in the statistical sense: their prompts are identical, including the seeded choices, and every first request is at temperature 0, so they differ only through what the providers vary. The paper-era code also reset its seed for each run.

4.7 Data availability and licences

The two runs of the anchor (Section 7) are not public: their simulations contain test-set text. Their numbers are in the audit bundle, by scene and without text. The repository states no licence (an open issue asks for one), so the platform loaded the test set from the pinned commit for evaluation, does not republish it, and quotes no prompts in this report beyond short phrases (Appendix A). The audit bundle contains no test-set text.

5 Judge validity

This reproduction checks that the platform runs the authors' procedure. It cannot check how well that procedure measures role-playing. The paper's evidence on that question is summarised here; none of it can be recomputed, because the annotations and individual scores were not released.

Three readings of these numbers matter. The GPT-4o judge agrees with people less often than BLEU does in the authors' own measurement. People rank GPT-4o lower among these models than the GPT-4o judge does, which the authors put down to self-preference; GPT-4o is also the judge and the environment here. And the judge is one API model, which Section 7.3 shows does not necessarily behave as it did for the authors.

6 Reproduction verification

6.1 Verification matrix

Table 6 states what is compared at each stage and the result. coser_spec_check.py compares the platform's protocol module with a Python reference whose official functions are the repository's source text, pasted mechanically and checked against the repository on every run. The reference is not an official program: where the protocol takes the paper-era rule, the reference applies that rule, so the comparison shows that the platform implements the protocol as defined in Section 3. No model is called in these checks, except the last two rows of Table 6. They ran on 2026-11-01; outputs and logs are in the audit bundle.

6.2 What the reviews found and changed

Three reviews were run independently of each other, from the paper, from the official code, and from the numbers. The reviewers are AI agents run by the operator of this platform, not outside researchers; no one outside has reviewed this reproduction. The first two wrote their conclusions before reading the existing matrix and report; the third started from the report, as its task was to check its numbers. Their reports (in Chinese) are in the audit bundle. Table 7 lists the findings that changed something.

Findings left as they are, with their reasons, are in the matrix (“交叉验证发现的处理”): the empty-line length bonus, the 512-token limit and the asymmetric repetition removal are official behaviour and are listed as limitations.

6.3 What this report establishes, and what it does not

1. Is every setting traced to a source? Yes: Table 3 names each one, and the platform choices say why.
2. Does the platform say the same as the paper? In the prompts, word for word except four listed items. In the formula, yes, with the floor at 0 taken from the code of the paper's time and the length term read as the released examples require. The next-speaker model is different.
3. Does the platform do what its protocol says? On every request and score checked, yes, both in the protocol module and on the real engine with stand-in models. The engine runs used one family of model rules (“other”); the other families were checked in the protocol module only. Those checks used stand-in models; of the live providers, only OpenAI through OpenRouter has been exercised, in the anchor.
4. Does a platform run reproduce the paper's numbers? No, by the criteria fixed in advance (Section 7.2). Two dimensions come close and two do not. No official commit equals the protocol, the released examples are three scenes outside the test set, and no simulation behind the paper's table was released, so the comparison cannot go further than Section 7.
5. Does the critic measure what people judge? This report cannot say. The authors' figure is 68.6%, and it cannot be checked (Section 5).
6. Is the test set what the paper describes? Its counts are. Its sampling and the “final 10%” cannot be checked beyond 4 of 200 scenes.

7 Paper anchor and withdrawal of the leaderboard

7.1 Design and criteria

The anchor runs the full test set under protocol 1.1 with openai/gpt-4o-2024-08-06, pinned to OpenAI, as the model under test, and compares it with the paper's GPT-4o row. The criteria below were written on 2026-11-01, before any measurement, and are in the matrix (V09).

Reproduced or same range would have opened platform runs. Anything else is not reproduced: runs stay closed, the cause is examined, and the project owner decides.

Basis. If the printed ± is taken as the standard deviation of three runs, which the paper does not state, the difference of two three-run means has a standard deviation of about √(2/3)·s: 0.41 for the Average (s = 0.50), 0.39 to 1.41 for the dimensions. 1.0 is 2.4 times the value for the Average; 2.5 is 1.8 times the value for character fidelity and at least 4.6 times the others. Both exceed the gap between the paper's own two GPT-4o figures (0.45 and at most 0.71). 3.0 is about a fifth of the range of the 21 Averages (45.53 to 59.95).

7.2 Result: not reproduced

Three cells exceed the “same range” limits, in bold, so by the criteria the result is not reproduced. The 95% interval of the Average, resampling scenes, is [53.52, 55.8], so this is not noise. The ceiling makes no difference: with it the Average is 54.67, and only 3 session dimensions exceed 100.

The run itself was clean. The model under test wrote 7,222 lines, with 0 empty, 0 cut at the limit and 0 refused. The next-speaker model's name was replaced at random in 3.7% of its calls, and it ended 61% of the sessions. Sessions have 12.0 character lines on average. 24 sessions failed at their first step with a provider's HTTP 503 (error code 1019), having produced nothing, and were retried as the protocol allows. The run cost US$60.01; a 20-scene sample run before it cost US$2.05 and showed the same pattern (Average 56.39, character fidelity 32.15).

Where the gap comes from. In 25% of sessions the judge lists 10 or more character-fidelity flaws, nearly one per line, in 13% it does so for anthropomorphism, and in the other two dimensions almost never. 161 of 600 sessions score 0 on character fidelity and 75 on anthropomorphism, against 1 and 1 on the others. The length of the simulation does not explain the Average, which stays between 53.5 and 56.4 across all lengths (Table 8b), though it does move the dimensions in opposite directions.

7.3 Rechecking the judge on the authors' own simulations

To separate the judge from everything before it, the three simulations the authors released were sent through the official judge prompts, to the same model and host, twice each, and compared with the critiques the authors stored for them. No simulation is generated, so the next-speaker model and the model under test play no part. The criteria were written into coser_judge_recheck.py before the first request; the result is harsher. The authors' file does not record which judge produced its critiques; the README asks for gpt-4o-24-08-06 and the example command names gpt-4o.

The pattern matches the anchor: character fidelity moves most, anthropomorphism a little, the other two hardly. The change therefore appears on identical input, before anything the platform generates.

No variant brings the stored critiques back. Filling in the placeholder that both code generations leave literal changes nothing, nor does the other host. The later snapshot stops listing line by line but stays above the stored severities.

What this establishes. On these three simulations, today's judge is harsher on character fidelity than the stored critiques, with the same prompts and the same model name. What it does not. Why: the authors' setting at the time is not recorded and a provider's changes cannot be observed. Three scenes outside the test set are a small sample, although two passes agree closely. That the judge is the whole gap: the next-speaker model and the model under test have not been measured separately, and the agreement of the other two dimensions only suggests that they are not the main cause.

7.4 Why the leaderboard was withdrawn

The anchor's failure raised a broader question: whether the score, however faithfully produced, measures what a ranking needs. The evidence, from 600 sessions of one model (Table 8e) and from the paper's own tables, is that it does not.

1. The score is a count of what the critic chooses to write, not a scale. It starts at 100 and only falls: good role-playing earns nothing, and the instruction to identify all instances of flaws invites longer lists. The list length is not stable. In 56 of 200 scenes the same model on the same scene received at most 5 character-fidelity flaws in one run and 10 or more in another. The lower end is cut off: the 161 sessions at 0 would average -48.3 without the floor, so the scale cannot tell them apart. The top is not 100 either: the official code writes 100 for the original conversation without judging it, and when we judged it (Table 8f) the length term took it to 115.9 on storyline consistency.
2. The length correction does not correct. One λ is applied to dimensions whose flaws grow with length very differently (Table 8e). After it, a longer simulation gains on storyline consistency and quality and loses on character fidelity and anthropomorphism. The term is not small: it supplies 18.1 points of the Average, about a third, and it counts lines however empty, so a blank line earns 1.5 points in every dimension. The length is largely decided by the next-speaker model, not by the model under test.
3. Comparing with the original adds little, on the authors' own evidence. The simulation starts from scratch and the actors never see the original conversation, so a sound continuation that goes elsewhere counts as deviation; one conversation stands in for everything the scene allows. In the paper's alignment table (Table 5), removing the reference moves GPT-4o from 68.6% to 64.3% and DeepSeek-R1 from 77.5% to 77.2%. Removing rubrics, length correction or dimension separation costs at most 4.1 points each. BLEU, a plain count of shared word sequences at 5.90 for GPT-4o, reaches 75.3%, above the whole judged pipeline with GPT-4o. The elaborate procedure does not earn its authority.
4. The instrument moves as much as the models differ. The anchor's shift of 5.27 on the Average exceeds the 3.71 between the paper's top model and its median; 15 of its 21 models lie within that distance of the top. The shift of 16.12 on character fidelity is nearly the whole range of the 21 models on that dimension (31.51–48.95), and it appears on identical input with the same model name (Section 7.3). Other judges move scores further: GPT-3.5 gets 52.8, 35.9 and 40.5 in the paper's own table, and later work with other judges gives CoSER-70B 35.95 against the paper's 59.06 (Section 9). The score has no scale independent of the judge, and human annotators order models differently from it: GPT-4o is first by its own judgement and third of seven by theirs (Table 5).
5. Averaging does not rescue it. For one session the score is dominated by noise: the median range of a scene's overall score across three runs is 13.8 points. Averaging 600 sessions narrows the interval to about ±1.1, but it narrows the error around whatever the instrument measures, and items 1 to 4 are about what it measures: list length, simulation length and the judge's current habits. A precise number of that kind is not a ranking of role-playing ability.

A test that went the other way. We expected the critic to find flaws in the book itself, and sent the original conversations of 24 scenes to it as if they were simulations. The criteria, written into coser_method_validity.py before the first request, asked for at least 90 on storyline consistency and more than GPT-4o on the same scenes on all four dimensions. The result is passes (Table 8f): the book scores 97.94 on average against 51.75 for GPT-4o, so the critic does separate them. The test is weaker than it looks, because it is not blind: the text being judged is word for word the reference the critic is given, and a critic that only compares the two would give the same result. It therefore shows that the score rewards matching a conversation the model under test never saw, and does not show how the critic would judge good writing it had no reference for. Even so, the critic lists 4.2 anthropomorphism flaws per scene in human-written dialogue, with 9.8 points of severity against 18.1 for GPT-4o, and rates the book below GPT-4o on that dimension in 3 of the 24 scenes.

These problems lie in the method, not in the reproduction, so no correction to this platform's implementation would remove them, and changing the judge, the scale or the formula until the paper's numbers return would be fitting the protocol to a result. A leaderboard would present those numbers as an ordering of models, so the platform withdrew it.

What this does not show. The evidence comes from one model on this platform and from the paper's tables; the order of models under today's judge was not tested. A 68.6% agreement with people is better than chance, and the critic does separate the book from GPT-4o (Table 8f), so the score is not pure noise. The authors' numbers may be exactly what their judge returned at the time. The CoSER dataset and its use for training are outside this analysis. For anyone comparing new GCA scores with the paper's table using gpt-4o-2024-08-06 as the judge, the practical warning is to check first that the judge still returns the authors' stored critiques. Reinstating a board would take a different scoring method, and with it a different benchmark.

8 Reproducing this report

The checks behind Sections 2, 3, 6 and 7 are published as an audit bundle, /audit/coser/2026-11-01/, with the SHA-256 of every file: the scripts and the platform source as run, each step's output and log, the outputs of the local end-to-end runs, the anchor's numbers by scene, the judge recheck and the analysis of Section 7.4, the three independent reviews, and the search behind Section 9. It contains no test-set text. With the bundle, the authors' repository and the arXiv e-prints, a third party can repeat the scripted checks without any model:

git clone https://github.com/Neph0s/CoSER repo
for v in 1 2 3; do mkdir -p eprint/v$v; curl -L https://arxiv.org/e-print/2502.09082v$v | tar xz -C eprint/v$v; done
python3 -m venv venv && venv/bin/pip install nltk==3.9.1 regex
venv/bin/python -c "import nltk; nltk.download('punkt_tab', download_dir='nltk_data')"
venv/bin/python -I tools/experiments/coser_spec_check.py repo nltk_data          # Section 6: the protocol module (needs node)
python3 -I tools/experiments/coser_prompt_check.py eprint/v3                     # Appendix A: prompts against the paper (needs node)
python3 -I tools/experiments/coser_table_checks.py eprint repo                   # Section 2: the paper's own numbers
python3 -I tools/experiments/coser_paper.py eprint/v3                            # Section 2.1: rewrites published.ts; git diff must stay empty
venv/bin/python -I tools/experiments/coser_rouge_check.py repo work              # Section 6: ROUGE-L; downloads the rouge releases, needs uv
python3 -I tools/experiments/coser_e2e_mock.py 18999 requests.jsonl              # the stand-in for every model
venv/bin/python -I tools/experiments/coser_e2e_check.py repo nltk_data requests.jsonl <api> <token file> <run>   # after a local run
venv/bin/python -I tools/experiments/coser_judge_recheck.py repo <file with an OpenRouter key> out.json          # Section 7.3: about US$0.25, calls the live judge

The end-to-end check needs a local Worker pointed at the stand-in (OPENROUTER_BASE_URL). Some findings rest on the reviews and are not re-checked by a bundled script: the reading of the commit history, the sections of the paper not concerning evaluation, and the search of Section 9.

9 Third-party results

We searched arXiv, GitHub, Hugging Face and the web on 2026-11-01 for later work reporting CoSER scores; queries and sources are in coser_third_party_search.md in the audit bundle. Google Scholar could not be used and Semantic Scholar returned only part of the citations, so the list is not exhaustive. We found no complete reproduction with the paper's configuration: no work combining gpt-4o-2024-08-06 as judge, all 200 scenes and the models of the main table. These results cannot be set beside the paper's, because their judges, next-speaker models and sometimes their test sets differ, most without stating them. CogDual, with a GPT-4o judge of unstated version, reports an Average for GPT-4o of 54.76, close to the 54.68 of Section 7.2; that may be a coincidence, as its other settings are not stated. They do show how strongly the score depends on the judge and the rest of the configuration. In the official repository's 12 issues nobody reports a failure to reproduce or a disagreement with the paper's numbers.

10 Platform results and history

The platform has no CoSER leaderboard. It made two runs of CoSER, both with GPT-4o (2024-08-06) as the model under test, both for the anchor: a 20-scene sample and the full test set of Section 7.2. Neither was ever ranked. Both stay in the platform's database for audit, not public; the addresses of the board now lead to this report.

10.1 Protocol history

1. 1.0 (labelled 2026-10-28; committed 2026-10-27 19:53 UTC) was the first protocol. It used the next-speaker prompt, parser and history of December 2025 and a ceiling of 100, and had the engine faults of Table 7. No run of it is on the leaderboard.
2. 1.1 (2026-11-01) is the last protocol: the paper's next-speaker prompt with the paper-era parser and history, no ceiling, the official retry sequence, and fixed settings. It was checked as in Section 6 and anchored as in Section 7, and was withdrawn the same day with the leaderboard.

10.2 Report history

1. 2026-10-27. First versions (their text was dated 2026-10-26): the original benchmark and the published results, then the published results separated from platform runs as the original leaderboard, with a short description of the planned reproduction.
2. 2026-11-01. Rewritten: the protocol and every setting's source; inconsistencies in the sources; verification; the anchor's criteria; third-party work; audit bundle. Corrections to earlier versions: the test set is held out from the final 10% of each book, not from the final 10% of books; the script of the paper's time is the most likely source of the numbers, not established as the source; the commit is dated 14 February, not the day of v1; “every model sees the same prompts” holds for the seeded choices only, as output examples and roles follow the model family. Later the same day: the anchor's result, the recheck of the judge, an analysis of what the score measures, the withdrawal of the leaderboard on that ground, and the paper's main table as Appendix B, since the original leaderboard no longer lists it.

11 Limitations

1. Not reproduced, and the score itself is in question. Scores from this protocol cannot be compared with the paper's, and Section 7.4 explains why they should not rank models either. The cause of the gap is established only as far as Section 7.3 goes.
2. Next-speaker model. It differs from the paper, and its separate effect cannot be measured without deploying CoSER-70B.
3. No official commit equals the protocol, and no published number was produced by it.
4. The judge is an evaluated model and a provider's model. GPT-4o judged itself in the anchor; the authors report self-preference. A dated model name does not guarantee the behaviour of the time (Section 7.3).
5. Judge validity cannot be checked (Section 5). The authors' alignment of 68.6% is lower than that of BLEU in the same table.
6. The protocol was built for 2024 models. Three official rules can affect today's models unevenly, and none has been measured, as only GPT-4o was run: the limit of 512 tokens may leave a model that reasons inside that limit with short or empty replies; empty lines earn the length bonus (1.5 per line in each dimension), which the judge may or may not outweigh; and repetition is removed from the replies of all but GPT and Claude models, while the rubric penalises repetition. Reasoning models averaged several hundred output tokens per reply on another benchmark here, so this limit would have mattered.
7. Refusals. Treating them as empty lines is a platform choice. A scene whose judge keeps being refused leaves the run incomplete.
8. Thinking removal favours models that write reasoning into their reply.
9. Three runs are not independent repetitions (Section 4.6), so their standard deviation understates what independent repetitions would show, and the paper's ± is undefined.
10. No outside review. The three reviews were run by the operator (Section 6.2).
11. Providers largely untested. The verification used stand-in models, and the anchor used OpenAI alone, which refused nothing. How other providers report refusals on this material is unknown.
12. Test set. Its construction cannot be verified, and the repository states no licence for it.
13. Partial reproduction of the paper. The anchor covers the main table's four dimensions, Average and BLEU. They do not cover ROUGE-L, win rates, the k = 3 setting, ablations, retrieval augmentation, other judges, other benchmarks or the human evaluation.
14. Model family table. It reads model ids, and a misclassified model would get different prompts. Only the “gpt” family has met a live model.

References

1. X. Wang, H. Wang, Y. Zhang, X. Yuan, R. Xu, J. Huang, S. Yuan, H. Guo, J. Chen, S. Zhou, W. Wang, Y. Xiao. CoSER: A Comprehensive Literary Dataset and Framework for Training and Evaluating LLM Role-Playing and Persona Simulation. ICML 2025. arXiv:2502.09082 (v3, 4 Jan 2026; v1 of 13 Feb 2025 and v2 of 28 May 2025 carry the title “CoSER: Coordinating LLM-Based Persona Simulation of Established Roles”). https://arxiv.org/abs/2502.09082
2. Neph0s. CoSER: code, test data and GCA evaluation. GitHub, commits 93c1474 (14 Feb 2025) and 624efc5 (2 Apr 2026). The repository states no licence. https://github.com/Neph0s/CoSER
3. Neph0s. CoSER-Llama-3.1-70B. Hugging Face, model weights (the model card is tagged MIT). https://huggingface.co/Neph0s/CoSER-Llama-3.1-70B
4. Y. Dubois, B. Galambosi, P. Liang, T. B. Hashimoto. Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. arXiv:2404.04475, 2024. https://arxiv.org/abs/2404.04475
5. S. Bird, E. Klein, E. Loper. NLTK 3.9.1, the implementation of BLEU and word tokenization this report ports. https://www.nltk.org/
6. pltrdy. rouge: a Python implementation of the ROUGE metric. PyPI, versions 0.2 to 1.0.1. https://pypi.org/project/rouge/
7. C. Liu, Y. Lu, F. Ye, et al. CogDual: Enhancing Dual Cognition of LLMs via Reinforcement Learning with Implicit Rule-Based Rewards. arXiv:2507.17147v1, 2025. https://arxiv.org/abs/2507.17147v1
8. Z. Xiao, M. Li, X. Wang, et al. DREAM: LLM-based Dynamic Role-playing via Event-Aware Memory Graph. arXiv:2608.05170v1, 2026 (KDD 2026). https://arxiv.org/abs/2608.05170v1
9. C. Du, X. Wang, A. Chen, et al. HER: Human-like Reasoning and Reinforcement Learning for LLM Role-playing. arXiv:2601.21459v4, 2026 (Findings of ACL 2026). https://arxiv.org/abs/2601.21459v4
10. X. Wang, S. Zheng, H. Wu, et al. Agentopia: Long-Term Life Simulation and Learning in Agent Societies. arXiv:2606.07513v1, 2026. https://arxiv.org/abs/2606.07513v1
11. Z. Xu, D. Chen, J. Li, et al. Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization (RAPO). arXiv:2606.27025v1, 2026. https://arxiv.org/abs/2606.27025v1
12. W. Sun, X. Zhou, J. Liu, et al. Reinforcing Human Behavior Simulation via Verbal Feedback (Ditto). arXiv:2605.20506v1, 2026. https://arxiv.org/abs/2605.20506v1
13. X. Zhou, W. Sun, W. Du, et al. OdysSim: Building Foundation Models for Human Behavior Simulation. arXiv:2606.14199v2, 2026. https://arxiv.org/abs/2606.14199v2
14. L. Wang, W. Xie, X. Mou, et al. Socio-Foundation: A Model for Generalizable Individual Behavior Simulation via Hierarchical Capability Distillation. arXiv:2610.08967v1, 2026. https://arxiv.org/abs/2610.08967v1
15. T. Chen, J. Ou, Z. Liu, et al. Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservation (CaMOPD). arXiv:2605.27115v2, 2026. https://arxiv.org/abs/2605.27115v2
16. H. Zhang, Z. Xiang, W. Xie, et al. PersMem: Internalizing Personality into Dual-Pathway Memory for LLM Agents. arXiv:2609.34372v1, 2026. https://arxiv.org/abs/2609.34372v1
17. Z. Zhang, L. Liu, Q. Chai, et al. Adversarial Closed-Loop Curriculum for Evolving Role-Playing Agents (AdvRole). arXiv:2609.28609v1, 2026. https://arxiv.org/abs/2609.28609v1

Appendix A: Prompts

The prompts are printed in the paper's tables (Tables/prompts_agent.tex, prompts_agent_2.tex, prompts_eval.tex, prompts_eval_2.tex) and in the repository at the pinned commits. Because the repository states no licence, this report identifies them instead of reprinting them. The platform's strings are extracted from the source with Python's parser and never typed by hand; coser_prompt_check.py compares them with the LaTeX tables word by word and fails on any difference not listed below. Table A1 is read from its output.

Models also receive text that the paper's tables do not print, from the code of both generations: a line limiting the reply to 60 words after the role-play instruction (“Limit your response to 60 words.”), a first user message marking the start of the conversation, one of five separators between the other characters' profiles, and, for the families of Section 4.1, a line telling the model to start with its character's name.

Appendix B: The paper's main table

The results the authors published, read unchanged from Tables/main_exp.tex of the arXiv e-print by coser_table_checks.py (byte-identical in v1, v2 and v3). These are the authors' results, not the platform's, and they are means of three runs with the paper's undefined ± left out. CoSER-8B and CoSER-70B are the authors' models. Section 7 explains why they could not be reproduced with today's judge.