---
title: "CoSER: An Independent Reproduction Report — SociaBench"
headline: "CoSER: An Independent Reproduction Report"
description: "CoSER [1] evaluates role-playing of established characters through given-circumstance acting (GCA): a model plays every character of a scene taken from a renow…"
canonical: "https://sociabench.com/b/coser/methodology"
markdown: "https://sociabench.com/b/coser/methodology.md"
type: "ScholarlyArticle"
entity: "https://sociabench.com/b/coser/methodology#report"
author: "SociaBench"
publisher: "SociaBench"
language: "en"
date_modified: "2026-10-26"
about: "https://sociabench.com/b/coser#benchmark"
leaderboard: "https://sociabench.com/b/coser"
is_based_on: "https://arxiv.org/abs/2502.09082"
keywords:
  - "CoSER"
  - "given-circumstance acting"
  - "GCA evaluation"
  - "literary role-playing"
  - "persona simulation"
  - "GPT-4o judge"
  - "ICML 2025"
  - "CoSER 复现"
  - "名著角色扮演"
audience: "Researchers citing CoSER results and teams evaluating multi-character role-play grounded in literature."
provenance:
  - section: "1 The original benchmark"
    url: "https://sociabench.com/b/coser/methodology#original"
    source: "原始论文或发布方的内容与结论"
  - section: "2 Published results and their provenance"
    url: "https://sociabench.com/b/coser/methodology#published"
    source: "原始论文或发布方的内容与结论"
  - section: "3 Reproduction on this platform"
    url: "https://sociabench.com/b/coser/methodology#reproduction"
    source: "本平台的分析与计划"
  - section: "4 Limitations"
    url: "https://sociabench.com/b/coser/methodology#limitations"
    source: "本平台的分析与计划"
references:
  - text: "X. Wang, H. Wang, Y. Zhang, et al. CoSER: A Comprehensive Literary Dataset and Framework for Training and Evaluating LLM Role-Playing and Persona Simulation. ICML 2025. arXiv:2502.09082 (v1 13 Feb 2025; v3 4 Jan 2026)."
    url: "https://arxiv.org/abs/2502.09082"
  - text: "Neph0s. CoSER: code, test data and GCA evaluation. GitHub, commit 624efc5 (2 April 2026)."
    url: "https://github.com/Neph0s/CoSER"
  - text: "Y. Dubois, B. Galambosi, P. Liang, T. B. Hashimoto. Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. arXiv:2404.04475, 2024."
    url: "https://arxiv.org/abs/2404.04475"
license: "https://sociabench.com/data-terms"
cite_as: "SociaBench. CoSER: An Independent Reproduction Report. 2026-10-26. https://sociabench.com/b/coser/methodology"
---

SociaBench · Independent reproduction report

# CoSER: An Independent Reproduction Report

SociaBench

CoSER · Stage 1: published results · Last updated 2026-10-26 · [Leaderboard](https://sociabench.com/b/coser)

**Scope.** This platform is not affiliated with the authors of CoSER. Its CoSER leaderboard ranks only runs made here; the results the authors published, read unchanged from the paper's LaTeX source, are listed beside it as the original leaderboard and are not ranked with them. No model has yet been evaluated on CoSER here (Section 3).

Contents

1.  [Abstract](#abstract)
2.  [1 The original benchmark](#original)
3.  [2 Published results and their provenance](#published)
4.  [3 Reproduction plan](#reproduction)
5.  [4 Limitations](#limitations)
6.  [References](#refs)

## Abstract

CoSER \[1\] evaluates role-playing of established characters through given-circumstance acting (GCA): a model plays every character of a scene taken from a renowned book in a multi-agent simulation, and GPT-4o identifies flaws against the original conversation in four dimensions. The authors published results for 21 models, averaged over three runs, with an overall score defined as the mean of the four dimensions. This platform lists them unchanged, ranked by that average. The table is identical in all three arXiv versions and each printed average agrees with its dimensions within rounding. Two features of the protocol are relevant when reading the ranking: the critic, GPT-4o, is also among the evaluated models, and the next-speaker predictor is the authors' CoSER-70B, which is evaluated as well.

## 1 The original benchmark

### 1.1 Sources and data

| Source | Version | Licence |
| --- | --- | --- |
| Paper \[1\] | arXiv:2502.09082 v1–v3 (results table identical); ICML 2025 | arXiv licence |
| Code and test data \[2\] | commit 624efc5 (2 Apr 2026) | None stated in the repository |

**Table 1.** Primary sources, pinned to the versions this report describes.

The dataset covers 17,966 characters from 771 books. CoSER Test holds 200 conversations from the final 10% of books: 100 from books used to train the CoSER models and 100 from books not used. Test conversations are in English.

### 1.2 Given-circumstance acting and scoring

1.  **Simulation.** The evaluated model plays every character, with the scenario, character profiles and motivations. CoSER-70B predicts the next speaker, GPT-4o acts as the environment, and the simulation stops at an end signal or after 20 turns. The main results start from scratch (k = 0) and judge without inner thoughts.
2.  **Judging.** GPT-4o, given the original conversation and the plot summary, lists flaws per dimension in separate requests; each flaw has a severity from 1 to 5. A dimension scores 100 − 5 × the sum of severities, plus 1.5 × the simulation length as a length correction \[3\].
3.  **Dimensions.** Storyline consistency (against the original), anthropomorphism, character fidelity and storyline quality. The overall score is their mean; BLEU and ROUGE-L against the original dialogue are reported for analysis only.

The paper reports 68.6% agreement between the GPT-4o critic and human judges.

## 2 Published results and their provenance

The original leaderboard lists the 21 models of the paper's main results table, extracted by a script (`tools/experiments/coser_paper.py`) that records the source's SHA-256 and writes the same data module for the server and the worker. The table is identical in arXiv v1, v2 and v3. Each cell is printed as a mean with a ± value; the paper states that results are averaged over three runs but does not define the ± further, so the leaderboard shows the means only. Each printed average agrees with the mean of its four dimensions within rounding.

Model versions are as given in the paper (e.g. GPT-4o 2024-08-06, Claude-3.5-Sonnet 2024-06-20, Doubao-pro 241215). CoSER-8B and CoSER-70B are the authors' models trained on this dataset and are marked on the board. The paper's appendix re-scores the models with other judges and reports judge-specific preferences.

## 3 Reproduction on this platform

Platform runs follow the paper and, where it is silent, the authors' code. The numbers in the paper come from the evaluation script committed on the day of v1 (`eval_reproduce.py`, commit 93c1474); the current repository differs from it in ways that change scores, and where the paper's text, prompt tables or released examples support the earlier behaviour, it is the one used: characters receive their motivations (the current code reads a field the test set does not have, so they get none), and every line is recorded with its speaker's name before the critic reads it. The model under test plays every character; GPT-4o (2024-08-06) plays the environment and judges, and GPT-4o-mini (2024-07-18), the current code's default, predicts the next speaker in place of the paper's CoSER-70B, a registered deviation. Each scene is simulated and judged three times; the dimension score is 100 − 5 × the integer severities + 1.5 × the characters' lines, clipped to 0–100. ROUGE-L cannot be reproduced from any version of the library the code uses and is not reported for platform runs.

The official run is not deterministic (its random choices depend on Python's per-process hash seed and on earlier scenes); the platform draws the same choices from a seed fixed by the scene, so every model under test sees the same prompts. Before platform runs are opened, GPT-4o (2024-08-06) is run as the model under test and compared with its row of the main table. Platform results are ranked separately from the paper's.

## 4 Limitations

1.  The leaderboard reproduces the authors' numbers; without the simulations they cannot be checked independently.
2.  The critic is also an evaluated model, and the next-speaker predictor is one of the authors' evaluated models.
3.  GPT-4o snapshots change over time; current results with the same model name may differ.

## References

1.  X. Wang, H. Wang, Y. Zhang, et al. CoSER: A Comprehensive Literary Dataset and Framework for Training and Evaluating LLM Role-Playing and Persona Simulation. ICML 2025. arXiv:2502.09082 (v1 13 Feb 2025; v3 4 Jan 2026). [https://arxiv.org/abs/2502.09082](https://arxiv.org/abs/2502.09082)
2.  Neph0s. CoSER: code, test data and GCA evaluation. GitHub, commit 624efc5 (2 April 2026). [https://github.com/Neph0s/CoSER](https://github.com/Neph0s/CoSER)
3.  Y. Dubois, B. Galambosi, P. Liang, T. B. Hashimoto. Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. arXiv:2404.04475, 2024. [https://arxiv.org/abs/2404.04475](https://arxiv.org/abs/2404.04475)

**转载声明** 转载或引用本站任何评测结果，请注明来源「SociaBench（AI 社交智能榜）」并附本站链接 [sociabench.com](https://sociabench.com)。

---

转载或引用本站任何评测结果，请注明来源「SociaBench（AI 社交智能榜）」并附本站链接：https://sociabench.com/b/coser/methodology