---
title: "CharacterEval: An Independent Reproduction Report — SociaBench"
headline: "CharacterEval: An Independent Reproduction Report"
description: "CharacterEval [1] evaluates Chinese role-playing conversational agents on 77 characters from Chinese novels and scripts. A model continues a multi-turn dialogu…"
canonical: "https://sociabench.com/b/charactereval/methodology"
markdown: "https://sociabench.com/b/charactereval/methodology.md"
type: "ScholarlyArticle"
entity: "https://sociabench.com/b/charactereval/methodology#report"
author: "SociaBench"
publisher: "SociaBench"
language: "en-US"
date_modified: "2026-10-29"
about: "https://sociabench.com/b/charactereval#benchmark"
leaderboard: "https://sociabench.com/b/charactereval"
is_based_on: "https://arxiv.org/abs/2401.01275"
keywords:
  - "CharacterEval"
  - "CharacterRM"
  - "BaichuanCharRM"
  - "Chinese role-playing benchmark"
  - "role-playing conversational agent"
  - "reward model"
  - "MBTI"
  - "ACL 2024"
  - "CharacterEval 复现"
  - "中文角色扮演评测"
  - "奖励模型"
audience: "Researchers citing CharacterEval results and teams evaluating Chinese role-playing agents, and anyone checking how a reproduction of its pipeline was traced to the official code."
provenance:
  - section: "1 The original benchmark"
    url: "https://sociabench.com/b/charactereval/methodology#original"
    source: "Content and conclusions from the original paper or publisher"
  - section: "2 Published results and their provenance"
    url: "https://sociabench.com/b/charactereval/methodology#published"
    source: "Content and conclusions from the original paper or publisher"
  - section: "3 Sources of the protocol"
    url: "https://sociabench.com/b/charactereval/methodology#differences"
    source: "This platform's reproduction and evaluation results"
  - section: "4 The protocol on this platform"
    url: "https://sociabench.com/b/charactereval/methodology#platform"
    source: "This platform's reproduction and evaluation results"
  - section: "5 Reward-model validation"
    url: "https://sociabench.com/b/charactereval/methodology#validation"
    source: "This platform's reproduction and evaluation results"
  - section: "6 Reproduction verification"
    url: "https://sociabench.com/b/charactereval/methodology#verification"
    source: "This platform's reproduction and evaluation results"
  - section: "7 Personality back-testing"
    url: "https://sociabench.com/b/charactereval/methodology#mbti"
    source: "This platform's analysis and plans"
  - section: "8 Reproducing this report"
    url: "https://sociabench.com/b/charactereval/methodology#reproduce"
    source: "This platform's reproduction and evaluation results"
  - section: "9 Third-party results"
    url: "https://sociabench.com/b/charactereval/methodology#third-party"
    source: "Third-party data"
  - section: "10 Platform results and history"
    url: "https://sociabench.com/b/charactereval/methodology#listing"
    source: "This platform's reproduction and evaluation results"
  - section: "11 Limitations"
    url: "https://sociabench.com/b/charactereval/methodology#limitations"
    source: "This platform's analysis and plans"
  - section: "Appendix A: Official templates"
    url: "https://sociabench.com/b/charactereval/methodology#appendix-templates"
    source: "Content and conclusions from the original paper or publisher"
references:
  - text: "Q. Tu, S. Fan, Z. Tian, R. Yan. CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation. arXiv:2401.01275 (v2, 9 January 2024). Published at ACL 2024 (2024.acl-long.638, pp. 11836–11850), where the author list has seven names."
    url: "https://arxiv.org/abs/2401.01275"
  - text: "morecry. CharacterEval: code, test set and reward-model training data. GitHub, commit c3d44a6 (27 May 2025, UTC). MIT licence."
    url: "https://github.com/morecry/CharacterEval"
  - text: "morecry. BaichuanCharRM (CharacterRM checkpoint). Hugging Face, commit e953da5 (10 January 2024)."
    url: "https://huggingface.co/morecry/BaichuanCharRM"
  - text: "X. Wang, Y. Fei, Z. Leng, C. Li. Does Role-Playing Chatbots Capture the Character Personalities? Assessing Personality Traits for Role-Playing Chatbots. arXiv:2310.17976v1, 2023 (the version the paper cites; later versions are titled InCharacter and use psychological interviews)."
    url: "https://arxiv.org/abs/2310.17976v1"
  - text: "Z. M. Wang, Z. Peng, H. Que, et al. RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models. arXiv:2310.00746, 2023."
    url: "https://arxiv.org/abs/2310.00746"
  - text: "J. Zhou, Y. Huang, B. Wen, et al. CharacterBench: Benchmarking Character Customization of Large Language Models. AAAI 2025. arXiv:2412.11912."
    url: "https://arxiv.org/abs/2412.11912"
  - text: "X. Wang, H. Wang, Y. Zhang, et al. CoSER: A Comprehensive Literary Dataset and Framework for Training and Evaluating LLM Role-Playing and Persona Simulation. arXiv:2502.09082 (v3), 2025."
    url: "https://arxiv.org/abs/2502.09082"
  - text: "L. Zheng, W.-L. Chiang, Y. Sheng, et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks. arXiv:2306.05685."
    url: "https://arxiv.org/abs/2306.05685"
license: "https://sociabench.com/data-terms"
cite_as: "SociaBench. CharacterEval: An Independent Reproduction Report. 2026-10-29. https://sociabench.com/b/charactereval/methodology"
---

SociaBench · Independent reproduction report

# CharacterEval: An Independent Reproduction Report

SociaBench

CharacterEval · Stage 2: implemented, reward model verified, anchor within one step, explained by numerics · Protocol charactereval-1.0 · 2026-10-28 · Last updated 2026-10-29 (UTC) · [Leaderboard](https://sociabench.com/zh-CN/b/charactereval) · [Audit bundle](https://sociabench.com/audit/charactereval/2026-10-29/)

**Scope.** This platform is not affiliated with the authors of CharacterEval. Its CharacterEval leaderboard ranks only runs made here; the results the authors published, read unchanged from the paper's LaTeX source, are listed beside it as the original leaderboard and are not ranked with them. The platform's implementation of the benchmark is complete and checked against the official code (Section 6). The reward model it needs has been deployed by an inference provider and passed the deployment check on 2026-10-28: its files match the released checkpoint, and it returns exactly the scores of the authors' scoring code (Section 5.2). On the authors' own released outputs it reproduces the paper's scores within one step, not within the tighter margin fixed in advance: all 150 pairwise orders of the five models hold, but 17 of 75 cells differ by more than 0.005, at most 0.012 (Section 5.3). A check fixed in advance shows that numerics alone move single scores by as much: changing only how the GPU library accumulates bfloat16 products moves them by a root-mean-square of 0.095, against the 0.121 the anchor implies (Section 5.3). By the criteria set beforehand, numerics explain the anchor and platform runs are open; the leaderboard has no platform result yet.

**Reproduction policy (protocol 1.0, 2026-10-28).** The platform follows the paper. Where the paper is silent it follows the authors' released code at a pinned commit, then the released files; where the paper's text and its own numbers disagree it follows the reading that reproduces the numbers. Only where none of these defines a setting does the platform choose, and it says so. Every setting is listed with its source in Section 3 and checked against that source by a script whose output, logs and inputs are published as an audit bundle (Sections 6 and 8). What this report establishes, and what it cannot, is stated in Section 6.2.

Contents

1.  [Abstract](#abstract)
2.  [1 The original benchmark](#original)
3.  [2 Published results and their provenance](#published)
4.  [3 Sources of the protocol](#differences)
5.  [4 The protocol on this platform](#platform)
6.  [5 Reward-model validation](#validation)
7.  [6 Reproduction verification](#verification)
8.  [7 Personality back-testing](#mbti)
9.  [8 Reproducing this report](#reproduce)
10.  [9 Third-party results](#third-party)
11.  [10 Platform results and history](#listing)
12.  [11 Limitations](#limitations)
13.  [References](#refs)
14.  [Appendix A: Official templates](#appendix-templates)

## Abstract

CharacterEval \[1\] evaluates Chinese role-playing conversational agents on 77 characters from Chinese novels and scripts. A model continues a multi-turn dialogue as the character; the authors' reward model CharacterRM \[3\] scores the reply on the subset of twelve metrics assigned to that example, each on a 1–5 scale, and a thirteenth, personality back-testing, compares the model's answers to an MBTI questionnaire with the character's known type. The paper gives neither the generation prompt nor any generation setting, nor the reward model's input format; the authors' code and checkpoint define them. This report traces each setting of the platform's protocol to the paper, the code at a pinned commit, the released files or a stated platform choice (21 settings, Section 3), and describes the implementation (Section 4). An automated check compares it with a Python reference built from the official code: all 4,531 generation requests and all 8,032 reward-model inputs are byte-identical, and aggregation agrees to machine precision (26 of 26 checks, Section 6). Three parts of the paper cannot be reproduced from the released materials: personality back-testing, whose labels, questionnaire and code were never published; the reward model's agreement with human ratings, whose held-out annotations were not released; and the appendix results judged by GPT-4, whose judge prompt was not released. The 15 published results are listed unchanged; their printed dimension averages agree with their metrics. The deployed reward model matches the released weights file by file and returns exactly the scores of the authors' scoring code on 26 probe inputs (Section 5.2); on the outputs the authors released for five models, it reproduces the paper's 75 cells within one step: every cell within 0.012 and every pairwise order kept, but 17 cells beyond the 0.005 fixed in advance as reproduction. The differences behave like random noise on single scores, and a check fixed in advance shows that numerical precision alone moves single scores by the same order, so platform runs are open (Section 5.3).

## 1 The original benchmark

### 1.1 Sources

| Source | Version | Licence |
| --- | --- | --- |
| Paper \[1\] | arXiv:2401.01275 v2 (9 Jan 2024); ACL 2024. LaTeX source `acl_latex.tex`, SHA-256 `f9829973…6c17` | arXiv licence |
| Code, test set, profiles, metric assignment, frozen outputs \[2\] | commit c3d44a6 (27 May 2025, UTC) | MIT |
| CharacterRM \[3\] | commit e953da5 (10 Jan 2024); Baichuan2-13B-Base backbone with a linear scoring head; custom model code | None stated; the base model’s licence may apply |

**Table 1.** Primary sources, pinned to the versions this report describes.

arXiv v1, v2 and the ACL camera-ready version print the same numbers in the paper's Table 4 and its appendix GPT-4 table (240 cells each), its correlation table (52 cells) and its Table 1; in these tables v2 only renamed BC-Character-Turbo to BC-NPC-Turbo (elsewhere v2 also changed the stated total of examples from 23,020 to 11,376 and two figures). The ACL version swaps the numbers of the correlation table and Table 3; this report uses the arXiv v2 numbering. The reward-model repository has no tagged release; e953da5 is its latest commit, and its weight files have the same hashes as the preceding commit e1b9f1e.

### 1.2 Data

The data comprise 1,785 multi-turn dialogues (9.28 turns on average) with 77 characters; profiles come from Baidu Baike. Examples (character, context, response) are split into 6,811 for training (not released) and 4,564 for testing. Each test example has a character, the name of its source work and a multi-line dialogue context; there is no reference reply. Each test example is scored only on the subset of metrics annotators judged relevant to it (sparse evaluation, `data/id2metric.jsonl`): 1 to 11 of the 12 metrics, 8,032 example–metric pairs in all, from 308 pairs (fluency) to 744 (empathy). 33 test examples have no metric, so the official pipeline scores 4,531 examples.

### 1.3 Metrics

| Metric | Definition | Dimension | Name sent to the reward model |
| --- | --- | --- | --- |
| Flu. | Fluency | Conversational ability | 对话流利度 |
| Coh. | Coherency | Conversational ability | 对话连贯性 |
| Cons. | Consistency | Conversational ability | 对话一致性 |
| KE | Knowledge exposure | Character consistency | 知识曝光度 |
| KA | Knowledge accuracy | Character consistency | 知识准确率 |
| KH | Knowledge hallucination (higher = fewer) | Character consistency | 知识幻觉性 |
| PB | Persona behavior | Character consistency | 行为一致性 |
| PU | Persona utterance | Character consistency | 言语一致性 |
| HL | Human-likeness | Role-playing attractiveness | 类人程度 |
| CS | Communication skills | Role-playing attractiveness | 交流技巧 |
| ED | Expression diversity | Role-playing attractiveness | 表现多样性 |
| Emp. | Empathy | Role-playing attractiveness | 共情度 |
| MBTI | Personality back-testing accuracy (0–1) | Personality back-testing | — |

**Table 2.** The thirteen metrics of CharacterEval in four dimensions. The name given to the reward model is the Chinese metric name of `id2metric.jsonl`, not the English abbreviation.

### 1.4 Scoring

CharacterRM reads the character profile, the context, the response and the metric name, and outputs one value that the released code maps to 1–5 (sigmoid output × 4 + 1); the score is deterministic. A metric's score is the mean over the examples assigned to it; the paper reports a mean per dimension without defining it, and defines no overall score. All metrics are oriented so that higher is better, including knowledge hallucination. The reward model was trained on scores from 12 annotators; the paper reports an overall Pearson correlation of 0.631 with human scores, against at most 0.385 for GPT-4 with one to three demonstrations. Personality back-testing uses 54 characters whose MBTI types were collected from a public database, an MBTI test administered as in Wang et al. \[4\], and reports an accuracy.

### 1.5 What the sources leave open

1.  The generation prompt: the paper states that all models used the same prompt, slightly adjusted for closed models, without giving it; the released generation script covers ChatGLM3 only.
2.  Every generation setting (temperature, top-p, output length, sampling).
3.  The reward model's input format, tokenization, truncation and numeric precision.
4.  How dimension averages are formed (the reading that reproduces the printed values is in Section 3).
5.  Personality back-testing: labels, questionnaire, prompt and scoring code (Section 7).
6.  The human annotations behind the reward model's reported correlation.

The sources also disagree with each other in several places; Section 2.2 lists each disagreement with its location and its effect on reproduction. None changes how the benchmark is run.

### 1.6 Related work

Role-play benchmarks differ mainly in what they generate and who scores it. RoleLLM \[5\] scores single responses against reference answers; CharacterBench \[6\] scores single responses to user-customized characters with a fine-tuned judge; CoSER \[7\] has the model play every character of a scene from a book and has an LLM critic count flaws. CharacterEval sits with the single-response designs: a fixed context from a novel or script, one reply, and a trained reward model instead of a prompted LLM judge. The paper motivates the reward model by its higher agreement with human scores than prompted GPT-4 (Section 1.4), in line with the broader finding that prompted LLM judges agree only moderately with people and carry biases\[8\]. Personality back-testing follows Wang et al. \[4\], who had role-playing chatbots take an MBTI test. For reproduction, the consequence of the reward-model design is that the judge is a fixed checkpoint: unlike a prompted LLM judge it cannot drift behind an API name, so it can be verified file by file and its scores can be compared with the paper's exactly (Section 5).

## 2 Published results and their provenance

### 2.1 The published table

The original leaderboard lists the 15 models of the paper's Table 4, extracted by a script (`tools/experiments/charactereval_paper.py`) from the LaTeX source, which records its SHA-256 and writes the same data module for the server and the worker; the audit bundle re-runs the extraction and confirms the module is unchanged. It is the same table as the leaderboard image in the authors' repository, and arXiv v1, v2 and the ACL version print the same 240 cells. CharacterGLM has no personality back-testing result. All 90 printed dimension averages (Table 4 and the appendix GPT-4 table) are within 0.00067 of the unweighted mean of their printed metrics, which is what averaging the unrounded metric means and then rounding produces; averages weighted by the number of examples per metric do not agree.

Because the paper defines no overall score, the original leaderboard ranks by the character-consistency average, which the paper calls the most crucial aspect, and labels the column accordingly; every other dimension and metric can be used for sorting. Personality back-testing is an accuracy and is shown as printed, outside the score charts. The published results carry no confidence intervals, generation settings or evaluation dates; GPT-4 and GPT-3.5 are `gpt-4` and `gpt-3.5-turbo-1106`, and the other models are named without versions (for InternLM-7B, for example, the chat model's v1.0 and v1.1 cannot be told apart). The repository also releases the outputs of five of the open models in the form the reward model reads (ChatGLM3-6B, Baichuan2-7B, XVERSE-7B, InternLM-7B and Qwen-7B, `results/generation_trans_*.jsonl`, 8,032 rows each), which Section 5.3 uses as an anchor.

### 2.2 Inconsistencies in the sources

The three independent reviews (Section 3) and a mechanical check of the paper's own numbers (`charactereval_table_checks.py`, in the audit bundle) found the following. Line numbers refer to `acl_latex.tex` of arXiv v2.

| Where | What the sources say | Effect |
| --- | --- | --- |
| Example count | Text (lines 126, 168): 11,376 examples. Paper Table 1: 6,811 training + 4,564 test = 11,375. README: 23,020 (the number of arXiv v1). | None: the platform uses the released test file (4,564). |
| Number of baselines | Line 530: “10 baselines”. Paper Table 4: 15 models (11 families). | None. |
| Table 4 marks | Cons.: the underline (second best) is on Baichuan2-7B (3.827); the second best is Baichuan2-13B (3.864). The other columns are marked correctly. | None: numbers are taken, marks are not. |
| Correlation table “Overall” | Overall is not the mean of the 12 metrics (CharacterRM 0.631 against a mean of 0.561; GPT-4 1/2/3-shot 0.362/0.385/0.375 against 0.368/0.385/0.395), and the paper does not define it. | Not reproducible without the annotations (Section 5.1). |
| GPT-4 with more demonstrations | Line 472: GPT-4 “will improve with the number of demonstration increase”. Overall: 0.362, 0.385, 0.375 (not monotone); only the mean of the 12 metrics increases. | None. |
| Reward-model backbone | Line 471 and the checkpoint configuration: Baichuan2-13B-Base. ACL appendix and an author reply: Baichuan-13B-Base. | None: the checkpoint is used as released. |
| Reward-model training data | ACL appendix: trained on the training split. Released rm\_train\_data.json: all 7,228 rows come from test contexts (4,241 of the 4,531 scored test examples). | The reward model has seen the contexts it scores; stated as a limitation. |
| Score normalization | ACL appendix: labels divided by 5. Released code: training loss against labels / 4 (labels 1–5); inference sigmoid × 4 + 1. | Inference follows the code (× 4 + 1). The paper’s means near 1.6 fit neither labels / 4 nor labels / 5 as training targets, only (label − 1) / 4, which neither source states; this is our inference. |
| Personality back-testing values | Accuracies printed without a definition; every one of the 14 values equals k/216 within rounding. | Consistent with per-letter accuracy over 54 characters × 4 letters; not reproducible (Section 7). |
| README | Names ChatGLM-6B and Baichuan-7B-Chat and data/id2metric.json; the files and the paper’s Table 4 name ChatGLM3-6B, Baichuan2-7B and id2metric.jsonl; the .jsonl files are single JSON documents (an object for id2metric, arrays for the others), not JSON lines. | None: the files are used. |

**Table 3.** Disagreements within and between the sources, and their effect on this reproduction.

## 3 Sources of the protocol

Each setting takes the first of these sources that defines it: (1) the paper, where its text and its own numbers agree, or else the reading that reproduces its numbers; (2) the official code at c3d44a6; (3) the released files (test set, metric assignment, profiles, checkpoint); (4) a platform choice, only where none of the above defines the setting, stated with its reason. Three reviewers examined the paper, the code and the released files independently, without seeing each other's conclusions; Table 4 keeps only settings on which all three agreed or which were settled by returning to the source files. The full matrix with line references is published as `docs/复现规范/CharacterEval-核查矩阵.md`.

| Setting | Paper | Official code / files | Platform | Basis |
| --- | --- | --- | --- | --- |
| Test set | Table 1: 4,564 test examples | `test_data.jsonl`: 4,564 examples, unique ids, 77 characters | The released file, hash-pinned | Same |
| Examples scored | Only some metrics per example | `transform_format.py` keeps ids present in `id2metric`; 33 have none | Generates only the 4,531 scored examples | Platform choice: the official code generates and then discards the other 33; scores are unaffected |
| Metric assignment | Sparse evaluation | `id2metric.jsonl`: 8,032 pairs | Used as released | Same |
| Metric name given to the model | English names in the text and Figure 2 | The Chinese `metric_zh` of `id2metric` | metric\_zh | Code |
| Profile text | From Baidu Baike; format not given | Python `str(dict)` of the profile, nested lists and dicts included | The 78 Python reprs (77 characters used) generated offline and frozen; never re-implemented in TypeScript | Code |
| Generation messages | Same prompt for all models; not given | `get_response.py`: no system message; profile and instruction as the first user turn; dialogue as alternating turns (Appendix A) | Built exactly as the code builds them, for every model; no per-model adjustment | Code; no adjustment is a platform choice: the paper’s “slight adjustment” for closed models is unspecified |
| Context splitting | Not given | `make_inputs`: lines split at `\n`, speaker at the first full-width colon | Same | Code |
| Generation settings | Not given | ChatGLM3 path only; its wrapper defaults (T = 0.8, top-p = 0.8) | The connection’s own settings, recorded per run; output limit at least 1,024 tokens | Platform choice: the defaults belong to one model’s wrapper, and the paper’s closed models ran on their providers’ APIs |
| Reasoning content | Not given | None | Only the final reply text is used | Platform choice |
| Reply post-processing | Not given | `transform_format.py`: first line only (`split("\n") · [0]`); no strip; a “name：” prefix is kept | Same | Code |
| Failed generations | Not given | `None` or `"ERROR"` are dropped from every metric; an empty string is scored | A provider moderation refusal is dropped as ERROR; an empty reply after retries is scored as empty; other failures (network, 5xx, rate limits) fail the session for retry and are never dropped | Code; which failures count as ERROR is a platform choice (an infrastructure fault must not remove an example from the denominator) |
| Reward model | CharacterRM, Baichuan2-13B-Base | BaichuanCharRM e953da5; `score = Linear(5120 → 1)` on the last position, then sigmoid | Served by an inference provider; verified before use (Section 5) | Same |
| Reward-model input | Not given | `run_char_rm.py` `format_input` (Appendix A) | Built by the platform and sent through the gateway’s pass-through form, which must not add a template | Code |
| Tokenization and precision | Not given | No BOS; eos appended; the last 4,096 tokens kept; batch size 1; bfloat16 throughout | Required of the deployment and checked (Section 5) | Code. In bfloat16 a single score moves in steps of 0.0039–0.0156 depending on the score; another precision changes scores |
| Score transform | Five-point scale | `sigmoid × 4 + 1` | Same | Code. The paper’s means near 1.6 also imply training targets of (label − 1) / 4 (Table 3) |
| Metric aggregation | One number per cell of its Table 4 | `compute_score.py`: arithmetic mean over all records of the metric | Same; no rounding before display | Same |
| Dimension average | Avg. columns, undefined | None | Unweighted mean of the dimension’s metric means | The paper’s numbers: all 90 printed averages (its Table 4 and appendix GPT-4 table) are within 0.00067 |
| Ranking | No overall score | None | By character consistency, as on the original board | Platform choice, grounded in the paper’s text |
| Rounding | Three decimals | Full floats | Three decimals for display only | The paper’s tables |
| Personality back-testing | Described | Nothing released | Not offered | Not reproducible (Section 7) |
| Reward model vs humans | Correlation table (arXiv Table 2) | No held-out annotations | Not reproduced | Not reproducible (Section 5.1) |

**Table 4.** Where each setting of protocol 1.0 comes from. Paper: arXiv v2 LaTeX. Code: github.com/morecry/CharacterEval at c3d44a6.

## 4 The protocol on this platform

Protocol `charactereval-1.0 · 2026-10-28`. Every run stores this version, the data hashes, the reward-model identity and the generation settings it used; any change that can affect scores raises the version, and runs of an older version leave the ranking.

### 4.1 Generation

For each of the 4,531 scored examples, the model under test receives the message array that `get_response.py` builds (Appendix A): no system message; a first user message with the character's profile, as Python prints the profile dictionary, followed by the role-play instruction; an assistant turn in which the character accepts the role (and, when the character speaks first, its first line); then the dialogue as alternating turns, each line prefixed with its speaker; the last line of the context is the final user message. The arrays are built from the released data by code whose output is checked byte for byte against the official functions (Section 6). The model's reply is cut at its first line break, as `transform_format.py` does, and is otherwise left as it is. Requests allow at least 1,024 output tokens; other parameters are those of the connection and are recorded.

### 4.2 Scoring

For every metric assigned to the example, the platform builds the reward-model input exactly as `run_char_rm.py` does (profile, context, reply and the Chinese metric name between the markers of Appendix A) and sends the batch of an example's inputs to the reward-model endpoint, which returns one score per input. The endpoint must tokenize and run the model as the official script does (Table 4) and return the score after the transform; Section 5.2 checks this. A metric's score is the mean over its examples, a dimension's the plain mean of its metrics, and nothing is rounded before display. Each example is scored once: the reward model is deterministic, and there is no sampling to repeat.

### 4.3 Failures

A generation refused by the provider's content moderation is recorded and the example leaves every metric, as the official filter drops `"ERROR"`; the number of dropped examples is shown with the run. A reply that is still empty after the client's retries is scored as an empty string, as the official code scores one. Network errors, server errors and rate limits fail the example's session, which can be retried from where it stopped; such examples are never dropped. A threshold on the share of dropped examples for leaderboard eligibility has not been set yet.

### 4.4 Run configurations and eligibility

| Configuration | Examples | Reward-model scores | Leaderboard |
| --- | --- | --- | --- |
| Full test set | 4,531 | 8,032 | Eligible |
| Quick sample | About 240 (20 per metric) | One per assigned metric | Report only |
| Smoke test | 12 (one per metric) | One per assigned metric | Checks the setup |

**Table 5.** Run configurations. Only the full test set is eligible for the leaderboard.

A run enters the leaderboard when it covers the full test set under the current protocol, every example is generated and scored (dropped examples count as complete), and the model under test was served by a single host with fallbacks disabled. It is ranked by character consistency among platform runs only.

### 4.5 Uncertainty

Each dimension average shown for a platform run carries a 95% percentile bootstrap interval (1,000 resamples): the 4,531 examples are resampled with all their metric scores together, and each metric's mean is taken over the resampled examples assigned to it. The reward model adds no sampling noise; since each example is generated once, the interval reflects both the choice of examples and the randomness of generation. Examples from the same character and dialogue are correlated; the interval does not account for this clustering and is therefore somewhat too narrow, and a character-clustered interval will be reported with the first runs. The paper reports no intervals.

### 4.6 Data availability and licences

For every run on the leaderboard, the run report, each example's reply and its reward-model scores are public on this platform without an account, with the run's protocol snapshot. The test set, profiles and metric assignment are the authors', under the MIT licence of the repository; the platform loads them from the pinned commit and does not redistribute them beyond what a run's report shows. The reward-model checkpoint states no licence, and as a fine-tune of Baichuan2-13B-Base the base model’s licence may apply; it is used for evaluation only and is not redistributed. The audit bundle contains no test-set text, profile or model reply.

## 5 Reward-model validation

### 5.1 What the paper reports, and what cannot be checked

The paper reports the reward model's Pearson correlation with human scores per metric and overall (0.631; the paper's Table 2). The held-out annotations behind it were not released, and the released training data come from the test contexts (Section 2.2), so the correlation cannot be recomputed and the platform does not report its own. The checks below therefore establish that the platform runs the authors' reward model as the authors ran it, not how well that model agrees with people.

### 5.2 Deployment check

BaichuanCharRM is a 13-billion-parameter model with its own model code, which no inference provider offers; it has been deployed for the platform by an inference provider. Before use, the operator runs `tools/experiments/charactereval_rm_server_check.py` on the serving machine and returns its output unchanged. It performs three checks, with criteria written into the script before any result:

| Check | What it compares | Passes when | Status (UTC) |
| --- | --- | --- | --- |
| A. Weights | Each of the 11 files of the served model directory with Hugging Face e953da5: SHA-256 for the weight shards and the tokenizer model, git blob SHA-1 for configuration, tokenizer configuration and the model’s own Python code | Every file matches | Passed 2026-10-28 16:38: 11 of 11 files equal |
| B. Official scoring | The official run\_char\_rm.py on the 26 probes, on the served weights: bfloat16, batch size 1, no BOS, eos appended, last 4,096 tokens | Produces the reference scores | Done 2026-10-28 16:38: 26 scores (A800, torch 2.7.0, transformers 4.36.2) |
| C. Endpoint | The same probes through the deployed endpoint, in the platform’s request format: as one batch, in reverse order, one by one | Same score in all three passes; all in \[1, 5\]; every |endpoint − reference| ≤ 10⁻⁶ | Passed 2026-10-28 16:38: three passes identical; 26 of 26 equal to the reference (maximum difference 0) |

**Table 6.** Deployment check of the reward model. The 26 probes are reward-model inputs built by the official logic from the repository’s frozen ChatGLM3-6B outputs: for each metric, the two distinct inputs with the smallest sample ids; the longest of all 8,032 inputs; and one synthetic input, the longest input with its context repeated until it exceeds 20,000 characters, which the official pipeline never scores and which tests truncation to the last 4,096 tokens (tools/experiments/charactereval\_rm\_probes.py).

Differences of at most 0.016 would point to batching, padding or kernel differences and must be explained before use; larger differences fail. Because the sigmoid is computed in bfloat16, the step between possible scores depends on the score: about 0.0156 at 3 and above, 0.0078 between 2 and 3 and 0.0039 between 1.5 and 2, so 0.016 is one step for high scores and up to four for low ones. The model scores the last position of the sequence, so an endpoint that batches inputs must pad on the left. The returned reference scores also let the platform call the endpoint itself and compare independently.

**Result.** The operator ran the check on 2026-10-28 at 16:38 UTC and every criterion was met. All 11 files equal those of e953da5. The official scoring code and the endpoint gave the same 26 scores, to the last bit, in all three passes. The two longest probes have 7,931 and 13,385 tokens, so both are cut to their last 4,096 tokens, which are the same text; both receive 1.53515625, as left truncation implies. The 26 scores range from 1.03 to 4.89. The operator's output, with the machine's host name and the endpoint address left out, is in the audit bundle.

### 5.3 Anchor on the authors' released outputs

The authors released the outputs of five models whose scores are in the paper's Table 4, already in the form the reward model reads. Scoring these 5 × 8,032 inputs with the deployed model and aggregating them as in Section 4.2 must reproduce the 60 metric cells and 15 dimension averages of those rows; no generation is involved, so this isolates the reward model and the aggregation. The criteria were fixed on 2026-10-28, before any measurement:

1.  **Reproduced:** every one of the 75 cells within 0.005 of the printed value.
2.  **Within one step:** every cell within 0.02 but some beyond 0.005; the cells are listed and the cause (hardware, kernels) is examined before platform runs open.
3.  **Not reproduced:** any cell beyond 0.02; platform runs stay closed while the checkpoint, input format and truncation are examined.

The printed values are rounded to three decimals (±0.0005). A hardware difference that moves some single scores by one bfloat16 step (at most about 0.0156) moves a mean over 308–744 examples by much less than 0.005. The report will also give the distribution of the differences, their mean sign, and whether the five models keep the order of the paper's Table 4 on character consistency.

**Result (2026-10-28, 19:05 UTC): within one step.** The 26 probes, sent first through the endpoint the platform itself calls, returned the official reference scores exactly. The 75 cells were then compared (Table 7): the largest difference is 0.0120; 8 cells are within 0.0005, 18 within 0.001, 58 within 0.005 and all 75 within 0.02. The mean signed difference is +0.0003, and the 15 dimension averages are all within 0.005. The five models keep the paper's order on character consistency (Baichuan2-7B > InternLM-7B > XVERSE-7B > ChatGLM3-6B > Qwen-7B), and in every one of the 15 columns every pair of models keeps its order (150 of 150). 17 cells exceed 0.005, so under the criteria the result is not a reproduction, and platform runs remain closed while the cause is examined.

| Metric | ChatGLM3-6B | Baichuan2-7B | XVERSE-7B | InternLM-7B | Qwen-7B |
| --- | --- | --- | --- | --- | --- |
| Flu. | **3.281 (+0.0120)** | **3.545 (−0.0056)** | **3.403 (+0.0102)** | 3.531 (+0.0042) | 3.189 (+0.0019) |
| Coh. | 3.648 (+0.0011) | 3.892 (−0.0016) | 3.750 (−0.0022) | 3.825 (+0.0024) | 3.563 (−0.0008) |
| Cons. | 3.278 (−0.0047) | 3.828 (+0.0008) | 3.518 (+0.0001) | 3.748 (+0.0041) | **3.241 (+0.0119)** |
| *CA avg.* | 3.402 (+0.0034) | 3.755 (−0.0018) | 3.557 (+0.0030) | 3.702 (+0.0036) | 3.331 (+0.0040) |
| KE | 2.021 (+0.0045) | 1.817 (+0.0038) | 1.830 (−0.0042) | 1.781 (−0.0005) | 1.960 (+0.0038) |
| KA | **2.785 (−0.0072)** | **2.840 (−0.0087)** | **2.780 (+0.0063)** | 2.802 (+0.0016) | 2.728 (+0.0004) |
| KH | **2.713 (+0.0086)** | 2.930 (+0.0007) | 2.759 (−0.0038) | **2.787 (+0.0063)** | **2.622 (−0.0109)** |
| PB | **2.462 (+0.0066)** | **2.820 (−0.0099)** | **2.569 (+0.0054)** | 2.717 (−0.0025) | 2.609 (+0.0039) |
| PU | 2.813 (+0.0007) | 3.079 (−0.0022) | 2.887 (+0.0001) | 3.013 (−0.0029) | **2.774 (−0.0059)** |
| *CC avg.* | 2.558 (+0.0024) | 2.697 (−0.0029) | 2.565 (+0.0011) | 2.620 (+0.0000) | 2.539 (−0.0013) |
| HL | **3.071 (+0.0067)** | 3.667 (−0.0032) | 3.391 (−0.0041) | 3.544 (−0.0019) | 3.031 (−0.0048) |
| CS | 2.928 (−0.0044) | 2.729 (+0.0006) | 2.740 (−0.0030) | 2.625 (+0.0027) | **2.786 (−0.0054)** |
| ED | 1.973 (+0.0043) | 2.114 (−0.0007) | 2.011 (−0.0018) | 2.070 (−0.0002) | 2.057 (+0.0046) |
| Emp. | **2.987 (−0.0064)** | 2.984 (+0.0000) | 2.936 (+0.0005) | 2.899 (+0.0021) | 2.837 (−0.0009) |
| *RA avg.* | 2.740 (+0.0005) | 2.873 (−0.0005) | 2.770 (−0.0023) | 2.784 (+0.0004) | 2.678 (−0.0014) |

**Table 7.** Anchor: the platform's scores of the authors' five released outputs, with the difference from the paper's Table 4 in brackets. Bold: beyond 0.005. Scored on 2026-10-28 through the deployed endpoint, aggregated as in Section 4.2 (results/anchor/ in the audit bundle).

**What has been ruled out.** The endpoint does not differ from the official scoring code (the probes are equal). The released outputs have not changed since they were uploaded on 2024-01-10, whose only later edit removed two annotation fields (`response`, `model`) and left every reply as it was; each file holds the 8,032 example–metric pairs of the metric assignment, with no duplicate or missing row. arXiv v1 and v2 print the same Table 4. Hugging Face holds a single version of the weights, uploaded on 2024-01-09; its later commit removed comments only. The differences are not systematic: their mean is close to zero, and the same metric moves up for some models and down for others.

**What remains.** The differences grow as the number of examples shrinks: they are largest for fluency (308 examples per model) and knowledge hallucination (486). Divided by its standard error, no difference exceeds 0.42, and their spread is 0.17 standard errors: as a statistical comparison, the paper's numbers and the platform's are the same. Read as noise on single scores, however, they imply a root-mean-square change of about 0.121 per score, far more than the one-bfloat16-step changes assumed when the criteria were set. Two explanations remain, and the released materials cannot separate them: the numerics of the authors' setup (GPU, PyTorch and cuBLAS versions) moved single scores by several steps; or Table 4 was computed on inputs or a checkpoint that differ slightly from those released (the paper's first version, with the same Table 4, predates both the released outputs and the released weights). A check fixed before any result separates them (matrix V07a): on 1,200 of the inputs, the operator scores the same weights with the same code in bfloat16, in bfloat16 with stricter cuBLAS accumulation, and in float32. If either variant moves single scores by a root-mean-square of at least 0.06, numerics suffice to explain the anchor and platform runs open; below 0.02, numerics are ruled out, the cause is put down to the inputs or checkpoint behind Table 4, and whether to open runs is decided and stated here.

**Result (2026-10-29, 11:19 UTC): numerics suffice.** The operator ran the check on an A800 GPU (PyTorch 2.7.0, CUDA 12.6, Transformers 4.36.2). With strict accumulation, 796 of the 1,200 scores change, by a root-mean-square of 0.095 (median change 0.023, largest 0.77); in float32 all 1,200 change, by a root-mean-square of 0.262. Both exceed 0.06. The operator's bfloat16 scores equal the platform's anchor scores for all 1,200 inputs, so the run measured the setup the platform calls. On our side, `charactereval_rm_numerics_verify.py` confirms that the 1,200 inputs are the ones the script selects from the repository and recomputes both values. A change of kernel alone thus moves single scores by the same order as the 0.121 the anchor implies, which is enough to produce Table 7 without assuming that Table 4 rests on other inputs or another checkpoint. It shows that numerics suffice, not that they are what differed in the authors' setup, which cannot be observed; the other explanation is not excluded, only no longer needed. By the criteria fixed in advance, numerics are taken as the cause and platform runs are open.

A check of the generation side against the paper would need one of the paper's models in the same version. The released outputs of these five models were generated with the released script’s message format, adapted per model as the README instructs, at unknown settings, so regenerating them is not an exact anchor; it is not part of the opening criteria.

## 6 Reproduction verification

### 6.1 Verification matrix

Table 8 states, for every stage from the paper to a score, what is compared, the acceptance criterion fixed before the comparison, and the result. `tools/experiments/charactereval_spec_check.py` compares the platform's protocol module with a Python reference whose official functions are the repository's source text, pasted mechanically and checked against the repository on every run; it runs no model and calls no endpoint. Its 26 checks passed on 2026-10-28 against commit c3d44a6; the run, with its log, is in the audit bundle.

| Stage | Compared with | Criterion | Result | Status | Check |
| --- | --- | --- | --- | --- | --- |
| Sources | Repository commit; SHA-256 of get\_response.py, transform\_format.py, run\_char\_rm.py, compute\_score.py; SHA-256 of the LaTeX source | Equal to the audited files | Equal | Pass | S01–S02; bundle logs 00, 02 |
| Reference implementation | Its 7 verbatim blocks against the repository’s source text; the generated strings and profiles against a fresh extraction | Identical | Identical | Pass | S03–S06 |
| Data | SHA-256 of test set, metric assignment, profiles; counts | 4,564 / 4,531 / 8,032; 77 characters with profiles | Equal | Pass | V01 |
| Generation messages | All 4,531 message arrays, official functions against the platform | 0 differences | 0 of 4,531 differ | Pass | V03 |
| Reply post-processing | First-line cut and failure filter on 16 edge cases | Equal | Equal | Pass | V04 |
| Reward-model input | All 8,032 inputs built from a released frozen file, official functions against the platform; the reference transform reproduces that file row for row | 0 differences | 0 of 8,032 differ | Pass | V02 |
| Failure handling | A test endpoint returning moderation refusals and empty replies | Refusals dropped, empty replies scored | 2 dropped, 2 scored | Pass | V05; local run recorded in the check matrix; output not kept |
| Engine end to end | Requests the engine actually sent, on a local Worker with a mock endpoint (12 examples) | Byte-identical to the official logic; aggregation equal | 12 / 12 requests, 34 / 34 inputs; equal | Pass | local run; check script in the bundle (code/e2e), output not kept |
| Aggregation | Metric means and dimension averages, official logic against the platform, 5 score sets | Difference below 10⁻⁹ | 0 | Pass | V08 |
| Metric table | Names, dimension membership, keys | Equal | Equal | Pass | M01–M03 |
| Published results | The paper’s Table 4 re-extracted from the LaTeX source against the leaderboard data | Unchanged | Unchanged (240 cells) | Pass | bundle step 02 |
| Released files | Formats; reward-model training data against the test set; frozen outputs; example-weighted averages | Reported as found | 7,228 training rows, all from test examples; 5 × 8,032 outputs, none failed or empty; weighted averages off by up to 0.063 | Reported | bundle step 05 |
| Reward-model weights | 11 files of the served model against Hugging Face e953da5 | Every file equal | 11 of 11 equal (2026-10-28) | Pass | V06 A |
| Reward-model scoring | Endpoint against the official run\_char\_rm.py on 26 probes | Every |difference| ≤ 10⁻⁶; three passes identical | 26 of 26 equal, maximum difference 0; passes identical | Pass | V06 B, C |
| Paper anchor | Five released outputs scored and aggregated, against 75 cells of the paper’s Table 4 | All within 0.005 (Section 5.3) | 58 of 75 within 0.005, all within 0.012; 150 of 150 pairwise orders kept | Within one step; explained by numerics (V07a) | V07 |
| Numerical sensitivity | 1,200 inputs scored in bfloat16, bfloat16 with strict accumulation and float32 on the served weights | Root-mean-square change ≥ 0.06: numerics explain the anchor; < 0.02: ruled out | Strict accumulation 0.095, float32 0.262; bfloat16 equal to the platform on 1,200 of 1,200 (2026-10-29) | Sufficient | V07a |

**Table 8.** Reproduction verification matrix of protocol 1.0. Status as of 2026-10-29 (UTC).

### 6.2 What this report establishes, and what it does not

1.  **Is every setting traced to a source?** Yes: the 21 settings of Table 4 each name their source, and the six platform choices (examples generated, no per-model prompt adjustment, generation settings, reasoning content, which failures count as ERROR, ranking) say why.
2.  **Does the platform build what the official code builds?** Yes, for everything up to the reward model’s input: all 4,531 generation requests and all 8,032 reward-model inputs are byte-identical to the official functions, and the aggregation is identical.
3.  **Does the deployed reward model score as the authors’ does?** On the evidence of V06, yes: its files are the released ones, and on 26 probes it returns exactly what the authors’ scoring code returns on the same weights. On the authors’ released outputs it reproduces the paper’s published numbers within one step (V07): the same orders, cells within 0.012, but not all within the 0.005 fixed as reproduction. Numerics alone move single scores by the same order (V07a), so by the criteria set in advance they are taken as the cause.
4.  **Would a platform run reproduce the paper’s scores for the same model?** Only partly testable. The anchor covers the reward model and the aggregation on the authors’ own outputs; the generation side cannot be anchored exactly, because the paper’s settings and model versions are not given.
5.  **Does the reward model measure what people judge?** This report cannot say: the annotations behind the paper’s correlations were not released, and the reward model was trained on the test contexts.

## 7 Personality back-testing

The paper evaluates personality back-testing on 54 characters whose MBTI types it collected from personality-database.com, with a test administered “as implemented in” Wang et al. \[4\], and reports an accuracy per model. No commit of the repository and no file on its Hugging Face page contains the labels, the questions, the prompt or the scoring code; the authors undertook in an issue to release them and have not done so. 216 is the smallest denominator that fits all 14 reported accuracies (k/216 within rounding), which suggests per-letter accuracy over 54 characters × 4 letters; this is an inference. Protocol 1.0 therefore does not offer personality back-testing, and the original leaderboard shows the paper's values as printed. A reconstruction would not be comparable with the paper's column and would be labelled as such.

## 8 Reproducing this report

The checks behind Sections 2, 3, 5 and 6 are published as an audit bundle, [/audit/charactereval/2026-10-29/](https://sociabench.com/audit/charactereval/2026-10-29/), with the SHA-256 of every file: the scripts and the platform source as run, each step’s output and its log (command, start and end time in UTC, exit status, full output), the manifest of the reward-model probes, and the three independent reviews (in Chinese) on which Table 4 rests. It contains no test-set text, profile or model reply; four lines of one review that quoted the test set are replaced by a marker. With the bundle, the authors’ repository and the arXiv e-print, a third party can repeat the scripted checks:

Commands

git clone https://github.com/morecry/CharacterEval repo && git -C repo checkout c3d44a6
curl -L -o eprint.tar.gz https://arxiv.org/e-print/2401.01275v2 && mkdir eprint && tar xzf eprint.tar.gz -C eprint
python3 -I tools/experiments/charactereval\_spec\_check.py repo            # Section 6: 26 checks (needs node)
python3 -I tools/experiments/charactereval\_paper.py eprint               # Section 2.1: Table 4 from the LaTeX source
python3 -I tools/experiments/charactereval\_table\_checks.py eprint        # Section 2.2: the paper's own numbers
python3 -I tools/experiments/charactereval\_release\_checks.py repo eprint # Sections 2.2 and 5.1: the released files
python3 -I tools/experiments/charactereval\_rm\_probes.py repo probes.json # Section 5.2: the 26 probes (SHA-256 8ad3fbaa…c494)
python3 -I tools/experiments/charactereval\_anchor\_diagnostics.py repo results/anchor   # Section 5.3: from the published scores
python3 tools/experiments/charactereval\_rm\_server\_check.py --model-dir <BaichuanCharRM> --reference --api <endpoint>
python3 tools/experiments/charactereval\_rm\_numerics\_check.py --model-dir <BaichuanCharRM> --repo repo   # V07a, on a GPU
python3 -I tools/experiments/charactereval\_rm\_numerics\_verify.py repo results/charactereval\_rm\_numerics\_check.json results/anchor   # V07a, our side

The platform's generated files (`strings.gen.ts`, `profiles.gen.json`) are written by `tools/experiments/charactereval_reference.py generate repo`; the check fails if they differ from a fresh extraction. Some findings rest on the reviews alone and are not re-checked by a bundled script: that arXiv v1, v2 and the ACL version print the same tables (compared by script in a review, against the ACL PDF), that the repository’s leaderboard image shows the same numbers (compared by eye), that the weight files of commits e1b9f1e and e953da5 have equal hashes (read from Hugging Face), and the statements of the ACL appendix. The local end-to-end and fault-injection runs of Section 6 are recorded in the check matrix; their check scripts are in the bundle (`code/e2e/`), their outputs were not kept.

## 9 Third-party results

We searched arXiv, the ACL Anthology and the web on 2026-10-28 for later work that reports CharacterEval scores (queries for CharacterEval, CharacterRM and BaichuanCharRM, and for role-play papers of 2024–2026; queries, sources and what we verified are recorded in`charactereval_third_party_search.md` in the audit bundle). Eleven sources report numbers (Table 9); about fifteen more cite the benchmark or reuse its data without per-model scores. None is listed on the leaderboard, for three reasons. Most models are research checkpoints that were not released. The protocols differ from the paper and from each other: several papers do not name the scorer or the test set, one evaluates on a 2,564-example subset, one on all 8,032 example–metric pairs, and “overall” is defined in three ways (the mean of the three dimensions, approximately the mean of the twelve metrics, or not at all), although the paper defines none. And several papers reprint rows of the paper’s Table 4 instead of re-running them (one states this; others agree with it to the last printed digit). None of the sources found reports personality back-testing.

| Source | Scorer and test set as stated | Reported | Relation to the paper’s Table 4 |
| --- | --- | --- | --- |
| Ji et al., PCL, arXiv:2503.17662v2 (ACL Findings 2025) | CharacterRM; original split and an own split by character | 12 metrics, 3 averages | Qwen-7B and GPT-4 rows equal to the paper’s. Baichuan2-7B: the three conversational-ability metrics are each exactly 0.500 below the paper’s (3.051, 3.394, 3.327 against 3.551, 3.894, 3.827)† |
| Tang et al., ERABAL, arXiv:2409.14710v2 (2024) | Scorer and test-set size not stated | 12 metrics | Baichuan2-Chat-13B re-run, below the paper in every metric |
| Tang et al., RoleAD, arXiv:2402.10618v1 (v2 withdrawn) | Scorer not stated | 12 metrics, 3 averages | No overlapping model |
| Feng et al., arXiv:2502.16940v2 (2025) | BaichuanCharRM via OpenCompass; 4,564 examples | One score per model (undefined); 12 metrics in an appendix | No overlapping model |
| Zhang et al., OmniCharacter, arXiv:2505.20277v2 (2025) | Not stated | 12 metrics, 3 averages, an overall | Reprints 12 of the paper’s rows. Baichuan2-13B conversational ability printed 3.759 against the paper’s 3.795; InternLM-7B overall 2.983 where its three printed averages give 3.034† |
| Li et al., Index SLM Technical Report, arXiv:2607.09885v4 (2026) | Not stated; baselines “follow Tu et al. (2024)” | 3 averages, an overall | Reprints the paper’s 15 rows exactly, plus two rows (CharacterYuyan, CharacterYuyan-RLHF) attributed to the paper that are not in it† |
| Fang et al., AAM, arXiv:2505.23923v2 (2026) | Scorer not stated; 4,564 examples | 3 averages, an average | MiniMax row differs from the paper by up to 0.064 (conversational ability 3.72 against 3.784) |
| Ye et al., CPO, arXiv:2508.09074v1 (2025) | CharacterRM; 4,564 examples | 12 metrics, 3 averages, an overall (2 decimals) | MiniMax row equal to the paper at two decimals except one metric |
| Zhang et al., AdvRole, arXiv:2609.28609v1 (2026) | BaichuanCharRM; a 2,564-example subset (the other 2,000 used for training)† | 3 averages, an overall | No overlapping model; its GPT-4o and Claude-3.7-Sonnet rows equal those CPO reports on all 4,564 examples† |
| Xu et al., RAPO, arXiv:2606.27025v1 (2026) | “Specifically trained reward models”; test set not stated | 12 metrics and an average close to the mean of the 12 | No overlapping model |
| Peng and Chen, arXiv:2603.03915v2 (2026) | The original reward model; 8,032 items | Per metric and per dimension | No overlapping model |

**Table 9.** Later work reporting CharacterEval scores (search of 2026-10-28). Agreement with the paper is checked only for models that also appear in the paper’s Table 4. Items marked † were verified by us in the paper’s arXiv HTML; the others are as reported by our search and not re-checked.

## 10 Platform results and history

No platform run has been made yet: the reward model has passed Section 5.2, and the anchor of Section 5.3 came out within one step, a difference that numerics suffice to explain (V07a); runs are open. Platform results will be ranked among themselves, under the current protocol only, and the paper's 15 results stay on the original leaderboard. The paper's results come from versions of the models available before January 2024; a platform run of a model with the same name is not a controlled comparison with them.

### 10.1 Protocol history

1.  **1.0** (2026-10-28) is the first protocol. It was built after three independent reviews of the paper, the code and the released files (Section 3) and checked as in Section 6 before deployment; no run has used another version.

### 10.2 Report history

1.  **2026-10-26.** First version: the original benchmark and the published results, without platform runs.
2.  **2026-10-28.** The protocol, its sources and its verification; reward-model validation with criteria fixed before any measurement; the inconsistencies in the sources; related and third-party work; the audit bundle; the result of the reward model’s deployment check. Corrections to the earlier versions: the paper prints 90 dimension averages (its Table 4 and appendix GPT-4 table), not 120 as the first text of this date stated; for the back-testing values, 216 = 54 × 4 is the smallest denominator that fits them all, where the first version said nothing about the unit.
3.  **2026-10-29.** The result of the anchor (Section 5.3, Table 7: within one step), what it rules out and what remains, and the numerical-sensitivity check fixed before its result; a new audit bundle with the anchor’s scores. Tables 7 and 8 of the previous version are now Tables 8 and 9. Later the same day, the operator's numerical-sensitivity result (V07a, Section 5.3: numerics suffice to explain the anchor), and platform runs opened.

## 11 Limitations

1.  The reward model’s agreement with human ratings cannot be checked from the released materials, and it was trained on the test contexts (Section 2.2).
2.  The deployment check (V06) rests on the operator running the published script on the serving machine; the anchor (V07) reproduces the paper’s published numbers within one step but not within 0.005; numerics suffice to explain the difference (V07a), but the authors’ setup cannot be observed, so this is shown to be sufficient, not shown to be what happened (Section 5.3).
3.  Personality back-testing is not offered (Section 7), so platform results cover twelve of the thirteen metrics.
4.  The generation prompt follows the released ChatGLM3 script for every model; the paper’s unspecified adjustment for closed models is not reproduced.
5.  Generation settings are the connection's own; the paper does not give the ones it used, so the generation side cannot be anchored exactly.
6.  Reward-model scores depend on numeric precision; the deployment must run in bfloat16, the anchor of Section 5.3 bounds the combined effect of hardware and of any difference in the authors’ inputs at 0.012 per cell.
7.  Confidence intervals resample examples and ignore their clustering by character and dialogue (Section 4.5).
8.  No threshold on the share of examples dropped for moderation has been set; the number is published with each run.
9.  The published results cannot be checked independently beyond the five models whose outputs were released; the paper's evaluations predate January 2024.
10.  Ranking by character consistency is a platform choice grounded in the paper's text; the paper ranks no models overall.

## References

1.  Q. Tu, S. Fan, Z. Tian, R. Yan. CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation. arXiv:2401.01275 (v2, 9 January 2024). Published at ACL 2024 (2024.acl-long.638, pp. 11836–11850), where the author list has seven names. [https://arxiv.org/abs/2401.01275](https://arxiv.org/abs/2401.01275)
2.  morecry. CharacterEval: code, test set and reward-model training data. GitHub, commit c3d44a6 (27 May 2025, UTC). MIT licence. [https://github.com/morecry/CharacterEval](https://github.com/morecry/CharacterEval)
3.  morecry. BaichuanCharRM (CharacterRM checkpoint). Hugging Face, commit e953da5 (10 January 2024). [https://huggingface.co/morecry/BaichuanCharRM](https://huggingface.co/morecry/BaichuanCharRM)
4.  X. Wang, Y. Fei, Z. Leng, C. Li. Does Role-Playing Chatbots Capture the Character Personalities? Assessing Personality Traits for Role-Playing Chatbots. arXiv:2310.17976v1, 2023 (the version the paper cites; later versions are titled InCharacter and use psychological interviews). [https://arxiv.org/abs/2310.17976v1](https://arxiv.org/abs/2310.17976v1)
5.  Z. M. Wang, Z. Peng, H. Que, et al. RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models. arXiv:2310.00746, 2023. [https://arxiv.org/abs/2310.00746](https://arxiv.org/abs/2310.00746)
6.  J. Zhou, Y. Huang, B. Wen, et al. CharacterBench: Benchmarking Character Customization of Large Language Models. AAAI 2025. arXiv:2412.11912. [https://arxiv.org/abs/2412.11912](https://arxiv.org/abs/2412.11912)
7.  X. Wang, H. Wang, Y. Zhang, et al. CoSER: A Comprehensive Literary Dataset and Framework for Training and Evaluating LLM Role-Playing and Persona Simulation. arXiv:2502.09082 (v3), 2025. [https://arxiv.org/abs/2502.09082](https://arxiv.org/abs/2502.09082)
8.  L. Zheng, W.-L. Chiang, Y. Sheng, et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks. arXiv:2306.05685. [https://arxiv.org/abs/2306.05685](https://arxiv.org/abs/2306.05685)

## Appendix A: Official templates

The strings below are read from `strings.gen.ts`, which `charactereval_reference.py` extracts with Python’s parser from https://github.com/morecry/CharacterEval at commit `c3d44a6`; each Python expression is shown in braces. They are not retyped. The first user message is the only message that carries the profile; there is no system message.

First user message: profile and role-play instruction `get_response.py`

{role\_information}
现在请你扮演一个角色扮演专家。请你根据上述信息扮演{role}进行对话。
Assistant turn accepting the role `get_response.py`

好的！现在我来扮演{role}。

Assistant turn when the character speaks first `get_response.py`

好的！现在我来扮演{role}。我首先发话：{conversations\[0\]\['value'\]}

Another speaker’s line (user turn) `get_response.py`

{conversations\[i - 1\]\['from'\]}：{conversations\[i - 1\]\['value'\]}

The character’s line (assistant turn) `get_response.py`

{conversations\[i\]\['from'\]}：{conversations\[i\]\['value'\]}

Final user message `get_response.py`

{conversations\[-1\]\['from'\]}：{conversations\[-1\]\['value'\]}

Reward-model input `run_char_rm.py`

<RoleInfo>

{str(character\_profile\[example\['role'\]\])}

<Context>

{example\['context'\]}

<Response>

{example\['model\_output'\]}

<Dimension>

{example\['metric\_zh'\]}

Context lines are split at `"\n"` and the speaker at the first `：`; a failed generation is marked `ERROR`; a reply is cut at the first `"\n"`.

**转载声明** 转载或引用本站任何评测结果，请注明来源「SociaBench（AI 社交智能榜）」并附本站链接 [sociabench.com](https://sociabench.com)。

---

转载或引用本站任何评测结果，请注明来源「SociaBench（AI 社交智能榜）」并附本站链接：https://sociabench.com/zh-CN/b/charactereval/methodology