---
title: "CharacterEval: An Independent Reproduction Report — SociaBench"
headline: "CharacterEval: An Independent Reproduction Report"
description: "CharacterEval [1] evaluates Chinese role-playing conversational agents on 77 characters from Chinese novels and scripts. A model continues a multi-turn dialogu…"
canonical: "https://sociabench.com/b/charactereval/methodology"
markdown: "https://sociabench.com/b/charactereval/methodology.md"
type: "ScholarlyArticle"
entity: "https://sociabench.com/b/charactereval/methodology#report"
author: "SociaBench"
publisher: "SociaBench"
language: "en"
date_modified: "2026-10-26"
about: "https://sociabench.com/b/charactereval#benchmark"
leaderboard: "https://sociabench.com/b/charactereval"
is_based_on: "https://arxiv.org/abs/2401.01275"
keywords:
  - "CharacterEval"
  - "CharacterRM"
  - "BaichuanCharRM"
  - "Chinese role-playing benchmark"
  - "role-playing conversational agent"
  - "MBTI"
  - "ACL 2024"
  - "CharacterEval 复现"
  - "中文角色扮演评测"
audience: "Researchers citing CharacterEval results and teams evaluating Chinese role-playing agents."
provenance:
  - section: "1 The original benchmark"
    url: "https://sociabench.com/b/charactereval/methodology#original"
    source: "原始论文或发布方的内容与结论"
  - section: "2 Published results and their provenance"
    url: "https://sociabench.com/b/charactereval/methodology#published"
    source: "原始论文或发布方的内容与结论"
  - section: "3 Reproduction on this platform"
    url: "https://sociabench.com/b/charactereval/methodology#reproduction"
    source: "本平台的分析与计划"
  - section: "4 Limitations"
    url: "https://sociabench.com/b/charactereval/methodology#limitations"
    source: "本平台的分析与计划"
references:
  - text: "Q. Tu, S. Fan, Z. Tian, R. Yan. CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation. ACL 2024. arXiv:2401.01275 (v2, 9 January 2024)."
    url: "https://arxiv.org/abs/2401.01275"
  - text: "morecry. CharacterEval: code, test set and reward-model training data. GitHub, commit c3d44a6 (28 May 2025). MIT licence."
    url: "https://github.com/morecry/CharacterEval"
  - text: "morecry. BaichuanCharRM (CharacterRM checkpoint). Hugging Face, commit e953da5 (10 January 2024)."
    url: "https://huggingface.co/morecry/BaichuanCharRM"
license: "https://sociabench.com/data-terms"
cite_as: "SociaBench. CharacterEval: An Independent Reproduction Report. 2026-10-26. https://sociabench.com/b/charactereval/methodology"
---

SociaBench · Independent reproduction report

# CharacterEval: An Independent Reproduction Report

SociaBench

CharacterEval · Stage 1: published results · Last updated 2026-10-26 · [Leaderboard](https://sociabench.com/b/charactereval)

**Scope.** This platform is not affiliated with the authors of CharacterEval. Its CharacterEval leaderboard lists the results the authors published, read unchanged from the paper's LaTeX source; no model has yet been evaluated on CharacterEval here, because the benchmark's reward model is not served by any inference provider (Section 3).

Contents

1.  [Abstract](#abstract)
2.  [1 The original benchmark](#original)
3.  [2 Published results and their provenance](#published)
4.  [3 Reproduction plan](#reproduction)
5.  [4 Limitations](#limitations)
6.  [References](#refs)

## Abstract

CharacterEval \[1\] evaluates Chinese role-playing conversational agents on 77 characters from Chinese novels and scripts. A model continues a multi-turn dialogue as the character; the authors' reward model CharacterRM \[3\] scores twelve metrics on a 1–5 scale, and a thirteenth, personality back-testing, measures how well the model's answers to an MBTI questionnaire match the character's known type. The authors published results for 15 models, which this platform lists unchanged. The paper defines no overall score; the leaderboard ranks by character consistency, which the authors call the most crucial aspect, and states this on the board. The printed dimension averages agree with their metrics within rounding.

## 1 The original benchmark

### 1.1 Sources and data

| Source | Version | Licence |
| --- | --- | --- |
| Paper \[1\] | arXiv:2401.01275 v2 (9 Jan 2024); ACL 2024 | arXiv licence |
| Code, test set \[2\] | commit c3d44a6 (28 May 2025) | MIT |
| CharacterRM \[3\] | commit e953da5 (10 Jan 2024); Baichuan2-13B-Base backbone with a scoring head; custom code | None stated |

**Table 1.** Primary sources, pinned to the versions this report describes.

The data comprise 1,785 multi-turn dialogues (9.28 turns on average) with 77 characters; profiles come from Baidu Baike. Examples (character, context, response) are split into 6,811 for training (not released) and 4,564 for testing. Each test example is scored only on the subset of metrics annotators judged relevant to it (sparse evaluation); 33 test examples have none, so the official pipeline scores 4,531 examples, 8,032 example–metric pairs. The reward model's released training data (7,228 rows) are all drawn from the test contexts.

### 1.2 Metrics

| Metric | Definition | Dimension |
| --- | --- | --- |
| Flu. | Fluency | Conversational ability |
| Coh. | Coherency | Conversational ability |
| Cons. | Consistency | Conversational ability |
| KE | Knowledge exposure | Character consistency |
| KA | Knowledge accuracy | Character consistency |
| KH | Knowledge hallucination (higher = fewer) | Character consistency |
| PB | Persona behavior | Character consistency |
| PU | Persona utterance | Character consistency |
| HL | Human-likeness | Role-playing attractiveness |
| CS | Communication skills | Role-playing attractiveness |
| ED | Expression diversity | Role-playing attractiveness |
| Emp. | Empathy | Role-playing attractiveness |
| MBTI | Personality back-testing accuracy (0–1) | Personality back-testing |

**Table 2.** The thirteen metrics of CharacterEval in four dimensions.

### 1.3 Scoring

CharacterRM reads the character profile, the context, the response and the metric name, and outputs a value that the released code maps to 1–5 (output × 4 + 1); the score is deterministic. A metric's score is the mean over the examples assigned to it; the paper reports a mean per dimension. The reward model was trained on scores from 12 annotators; its overall Pearson correlation with human scores is 0.631, against at most 0.385 for GPT-4 with one to three demonstrations. Personality back-testing uses 54 characters whose MBTI types were collected from a public database; the paper reports accuracy without specifying whether it is computed per type or per letter.

## 2 Published results and their provenance

The leaderboard lists the 15 models of the paper's Table 4, extracted by a script (`tools/experiments/charactereval_paper.py`) from the LaTeX source, which records its SHA-256 and writes the same data module for the server and the worker. It is the same table as the leaderboard image in the authors' repository. CharacterGLM has no personality back-testing result. Each printed dimension average agrees with the mean of its printed metrics within rounding (0.001 on three-decimal values). arXiv v1, v2 and the ACL camera-ready version print the same numbers in every cell of this table; v2 only renamed BC-Character-Turbo to BC-NPC-Turbo.

Because the paper defines no overall score, the leaderboard ranks by the character-consistency average and labels the column accordingly; every other dimension and metric can be used for sorting. Personality back-testing is an accuracy and is shown as printed, outside the score charts. The published results carry no confidence intervals, generation settings or evaluation dates; GPT-4 and GPT-3.5 are `gpt-4` and `gpt-3.5-turbo-1106`.

## 3 Reproduction on this platform

The paper gives neither the generation prompt nor any generation setting, so platform runs follow the authors' released code: the model under test receives the messages `get_response.py` builds (the character profile and the role-play instruction as the first user message, the dialogue as alternating turns), only the first line of its reply is kept (`transform_format.py`), and BaichuanCharRM scores it once per assigned metric with the input `run_char_rm.py` builds, in bfloat16 as that script runs it. Each metric is the mean over its examples and each dimension the plain mean of its metrics, the reading that reproduces every printed average. A generation refused by the provider's moderation leaves the scores, as the official filter drops failed generations; an empty reply is scored as it is. The generation parameters are the connection's own and are recorded with every run.

Personality back-testing cannot be reproduced: neither the 54 MBTI labels, nor the questionnaire, the prompt or the scoring were released, and platform runs do not include it. Before platform runs are opened, the served reward model is checked against the released weights and on the authors' five released result files against the corresponding rows of Table 4. Platform results are ranked separately from the paper's.

## 4 Limitations

1.  The leaderboard reproduces the authors' numbers; without per-example outputs they cannot be checked independently.
2.  The evaluations were run before January 2024; current versions of the same products may score differently.
3.  Ranking by character consistency is this platform's choice, grounded in the authors' text; the paper itself ranks no models overall.

## References

1.  Q. Tu, S. Fan, Z. Tian, R. Yan. CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation. ACL 2024. arXiv:2401.01275 (v2, 9 January 2024). [https://arxiv.org/abs/2401.01275](https://arxiv.org/abs/2401.01275)
2.  morecry. CharacterEval: code, test set and reward-model training data. GitHub, commit c3d44a6 (28 May 2025). MIT licence. [https://github.com/morecry/CharacterEval](https://github.com/morecry/CharacterEval)
3.  morecry. BaichuanCharRM (CharacterRM checkpoint). Hugging Face, commit e953da5 (10 January 2024). [https://huggingface.co/morecry/BaichuanCharRM](https://huggingface.co/morecry/BaichuanCharRM)

**转载声明** 转载或引用本站任何评测结果，请注明来源「SociaBench（AI 社交智能榜）」并附本站链接 [sociabench.com](https://sociabench.com)。

---

转载或引用本站任何评测结果，请注明来源「SociaBench（AI 社交智能榜）」并附本站链接：https://sociabench.com/b/charactereval/methodology