Role-Play Bench: Independent Reproduction Report and Validation of the Official Judges

1 Introduction

Role-play evaluation lacks a single correct answer: many responses can be appropriate for a character, but responses that contradict the character, the world or the dialogue history can be identified with comparatively high agreement. MiniMax built Role-Play Bench on this observation, describing misalignment as "surprisingly objective" while alignment is subjective [1]. The benchmark evaluates long, multi-turn trajectories because many failures, such as repetition, forgetting and drifting length, only appear after the first 20 turns [1, 2].

This document has four purposes:

1. to describe the original benchmark strictly as specified by its authors (Section 2);
2. to describe our implementation, separating what is verified against the sources, what we inferred or assumed, and what we changed, with the version and time of every change (Section 3);
3. to report checks of the official judges on the data available, with the size of each check (Section 4);
4. to list deviations, limitations and the configuration needed to reproduce our numbers (Sections 5–7 and the appendices).

Throughout, original refers to MiniMax's method as published, implementation to our reproduction of it, and change to a departure we introduced deliberately. The aim is to restore the original method as closely as the published materials allow; the report does not claim that the implementation is identical to the original or more accurate than it, since the original judge is unpublished. Where the two disagree, it reports what reading the dialogues showed, case by case.

1.1 How the implementation was built

Reproducing a benchmark whose judge and scoring rule are unpublished requires decisions that the sources do not dictate. We made them by a fixed procedure, so that each component can be traced to a source, to evidence in the released data, or to a stated assumption:

1. Source audit. Every statement about the method was extracted from the three primary sources and classified as specified, inferable from the released data, or unspecified (Sections 2.1–2.6).
2. Reconstruction from released data. Where the release constrains a component, the reconstruction follows it: the pass/fail rule for Basics and Logic follows the granularity of the released scores (Section 2.5), and the simulated user's turn lengths follow the user side of the released dialogues (Section 3.3). Components the release does not constrain are documented assumptions (Table 4, Section 5).
3. Comparison on released dialogues. The official judges score MiniMax's released dialogues, so their verdicts can be compared with MiniMax's on identical text (Section 4.1).
4. Checks on complete runs. Agreement between the two judges, discrimination between models and the stability of their order are measured on the complete full-protocol runs generated here (Sections 4.2–4.4).
5. Reading the disagreements. Where our verdicts and MiniMax's diverge, dialogues are read and each disputed failure is classified as genuine or spurious (Section 4.1).
6. Versioning and disclosure. The protocol is versioned; each run stores a snapshot of it and is assigned the version in force when it was created. The full protocol, including every prompt, is public (Sections 3.10 and 7, Appendices A–B).

1.2 What the evidence supports

Following argument-based validation [19, 20], we state the inferences that lead from a dialogue to a model's position under this implementation and the evidence for each (Figure 2, Section 4). In brief: recorded failures were genuine in the samples we read; on 44 released dialogues the official judges' model-level verdicts resemble MiniMax's, with differences in Basics and Logic that are only partly explained; on the two complete runs the two judges agree moderately, each orders the two models the same way, and the order holds in both languages. Not yet supported: that the order of many models under our generation protocol is stable, that scores reflect the experience of human users, and that the simulated-user protocol reproduces the original orderings.

1.3 What is disclosed

Every step from a dialogue to a score is open to inspection. Table 1 compares what the original release and this implementation disclose; disclosure makes a score checkable, not correct. The main properties are these.

1. Disclosed judges. The official panel consists of two reasoning models from different developers, two reasoning models, with pinned model identifiers. Each scores every segment independently, the scores are averaged, and disagreements beyond fixed thresholds are flagged, not resolved. Both developers also have models among those evaluated, and LLM judges can prefer their own family's outputs [21, 7]; an exploratory check found no significant family effect but cannot exclude one of several points (Section 6).
2. Judges detect; a published rule scores. Judges never output a score. They list failures, each with its round, a verbatim quotation and a reason, and a deterministic, published rule turns the list into scores (Section 3.6). Holistic ratings by LLM judges agree poorly with humans in role-play and tend to exceed human ratings [5, 12, 9]; checklist-style detection agrees better [12]. Every deduction in a run on this platform is stored with the quotation the judge gave; quotations are not yet checked automatically against the dialogue.
3. The whole dialogue is in view. Judges read the complete preceding dialogue with each segment, so forgetting and contradictions across 100 turns are detectable, not only errors within a 20-turn window.
4. Controlled generation conditions. Every model meets a simulated user with the same persona and the same sequence of turn-length budgets per scenario and sample. Since v6.0.2 (2026-10-26 13:19:51 UTC) the simulator model is also fixed per scenario and sample, so every model meets the same simulator in the same slot; runs created earlier chose it by a hash of the session id, so two models could meet different simulator models on the same scenario (Section 3.3). Every run eligible under the current version uses the fixed assignment. Each model is served by one pinned host with a sufficient output limit, and its reasoning setting is labelled.
5. Reported uncertainty and corrections. Every score carries a bootstrap confidence interval; small samples are labelled and kept off the leaderboard; checks are reported with their sample sizes, including negative results and corrections (Section 4). Where our judges and the original disagree, we read the dialogues: on the three models ranked lowest on MiniMax's leaderboard, 13 of 14 sampled failures that only our judges recorded were genuine (Section 4.1).

These properties make the scores checkable; they do not by themselves make every score correct. The checks have also found open issues, reported alongside the strengths: a classification of viewpoint switches that differs from the original's (Section 4.1), differences in Basics verdicts that have not yet been examined, and weak agreement between the two judges on Basics per segment (Sections 4.2, 6).

2 The original benchmark

2.1 Sources

Three primary sources describe Role-Play Bench: the MiniMax-M2-her technical blog post [1], the dataset card and files on Hugging Face [3], and Appendix A with Tables 7–9 of the HER paper [2], a joint work of Fudan University and MiniMax that follows the benchmark's "official protocol" as an external evaluation. The HER paper does not describe the judge or the scoring procedure; its evaluation details (judge model, decoding settings, a severity-based deduction formula and human-agreement figures) refer to the CoSER benchmark, not to Role-Play Bench, and we do not attribute them to it.

2.2 Design

• Negative evaluation. The benchmark detects failures ("misalignment") instead of defining ideal answers [1, 3].
• Scenarios. 100 character (NPC) settings per language were sampled uniformly from a structured taxonomy built over an internal library of more than one million NPC and user prompts, and the data team wrote a scenario for each [1]. The HER paper refers to "an internal dataset of 100 dialogue prompts" [2]. The public release is described as a partial subset [3].
• Dialogue generation. Model-on-model self-play: the evaluated model plays both the character and the user; external models received user prompts refined on real online user behaviour. Each scenario is run for 100 turns, three times [1, 3]. The authors report that a fixed user model produced results statistically similar to self-play, without giving the test or the numbers [1].
• Judging. Only the character side is scored. Dialogues are split into 20-turn chunks; the scoring stack includes consistency checks across samples and manual calibration against human preference data [1, 3]. The judge model, its prompts and calibration statistics are not published.

2.3 Failure taxonomy

Table 2 lists the sub-dimensions as defined in Table 9 of [2], grouped as in its Table 7. The blog post and the dataset card additionally describe a Knowledge dimension (adherence to the world's physical or magical rules) and formatting glitches under Basics [1, 3]; neither appears as a separate sub-dimension in Table 9. The released score files contain six dimensions: basics, logic, knowledge, diversity, content logic and interaction [3].

The assignment of the five Worlds sub-dimensions to Basics and Logic in Table 2 follows the prose of [2] (Appendix A) and of the blog post; the paper itself does not state it.

2.4 Scoring as published

The overall score is 0.5 × Worlds + 0.25 × Stories + 0.25 × Preferences (Eq. 3 of [2]). Stories and Preferences are the means of their sub-dimensions (caption of Table 7 of [2]). The sources do not specify how detected failures become scores, how the Worlds score is formed, how scores are aggregated over chunks, runs, scenarios and languages, or how the reported 95% confidence intervals are computed.

2.5 Released data

The release (revision 3c1be2a5) contains, per language, the scenario seeds, the dialogues and session-level scores: 50 Chinese and 45 English scenarios, 11 models, three runs per scenario, 3,135 dialogues of 100 turns, and one 0–100 score per dimension per session [3]. The card states that the dialogues were synthetically constructed by the data team, intentionally skewed toward chat-oriented role-play, and that user-side fields were varied in parts of the data. The licence is stated inconsistently: the metadata header says Apache-2.0, while the card's licence section says CC BY 4.0 / CC BY-SA 4.0.

Two properties of the released scores matter for reproduction. First, about 97% of the basics and logic session scores are multiples of 20, which is consistent with a pass/fail decision per 20-turn chunk over five chunks; the other dimensions take graded values. This is our inference from the data, not a documented rule. Second, the published leaderboard cannot be recomputed from the released scores: with the published group weights, the released sessions give MiniMax-M2-her about 87 instead of the published 84.65, and the order of two Claude variants reverses (Table 3). The published values were presumably computed on the full, partly unreleased scenario set or with an unpublished aggregation.

2.6 Unspecified components

The following are required for an exact reproduction and are not specified in the sources:

1. the 5 unreleased scenarios per language and how the released ones were selected;
2. the user-simulation prompts, persona handling, length control and decoding settings, and which model plays the user for external models;
3. the decoding settings of evaluated models (temperature, maximum output length, reasoning budget);
4. the judge model(s), prompts, decoding settings and the context given with each chunk;
5. the mapping from failures to scores (per chunk or per turn; counts, pass/fail or severity) and the formation of the Worlds score;
6. aggregation over chunks, runs, scenarios and languages, and the confidence-interval method;
7. the human-calibration data, procedure and agreement statistics.

2.7 Related work and its bearing on this implementation

Role-play benchmarks. Earlier benchmarks score single responses or short, pre-written contexts against character profiles [4, 5, 6]. Multi-turn and simulation-based designs followed: RMTBench uses pre-generated user turns over long dialogues [8], and CoSER simulates multi-character scenes and has critics count flaws with severity penalties [7]. Role-Play Bench differs by generating 100-turn interactive trajectories and judging only the character side for predefined failures; a survey places these approaches in context [23].

LLM judges. Prompted general-purpose judges correlate only moderately with human ratings of role-play: in CharacterEval, GPT-4 reached Pearson r ≈ 0.36–0.39 against 0.63 for a reward model trained on human labels [5]; in HumanLLM, holistic judgements correlated at r = 0.43–0.61 while checklist judgements reached 0.90 [12]. Judges tend to rate agents higher than humans do, particularly agents trained for social interaction and late in long interactions [9, 10, 11], and can prefer outputs of their own model family [7]. Fine-grained detection of out-of-character content finds failures that response-level scores miss [13]. These results support failure-based (checklist-like) judging, which is what Role-Play Bench uses, and they motivate our comparison with the original judges' verdicts (Section 4.1) and our reporting of a self-preference risk (Section 6).

User simulation. Default LLM user simulators are reported to be overly cooperative and consistent; replacing the simulator can reorder model rankings, and simulators can be validated against real-user traces with discriminators or distributional measures [14, 18]. Simulated personas drift over long dialogues [15], instruction-tuned models are argued to be structurally poor simulators of people [16], and the composition of a persona pool biases outcomes [17]. Our simulator (Section 3.3) addresses length and coupling, which we can measure from the released dialogues, but it has not been validated against real users in the sense of these works; we list this as a limitation and as planned work.

3 Our implementation (protocol v6.0.6)

The evaluation standard is fixed per benchmark and versioned; every run stores a snapshot of the protocol, prompts, judge configuration and prices in force when it started, so scores remain attributable after later changes (Section 3.10, Appendix B). Table 4 classifies each component by how closely it follows the original.

3.1 Data and run configurations

We use the released seeds at revision 3c1be2a5 unchanged. A full-protocol run covers all 95 scenarios (50 Chinese, 45 English) × 3 independent samples × 100 turns = 285 sessions, chunked into 20-turn segments. Smaller configurations exist for development; they are never eligible for the leaderboard, are judged by other models than the official panel, and none of their results is used in this report.

3.2 Dialogue generation

Each session starts from the seed's prologue (character turn, round 0) and initial user input (round 1); the character speaks at even rounds. The evaluated model receives a system prompt rendered from the seed (character and user settings; Appendix A) and the dialogue history; it receives no instruction about response length. Length is treated as part of what is measured: the only limit is the maximum output below, so a model aligned to answer briefly like a person in chat is neither corrected nor penalised for brevity, and a verbose one is not rewarded for length; the judges count failures, not words. Requests set a maximum output of at least 8192 tokens because reasoning tokens count toward the limit and a 1,024-token limit truncated 61% of one model's replies in an earlier version; the finish reason of every reply is stored and truncation rates are reported. Reasoning ("thinking") settings are passed explicitly and the run name carries them, so the same model with reasoning on and off is ranked separately. A reply blocked by the provider's content moderation is recorded as no response and judged as AI silence.

3.3 User simulator (change)

The original uses self-play. We use a fixed user simulator (two models, reasoning effort provider default) for two reasons: in self-play the user side inherits the evaluated model's verbosity, so user turns grow with the model's replies (in an early self-play run one model reached 2,401 words per user turn); and the score then mixes the model's quality as a character with its behaviour as a user.

The simulator plays a persona fixed per scenario and sample. Its length per turn is drawn from a distribution of real user messages, independently of the character's length; the reference is the user side of MiniMax-M2-her's released dialogues, which the blog describes as reproducing real user habits [1] (Chinese median 9 characters, English median 5 words). A share of turns are action-only. A reply longer than twice its budget is rewritten once. Self-play and the earlier simulator remain available as experimental controls; such runs are never eligible for the leaderboard.

Which simulator model plays the user (v6.0.2, from 2026-10-26 13:19:51 UTC). The simulator model is fixed by scenario and sample: a language's scenarios in id order, three samples each, alternate between the two models. Every model under test therefore meets the same simulator in the same slot, each simulator plays half of each language (75 and 75 Chinese sessions, 68 and 67 English), and every scenario meets both. Before this, a hash of the session id chose the model; in a full run of 285 sessions that gave one simulator 85 of the 150 Chinese sessions, and the assignment differed between models on the same scenario, which adds noise to comparisons between models. Runs created earlier keep their assignment, so resumed sessions keep their simulator; among the full-protocol runs, only Doubao-Seed-Character run 1547acce31 (created 2026-10-26 09:20:32 UTC) uses the earlier assignment, and it is not eligible under the current version.

Moderation on the user side (v6.0.1, from 2026-10-26 11:41:25 UTC). When the provider's content moderation blocks the simulated user, the same model cannot produce the turn on retry. That one turn is written by the other simulator model, the switch is logged and the message records which model wrote it; the next turn returns to the session's simulator. In the complete runs this happened for 2 of 14,250 user turns of run 1547acce31, both in one session. The rule is the same for every model under test and changes nothing on the side of the evaluated model or the judges.

3.4 Generation conditions (change)

Models served through a router can be routed to different hosts with different quantization and default sampling settings within one run, so the dialogues the judges see can depend on the host. A run is therefore eligible for the leaderboard only if the evaluated model is pinned to a single host with fallbacks disabled and the stored replies confirm a single host; direct provider endpoints satisfy this by construction. Temperature is the endpoint's configured value (provider default unless set) and is recorded per run.

3.5 Judges (reconstruction)

Judges read the scenario setting, the whole preceding dialogue (undefined turns) and one 20-turn segment, and return a list of failures with round, type, quoted evidence and reason. The official panel (Table 5) scores every full-protocol run and is the only panel whose results enter the leaderboard or this report's evidence.

When two judges score the same dimension, their scores are averaged with the stated weights. A session is flagged as disputed when the judges differ by at least 15 points on a dimension (for pass/fail dimensions, when they disagree on at least two segments), and a segment when they differ by at least 30; disputes are reported, not resolved.

Judging mode and failures. Segments are sent through the provider's batch interface or directly; both send the same prompt to the same model with the same settings, and the mode is recorded. A batch that the provider lets expire is judged directly (v6.0.4); a direct request whose output is unusable is retried once, and a second unusable output is final (v6.0.6). If one judge's provider refuses a segment through content moderation, that judge's score for the segment is taken from the other official judge (v6.0.5); the refusal and the substitution are recorded and listed on the session page. A reply cut off at the output limit is judged again with twice the limit (v6.0.3). In the two complete runs the largest judge outputs were 7,764 tokens (Claude Opus 5.5) and 4,737 tokens (GPT-6.1 Sol); in the run of qwen-plus-character, still in progress, Claude Opus 5.5 reached 13,290 tokens, below its limit of 16,000.

3.6 Scoring and aggregation (reconstruction)

1. Seventeen failure types are detected: the fifteen of Table 2, a world-rule violation for Knowledge, and formatting glitches under Basics (both described in [1] and present as released score columns).
2. Per segment, each sub-dimension scores 100 − 10 × (number of character turns showing the failure), floored at 0. Basics and Logic are pass/fail per segment (0 if any of their sub-dimensions occurs, otherwise 100), following the inference in Section 2.5. Knowledge is its single sub-dimension; Diversity, Content logic and Interaction are means of their sub-dimensions.
3. A session's dimension score is the mean over its segments.
4. Worlds = mean of Basics, Logic and Knowledge (our assumption; unspecified in the sources). Stories = (4 × Diversity + 2 × Content logic) / 6, which equals the mean of its six sub-dimensions as in [2]. Preferences = Interaction. Overall = 0.5 Worlds + 0.25 Stories + 0.25 Preferences.
5. A model's score is the mean over sessions, Chinese and English pooled. The 95% confidence interval shown on the leaderboard is a percentile bootstrap over sessions (1,000 resamples; reported from 5 sessions upward). Because the three samples of a scenario are correlated, this interval is too narrow: on the two complete runs, resampling scenarios instead of sessions widens it by 25–39% (Section 4.3).

3.7 Leaderboard eligibility

A run enters the leaderboard only if it uses the full protocol, the current protocol version, has every session scored, and was generated by a single host (Section 3.4). A run's version is the patch version in force when it was created (Section 3.10); under v6.0.6 only runs created from 2026-10-27 13:36:15 UTC are eligible. The leaderboard ranks only these runs. MiniMax's published leaderboard is shown beside it as the original data and is not ranked with them: its scores come from MiniMax's unpublished judge and self-play dialogues and cannot be compared with scores generated here. The scores recomputed from MiniMax's released sessions (Table 3) are used only in this report.

Full-protocol runs on the leaderboard as of 2026-10-27 22:14 UTC, all created under v6.0.6: minimax/minimax-m2-her (run 154e0c2dc7, created 20:09:55, completed 22:12:31), deepseek-v4-1-flash-261017 (44e958def4, 13:38:04 – 15:22:41), Doubao-Seed-Character (4cfe3e36f8, 20:20:36 – 22:13:11) and qwen-plus-character (cd542f9e50, 13:37:38 – 22:11:06). The earlier Doubao-Seed-Character run 1547acce31 (created 2026-10-26 09:20:32, version v6.0, overall 88.81) is complete but not eligible. Section 4 analyses the runs complete at 20:00 UTC only; the leaderboard is authoritative for scores.

3.8 Supplementary experience-quality score (change, not part of the overall)

Because the failure-based score is close to its ceiling on several dimensions (Knowledge, Content logic and Interaction average 94–100 in the complete runs; Section 4.2), runs created since 2026-10-25 14:28:53 UTC also receive an experience-quality score. One judge (a reasoning model) rates each 20-turn segment from 1 to 10 on five anchored scales (characterization, expression, narrative, responsiveness, naturalness); the mean is rescaled to 0–100. It is reported beside the overall score and does not affect the overall or the ranking. Its agreement with other judges and with human judgement has not been established on the runs reported here.

3.9 Session rule: replies in the wrong language (change, v6.0.2)

A session in which the evaluated model mostly answers in the other language of the scenario scores 0 on every dimension. A reply counts as in the wrong language when more than half of its letters belong to the other script (CJK characters against Latin letters), so names, loanwords and short quotations do not count; the session scores 0 when at least half of its replies from round 2 on do. The judges' own scores are kept beside the 0 and shown on the session page. The rule is part of this benchmark's standard and applies to every model, including the 11 models of the public release. It took effect with the Worker version created 2026-10-26 13:19:51 UTC; the re-scoring of all stored results finished at 16:19:18 UTC.

Why. In the full run of Doubao-Seed-Character (1547acce31), three English sessions were answered in Chinese from the first reply on, and the simulated user followed into Chinese from its first turn; the dialogue the judges saw was a Chinese role-play, and the failure to keep the scenario's language would otherwise cost only part of the Basics score. A fourth session was answered in Chinese in 32 of its 50 replies. The public release shows the same pattern: four English sessions of doubao-1.5-pro answered at least half in Chinese, and MiniMax's scores for them average 86.0, close to the model's mean, so the original method did not treat it as a failure of the session.

Effect. On the public release the rule zeroes those 4 sessions and no others (at a threshold of 20% instead of 50% it would zero 7 sessions of three models): doubao-1.5-pro falls from 87.11 to 85.90 and keeps its rank, and the gap to each model below it shrinks by 1.21 points. On run 1547acce31 it zeroes 4 sessions: 90.02 → 88.81. Before the rule, a scenario-paired sign-flip test with Holm correction separated 41 of the 55 model pairs in MiniMax's released scores; after it, 34 (Section 4.6). The seven pairs that lost significance are doubao-1.5-pro against GPT-5.1, Claude Opus 4.5 (high and low reasoning), Gemini 3 Pro and Claude Sonnet 4.5, and two pairs whose own differences are unchanged but whose Holm-adjusted p rose from 0.048 to 0.063 (MiniMax-M2-her against GPT-5.1, DeepSeek-V3.1 against Grok 4.1). None of the 41 earlier orderings is reversed (anchor_check.py, in the audit bundle).

3.10 Versions and their traceability

Protocol v6.0 was first deployed in the Worker version created 2026-10-24 19:59:48 UTC, under the label "v6.0 · 2026-10-25" (a date in UTC+8). Between 2026-10-26 11:41 and 2026-10-27 13:36 UTC, further changes were deployed in six steps without changing that label (Appendix B). One of them, the wrong-language rule, changed stored scores; one changed how new runs assign the simulator; the others change how a moderated turn is handled and how judging is carried out, and leave the scoring of a completed session unchanged. Deploying them under one label was an error in version management: two runs labelled v6.0 could differ in simulator assignment and judging procedure without the label showing it.

We corrected this on 2026-10-27 by registering the changes as patch versions v6.0.1–v6.0.6. Each run's version is derived from the label stored in its snapshot and its creation time: the latest patch whose start time is not after the run's creation. Start times are the creation times of the first Worker version containing each change, taken from the platform's version list; deployment records are kept only for the ten most recent deployments (from 2026-10-27 11:56 UTC), so for earlier patches the moment the change went live cannot be confirmed beyond the version's creation time. The start of v6.0.6 is its recorded deployment time. Runs of different patch versions are not ranked together; only the current version is eligible (Section 3.7). From v6.0.6 on, a change that can affect scores raises the version before it is deployed. Evidence: the audit bundle (/audit/minimax-rp/2026-10-27/: results/v6_changes.md, v6_timeline.csv, version_markers.csv).

4 Validation of the official judges

This section examines the official judge panel, the only panel whose scores enter the leaderboard, on two sources of evidence (Table 6).

1. Released dialogues. The panel re-scored 44 of MiniMax's released sessions: 11 models × 4 scenarios (zh_dialogue_014, zh_dialogue_023, en_dialogue_031, en_dialogue_043) × the first of the three released samples. It used the judge models, settings and prompts of the production runs (the judge prompts are identical, SHA-256 prefixes 9f6fd89243 and 4b8fe79a49). This run was made on the platform's development instance on 2026-10-24 19:30:58 UTC, and its data are kept there, not in the production database. Earlier versions of this report described these comparisons as covering "4 scenarios"; this is what the phrase meant: 4 scenarios per model, 44 sessions in all.
2. Complete platform runs. Full-protocol runs judged by the panel in production. As of 2026-10-27 20:00 UTC two were complete.

The two sources answer different questions, and neither is large. How much a run covers (95 scenarios × 3 samples) is not the scale at which the judges are validated: two complete runs give 570 sessions for measuring how the two judges agree with each other, but only two models for any statement about rankings, and the comparison with MiniMax rests on 4 scenarios. Statistics below are computed by the audit scripts from a read-only export of the production database, except where the development run is named; the scripts, their results and the frozen scores are in the audit bundle (/audit/minimax-rp/2026-10-27/), where the reliability results of Sections 4.2–4.4 are recomputed from the frozen scores alone and match byte for byte. Confidence intervals are 95% percentile bootstrap intervals [22].

Run 1547acce31 was created under v6.0 and differs from the current version in its simulator assignment (Section 3.3) and judge output limit (32,000 tokens; no output reached 8,000). Its sessions were judged by the same models with the same prompts, so it is used here to study the judges, though it is not eligible for the leaderboard.

Revision (2026-10-27). This version of the report removes the results of a single-rater blind comparison (29 pairs) and of the judges used for development runs, which are not part of the benchmark standard, together with the experiments that relied on them (self-play conditions, a rerun, segment-level horizon, alternative scoring rules and a candidate recalibration). It adds the checks on the complete runs, registers the patch versions of v6.0 (Section 3.10) and corrects statements that no longer matched the data: that no platform run was on the leaderboard, the number of orderings separated in the released data (41 before the wrong-language rule, 34 after), the simulator assignment, the judging mode and the protocol history.

4.1 Agreement with MiniMax’s released scores

We scored MiniMax's released dialogues with the official panel and compared model-level results with the released session scores of the same dialogues and with the published leaderboard (Table 7).

In what follows, groups of models refer only to positions on MiniMax's published Role-Play Bench leaderboard (Table 3): the eight models ranked 1–8 and the three ranked 9–11. The grouping describes results on this benchmark and implies no judgement of the models' capabilities in general.

The small mean difference hides a structured pattern (Figure 3, Table 8). For the models ranked 9–11, the panel's overall scores are 7.9 points lower than MiniMax's, almost entirely through Basics and Logic. For the models ranked 1–8 they are 1.2 points higher on average, again mainly through Basics and Logic, with wide variation between models (from −3.8 for doubao-1.5-pro to +9.7 for Gemini 2.5 Pro). Diversity is about 11 points lower than MiniMax's for the models ranked 1–8 and about equal for those ranked 9–11. With one sample of four scenarios per model, each model's difference carries several points of uncertainty.

Agreement with MiniMax is evidence of reproduction, not of correctness, because MiniMax's judge is itself unpublished. We therefore read the disagreements. Models ranked 9–11. From the Basics and Logic failures that the official panel recorded on DeepSeek-V3.1, DeepSeek-V3.2 and Grok 4.1 in these sessions (486 in total), we drew 14 at random and checked each against the dialogue. Thirteen were genuine: whole passages copied verbatim from earlier turns or from the user (7), prompt residue such as **rec:** and *output exactly:* leaking into the reply (1), contradictions of place and movement (3), and narration switching to the user's viewpoint (2). One, a reply ending mid-sentence in an explicit scene, may be intentional. Several sampled failures come from sessions to which MiniMax gave full Basics scores. The lower scores of these models on our panel therefore reflect failures that occur in their dialogues and that MiniMax's released scores do not reflect, and they do not change the order of the models. Models ranked 1–8. Here MiniMax records more Basics and Logic failures than our panel on several models. One dialogue was examined in full (Claude Opus 4.5 with high reasoning, scenario zh_dialogue_001, third sample; official panel, development instance, 2026-10-25 15:53 UTC). MiniMax failed Logic in 4 of 5 segments, our panel in 1. The character's narration switches without any marker to the user's viewpoint in 9 turns spread over all five segments: it refers to its own character in the third person and writes the user's lines and actions. Both judges recorded each of the 9 turns, but as "AI speaks for user" (Interaction, about one overall point) rather than as a Logic failure (a failed segment, about 13 overall points for this session). We consider MiniMax's verdict the more appropriate one here; the protocol has not been changed for it. The remaining differences, mainly in Basics, have not yet been examined against the dialogues.

4.2 Agreement between the two judges

On the two complete runs, we compare the two judges' scores of the same sessions and segments (Table 9). Sessions with a substituted segment (Section 3.5) are excluded, leaving 568 sessions and 2,840 segments.

Agreement is high for Diversity and Interaction, moderate for Logic and the overall score, and weak for Basics per segment: the judges give the same pass/fail verdict on 91% of segments, but mostly because most segments pass, and κ = 0.36 shows that they often disagree on which segments fail. Claude Opus 5.5 is the stricter judge, by 8.8 points on Basics and 7.5 on Logic. Knowledge, Content logic and Interaction are near their ceiling, so their correlations rest on few sessions. As recorded by the platform's dispute rule (Section 3.5), 96 of 285 sessions (33.7%) of Doubao-Seed-Character and 77 of 285 (27.0%) of deepseek-v4-1-flash-261017 are disputed, mostly on Logic and Basics.

On the 44 released dialogues, which span the whole range of MiniMax's leaderboard, the same judges agree more closely on session overall scores (Pearson r = 0.88, ICC 0.83, MAD 4.23; development instance). There, session scores spread widely (standard deviation 10.9 points for Claude Opus 5.5 and 9.1 for GPT-6.1 Sol); within the two platform runs they spread less (5.9 and 4.6), which lowers correlation-based measures, while the absolute differences are similar (MAD 4.23 and 3.82).

4.3 Discrimination and stability of the order

The two complete runs differ by 1.79 points; the scenario-paired 95% interval is [0.29, 3.83] and a sign-flip test gives p = 0.033. The intervals over scenarios are 25–39% wider than those over sessions and overlap. Resampling scenarios, deepseek-v4-1-flash-261017 ranks first in 99.1% of resamples; over random halves of the scenarios the order is the same in 96.8%. The order is the same in Chinese and in English, under each judge alone, and with either simulator model. The partial run of qwen-plus-character lies more than 9 points below both.

These results show that a difference of about 2 points between two models can be detected with a full run. They do not show that the order of many models is stable: with two models a rank correlation can only be +1 or −1, and closer pairs will need more evidence.

4.4 Generalisation across scenarios

Ranking the two complete models within each scenario (mean of its three samples), Kendall's coefficient of concordance is W = 0.025: deepseek-v4-1-flash-261017 is ahead on 55 of 95 scenarios and behind on 40. Including the partial run of qwen-plus-character†, W = 0.47 and 42 of 95 scenarios reproduce the overall order of the three. The model-level difference therefore rests on averaging over many scenarios; a single scenario, or a handful, says little about which of two close models is better, which is why only the full protocol is eligible for the leaderboard. The released seeds carry no scenario category, so language and simulator model are the only strata available (Section 4.3).

4.5 Realism of the simulated user

We compared the simulated user's turns with targets set beforehand from the user side of MiniMax-M2-her's released dialogues, using development runs of three evaluated models (MiniMax-M2-her, DeepSeek-V3.1, DeepSeek-V3.2; 10 scenarios each) with the v6.0 simulator. These statistics concern only the simulator's output, not any judge.

These statistics describe surface form only. Whether the simulated user elicits the same failures as real users has not been tested.

4.6 Preserving the orderings in the released data

Any judge, scoring or generation change intended to reproduce MiniMax's results is checked against the orderings that MiniMax's released scores establish. Reference: the released session scores (11 models × 95 scenarios × 3 samples), overall at the published weights, averaged per scenario; a pair counts as established if a scenario-paired sign-flip test survives Holm correction over all 55 pairs (α = 0.05). With the wrong-language rule applied to the released scores (Section 3.9) this holds for 34 pairs; before it, for 41.

For every established pair a candidate covers, on the scenarios they share, the candidate fails if it reverses the pair significantly, or reverses it with a gap significantly different from MiniMax's on the same scenarios (Holm over the candidate's pairs). It passes if the one-sided 95% lower bound of every pair's difference exceeds −1 point (half the smallest established gap, fixed before any candidate was scored). Otherwise the result is undecided. Gap retention is the sum of the candidate's gaps divided by the sum of MiniMax's on the same scenarios.

The panel reverses no established ordering significantly and records larger gaps than MiniMax's scores. Its four reversed point estimates are Claude Sonnet 4.5, doubao-1.5-pro and GPT-5.1 against Gemini 2.5 Pro, consistent with the Basics and Logic differences of Section 4.1, and MiniMax-M2-her against Gemini 3 Pro, a pair that MiniMax's own scores also reverse on these four scenarios (−5.04). A decision requires scoring about 30 released scenarios with the official panel (Section 8). Whether dialogues generated with the simulated user preserve these orderings has not been tested with the official panel.

5 Deviations and their status

6 Limitations

1. Scores produced here are not comparable with the published Role-Play Bench values: scenario set, user side, judge and scoring rule differ or are reconstructed.
2. The scoring rule for Basics and Logic is inferred from released score values; the Worlds composition is an assumption.
3. The judges are validated at a small scale. Agreement with MiniMax rests on 44 released sessions (4 scenarios × 11 models × 1 sample); agreement between the judges, discrimination and the stability of the order rest on two complete runs, that is, two models. Neither supports a claim about the ranking of many models, and the leaderboard should be read with this in mind.
4. On several of the models ranked 1–8 on MiniMax's published leaderboard, MiniMax records more Basics and Logic failures than our panel (Section 4.1). One cause is identified, a different classification of viewpoint switches; the protocol has not been changed for it, and the remaining Basics differences have not been examined against the dialogues.
5. Calibration targets MiniMax's own scores, which come from an unpublished judge without published human agreement and which do not reflect many low-level failures of the models ranked 9–11 (Section 4.1). Agreement with them is evidence of reproduction, not of correctness.
6. Whether recorded failures are genuine was checked by reading for one dialogue of a model ranked 1–8 and a random sample of 14 failures of the models ranked 9–11; there is no estimate of precision and recall against human labels, and no human evaluation of any kind is part of the evidence.
7. The two judges agree weakly on which segments fail Basics (κ = 0.36) and moderately on Logic (κ = 0.51); a quarter to a third of sessions are flagged as disputed. Model scores average both judges over 285 sessions, but single sessions are unreliable.
8. Scores of two close models are consistent across languages and judges but not across scenarios (Kendall's W = 0.025); the intervals shown on the leaderboard bootstrap sessions, and scenario-clustered intervals are 25–39% wider.
9. Knowledge, Content logic and Interaction are close to their ceiling for the models tested, so most of the differences between models come from Basics, Logic and Diversity.
10. Changes to v6.0 deployed in six steps between 2026-10-26 and 2026-10-27 carried no version change and were registered as patch versions afterwards (Section 3.10). Their start times before 2026-10-27 11:56 UTC are Worker version creation times; the corresponding deployment records were not retained.
11. The simulator model is one of two. Run 1547acce31 assigned it per session by hash; runs eligible under the current version assign it per scenario and sample. Simulator realism is assessed on surface statistics only; no comparison with real users' behaviour has been made.
12. Whether dialogues generated with the simulated user preserve the orderings established by MiniMax's released data has not been tested with the official panel; results generated here are not a reproduction of the original scores.
13. Judge and simulator models are proprietary and accessed through APIs. Model identifiers are pinned where providers allow it, but providers may change models behind an identifier.
14. The official judges come from two model families that are also evaluated; judges may prefer their own family's outputs [7]. An exploratory difference-in-differences on the 44 released sessions scored by the official panel (Opus − Sol on each developer's models against the other models) gave +1.6 [−1.7, +4.7] points for the Anthropic models and −1.4 [−6.1, +3.8] for the OpenAI model: compatible both with no effect and with effects of several points.
15. Chinese and English sessions are pooled; per-language results are reported but language effects are not modelled.

7 Reproducibility

The complete evaluation standard, including every prompt in Appendix A, is public and served by the endpoint GET /api/benchmarks/minimax-rp/protocol. Each run stores the snapshot it used; run reports list the hosts that served the evaluated model, finish reasons and truncation rates, per-turn token counts, costs, and every judge output with its evidence. The analyses of Section 4 are published as an audit bundle, /audit/minimax-rp/2026-10-27/, with the SHA-256 of every file: the audit scripts and their results, the anchoring check, the 44 development-instance sessions as scores, and the frozen production scores (no dialogue, no judge rationale, no endpoint address) from which Sections 4.2–4.4 can be recomputed without access to the platform. The inventory and the v6.0 timeline need the production database and the deployment history, which are not public; their outputs are in the bundle.

Failure definitions used by the judges (Chinese and English versions are in the prompts): basics: ; logic: ; knowledge: ; diversity: ; content_logic: ; interaction: .

8 Planned work

The following steps address the limitations above. Each will be reported with its sample sizes when completed.

1. More complete runs. Full-protocol runs of further models under the current version, to measure the stability of the order beyond two models.
2. Anchoring the official panel. Score about 30 released scenarios with the official panel and apply the rule of Section 4.6 to the 34 established orderings.
3. Human-labelled failure set. Label a stratified sample of segments (models across the whole published ranking, both languages) for every failure type by at least two independent annotators, and estimate the precision and recall of each judge against it.
4. Basics and Logic on the models ranked 1–8. Examine the remaining differences from MiniMax's verdicts against the dialogues, including the classification of viewpoint switches; any resulting change will be a new protocol version.
5. Simulator validation against real users. Compare simulated and real user turns with a human-likeness discriminator and distributional measures, following [14, 18], once a licensed real-user corpus is available.
6. Judge self-preference. Score the same frozen dialogues with judges of additional model families and estimate family effects.
7. Scenario-clustered intervals on the leaderboard for scores with several samples per scenario.
8. Harder scenarios. Long-range recall, deliberate out-of-character prompts and emotionally intense turns, to test whether the highest-scoring models can be separated by failures.

References

1. MiniMax. A Deep Dive into the MiniMax-M2-her. Technical blog post, 27 January 2026. https://www.minimax.io/news/a-deep-dive-into-the-minimax-m2-her-2
2. C. Du, X. Wang, A. Chen, W. Li, R. Xu, J. Liu, Z. Huang, R. Tian, Z. Sun, Y. Li, L. Feng, D. Ding, P. Zhao, Y. Xiao. HER: Human-like Reasoning and Reinforcement Learning for LLM Role-playing. arXiv:2601.21459 (v4, 29 April 2026). Fudan University and MiniMax. https://arxiv.org/abs/2601.21459
3. MiniMax. Role-play Benchmark (dataset). Hugging Face, released 28 January 2026; revision 3c1be2a5. https://huggingface.co/datasets/MiniMaxAI/role-play-bench
4. Z. M. Wang, Z. Peng, H. Que, et al. RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models. arXiv:2310.00746, 2023. https://arxiv.org/abs/2310.00746
5. Q. Tu, S. Fan, Z. Tian, et al. CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation. arXiv:2401.01275, 2024. https://arxiv.org/abs/2401.01275
6. J. Zhou, Y. Huang, B. Wen, et al. CharacterBench: Benchmarking Character Customization of Large Language Models. AAAI 2025. arXiv:2412.11912. https://arxiv.org/abs/2412.11912
7. X. Wang, H. Wang, Y. Zhang, et al. CoSER: A Comprehensive Literary Dataset and Framework for Training and Evaluating LLM Role-Playing and Persona Simulation. arXiv:2502.09082 (v3), 2025. https://arxiv.org/abs/2502.09082
8. H. Xiang, T. Tang, Y. Su, et al. RMTBench: Benchmarking LLMs Through Multi-Turn User-Centric Role-Playing. arXiv:2507.20352, 2025. https://arxiv.org/abs/2507.20352
9. X. Zhou, H. Zhu, L. Mathur, et al. SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents. ICLR 2024. arXiv:2310.11667. https://arxiv.org/abs/2310.11667
10. R. Wang, H. Yu, W. Zhang, et al. SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agents. arXiv:2403.08715, 2024. https://arxiv.org/abs/2403.08715
11. H. Goel, H. Zhu. LIFELONG-SOTOPIA: Evaluating Social Intelligence of Language Agents over Lifelong Social Interactions. arXiv:2506.12666, 2025. https://arxiv.org/abs/2506.12666
12. X. Wang, J. Yang, W. Li, et al. HumanLLM: Benchmarking and Improving LLM Anthropomorphism via Human Cognitive Patterns. arXiv:2601.10198, 2026. https://arxiv.org/abs/2601.10198
13. J. Shin, J. Oh, E. Kim, et al. Spotting Out-of-Character Behavior: Atomic-Level Evaluation of Persona Fidelity in Open-Ended Generation. arXiv:2506.19352, 2025. https://arxiv.org/abs/2506.19352
14. H. Chopra, K. Ghate, A. Caliskan, et al. Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents. arXiv:2605.12894, 2026. https://arxiv.org/abs/2605.12894
15. M. Abdulhai, R. Cheng, D. Clay, et al. Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning. NeurIPS 2025. arXiv:2511.00222. https://arxiv.org/abs/2511.00222
16. M. Kang, T. Wright, S. Eisape, et al. Pretrained Persona Mixture Models and Tandem Models for Human Simulation. arXiv:2609.22607, 2026. https://arxiv.org/abs/2609.22607
17. Z. Hu, J. Lian, Z. Xiao, et al. Population-Aligned Persona Generation for LLM-based Social Simulation. arXiv:2509.10127, 2025. https://arxiv.org/abs/2509.10127
18. E. Bao, A. Perez, J. Parapar, et al. Eval4Sim: An Evaluation Framework for Persona Simulation. CIKM 2026. arXiv:2603.02876. https://arxiv.org/abs/2603.02876
19. M. T. Kane. Validating the Interpretations and Uses of Test Scores. Journal of Educational Measurement 50(1):1–73, 2013. https://doi.org/10.1111/jedm.12000
20. S. Messick. Validity of Psychological Assessment: Validation of Inferences from Persons’ Responses and Performances as Scientific Inquiry into Score Meaning. American Psychologist 50(9):741–749, 1995. https://doi.org/10.1037/0003-066X.50.9.741
21. L. Zheng, W.-L. Chiang, Y. Sheng, et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks. arXiv:2306.05685. https://arxiv.org/abs/2306.05685
22. B. Efron, R. J. Tibshirani. An Introduction to the Bootstrap. Chapman & Hall/CRC, 1993. https://doi.org/10.1201/9780429246593
23. Y. Wang, J. Chen, H. Xiao. Role-Playing Agents Driven by Large Language Models: Current Status, Challenges, and Future Trends. arXiv:2601.10122, 2026. https://arxiv.org/abs/2601.10122

Appendix A: Prompts

The prompts below are served live from the protocol in force (…). Variables in double braces are filled from the scenario seed and the dialogue at run time. Development runs use variants that are not part of the protocol.

Loading…

Appendix B: Protocol history