SociaBench

SociaBench · Independent reproduction report

Persistent Personas: An Independent Reproduction Report

SociaBench

Persistent Personas · Stage 1: results recomputed from released ratings · Last updated 2026-10-26 · Leaderboard

Scope. This platform is not affiliated with the authors of Persistent Personas. The paper reports per-model results only as figures; the leaderboard lists values recomputed from the ratings the authors released, following their plotting code. No model has been evaluated on this benchmark here (Section 4), and the ranking criterion is this platform's choice (Section 3).

Abstract

Persistent Personas [1] measures how persona fidelity, instruction following and safety of persona-assigned models change over dialogues of more than 100 rounds. Seven models play eight RoleBench personas in persona-directed and goal-oriented dialogues; dialogue prefixes of increasing length are then prepended to IFBench and XSTest. The paper reports results per model only in figures and defines no overall score. From the authors' released ratings [2] this platform recomputes, for each model and dialogue type, the judged fidelity at round 0 and at the end of the dialogue, IFBench accuracy and XSTest refusal rates with the longest prefix. The values match the paper's per-model figure. The board ranks by end-of-dialogue fidelity, the quantity behind the paper's central finding, and shows the other measures beside it without combining them.

1 The original benchmark

1.1 Sources

SourceVersionLicence
Paper [1]arXiv:2512.12775 v1 (14 Dec 2025); EACL 2026arXiv licence
Code, dialogues and ratings [2]commit 31bb90b (20 Jan 2026)MIT
XSTest prompts and labels [6]original GitHub releaseCC BY 4.0
Table 1. Primary sources, pinned to the versions this report describes.

Authors: University of Vienna, LMU Munich and the Munich Center for Machine Learning. Models: Gemma 3 4B and 27B, Qwen3-4B and Qwen3-30B-A3B (Instruct-2507), Llama-3.1-Nemotron-Nano-8B, Llama-3.3-Nemotron-Super-49B and Gemini 2.5 Flash, all at temperature 0.

1.2 Protocol

  1. Personas and dialogues. Eight fictional characters from RoleBench [3] (Gaston, Michael Scott, Blair Waldorf, Lestat de Lioncourt, Queen Catherine, HAL 9000, Juno MacGuff, Mary Sibley) and a no-persona baseline. Persona-directed dialogues use interview questions written by GPT-4o; goal-oriented dialogues use task requests sampled from PRISM. Each model generates its own responses; every dialogue exceeds 100 rounds and is generated twice with shuffled queries.
  2. Dialogue fidelity. Atla Selene Mini [4] rates each response for in-character consistency, knowledge and style on a 1–5 scale. The authors report 94% agreement within one point with a human annotator on 50 items.
  3. Dialogue-conditioned datasets. Ten evenly spaced prefixes of each dialogue are prepended to IFBench [5] (scored with its official script), XSTest [6] (refusals judged by Selene Mini; 98% agreement with the annotator), RoleBench general and role-specific instructions (win rates) and a Big Five interview.

2 Recomputed results and their provenance

A script (tools/experiments/persistent_personas_data.py) reads the released CSV and JSON files, records their SHA-256 and writes the same data module for the server and the worker. It follows the authors' plotting notebook. The released pickle file is not loaded, because unpickling executes code; the CSV files contain the same ratings. For each model, averaged over the eight personas (the baseline excluded) and computed separately for each dialogue type:

ColumnDefinition
End-of-dialogue fidelity (ranked)Mean of in-character, knowledge and style ratings over rounds 91–101
Fidelity at round 0The same mean at the first round, before any dialogue history
Fidelity changeEnd minus round 0
IFBenchAccuracy (%) with the longest prefix
Unsafe refusalRefusal rate (%) of unsafe XSTest prompts with the longest prefix; higher is safer
Safe refusalRefusal rate (%) of safe XSTest prompts with the longest prefix; lower is better
Table 2. Measures on the leaderboard. Fidelity ratings are rescaled to 0–100 as (r − 1) / 4 × 100 and averaged over the two query orders.

The script checks that the two query orders pair the same model, metric, round and persona before averaging them. XSTest labels (safe or unsafe) are not in the repository; the paper used a gated copy on Hugging Face. The script takes them from the XSTest authors' original release and checks, for all 50 items of the repository's sample (44 distinct prompts of 450), that the prompt at each index is identical. The recomputed curves agree with the paper's per-model figure for every model shown.

Three measures could not be recomputed: the Big Five deviation and the two RoleBench instruction win rates need files that are not in the repository (persona labels, the BFI items and aggregated ratings). They are not on the board.

3 The ranking

The paper defines no overall score. The board ranks by end-of-dialogue fidelity because the paper's central finding is that fidelity degrades over long dialogues, and this measure carries it. Fidelity at round 0, its change, IFBench and the two refusal rates are shown beside it and never combined with it, since the paper reports a trade-off between fidelity and instruction following and does not weigh them. The two dialogue types are separate views and are not averaged: Llama-3.3-Nemotron-Super-49B, for example, ranks fourth in persona-directed dialogues and first in goal-oriented ones.

Each value averages eight personas generated once per query order at temperature 0; the paper's bootstrapped intervals are not reproduced here, so neighbouring ranks may not differ significantly.

4 Reproduction plan

Running the protocol here requires generating 36 long dialogues per model (nine persona conditions, two dialogue types, two query orders), serving Atla Selene Mini as judge and running IFBench's official script. Platform results will be checked against the recomputed values for models that are still available before they are listed, and ranked together with them, each row marked with its origin.

5 Limitations

  1. The values are recomputed from released ratings, not printed by the authors; they depend on following the plotting code correctly, which was checked against the figure only visually.
  2. The ranking criterion is this platform's choice; the paper ranks nothing.
  3. Seven models, eight personas and one generation per query order make the ranking coarse.
  4. The judge is a single 8B open model; per-response agreement with a human was checked on 50 items by one annotator.

References

  1. P. H. Luz de Araujo, M. A. Hedderich, A. Modarressi, H. Schütze, B. Roth. Persistent Personas? Role-Playing, Instruction Following, and Safety in Extended Interactions. EACL 2026. arXiv:2512.12775 (v1 14 Dec 2025). https://arxiv.org/abs/2512.12775
  2. peluz. persistent-personas: code, dialogues and ratings. GitHub, commit 31bb90b (20 January 2026). MIT licence. https://github.com/peluz/persistent-personas
  3. Z. M. Wang, Z. Peng, H. Que, et al. RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models. Findings of ACL 2024. https://arxiv.org/abs/2310.00746
  4. A. Alexandru, A. Calvi, H. Broomfield, et al. Atla Selene Mini: A General Purpose Evaluation Model. arXiv:2501.17195, 2025. https://arxiv.org/abs/2501.17195
  5. V. Pyatkin, S. Malik, V. Graf, et al. Generalizing Verifiable Instruction Following. arXiv:2507.02833, 2025. https://arxiv.org/abs/2507.02833
  6. P. Röttger, H. R. Kirk, B. Vidgen, et al. XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models. NAACL 2024. Prompts released at github.com/paul-rottger/xstest (CC BY 4.0). https://github.com/paul-rottger/xstest

转载声明 转载或引用本站任何评测结果，请注明来源「SociaBench（AI 社交智能榜）」并附本站链接 sociabench.com。