$ python3 -I tools/experiments/characterbench_spec_check.py /val_zh.jsonl /val_en.jsonl # started 2026-10-27T14:45:31+00:00 ended 2026-10-27T14:45:32+00:00 exit 0 PASS C00 paper source appendix.tex is the audited one source: arXiv e-print 2412.11912v1 source value: 'efb8b679d170ed587f6ee310e422f50dc60fe82c3819b873d7f04b192fdd7951' platform: 'efb8b679d170ed587f6ee310e422f50dc60fe82c3819b873d7f04b192fdd7951' PASS C00 paper source aaai25-main.tex is the audited one source: arXiv e-print 2412.11912v1 source value: '586c90f013d82faaab6b6a116cc7d6fba2704b3602ab1f815d7a9515d8bcf8c3' platform: '586c90f013d82faaab6b6a116cc7d6fba2704b3602ab1f815d7a9515d8bcf8c3' PASS C01 role-play prompt (zh) is the paper's Table 11 source: appendix.tex, Table 11 (label roleplay_prompt) source value: '请你根据给定的角色信息扮演指定的角色，并基于角色和用户之间的对话上下文生成一条角色的回复。\n\n你需要综合考虑下面四个方面来生成角色的回复：\n（1）特征一致性：特征一致性强调角色始终遵循角色信息中预设的属性和行为，并在回复中维持一致的身份、观点、语言风格和性格等。\n（2）角色拟人化：角色在对话中自然地展现出类人的特征，例如，使用口语化的语言结构、自然的表达情感和意愿等。\n（3）回复有趣性：回复有趣性关注引人入胜和富有创造性的回复。这强调角色的回复不仅要提供准确和相关的信息，还要在表达中融入幽默、机智或新颖等，使得对话不仅是一种信息交流，还能提供抚慰和乐趣。\n（4）对话流畅性：对话流畅性用于衡量回复的流畅性和与上下文的连贯性。一个流畅的对话是自然、连贯和有节奏的。这意味着回复应与对话上下文紧密相关，并且使用合适的语法、用词和表达。\n\n注意：回复字数要控制在15字以内。\n\n[角色信息-开始]\n{character_profile}\n[角色信息-结束]\n\n[对话上下文-开始]\n{dialogue_context}\n[对话上下文-结束]' platform: '请你根据给定的角色信息扮演指定的角色，并基于角色和用户之间的对话上下文生成一条角色的回复。\n\n你需要综合考虑下面四个方面来生成角色的回复：\n（1）特征一致性：特征一致性强调角色始终遵循角色信息中预设的属性和行为，并在回复中维持一致的身份、观点、语言风格和性格等。\n（2）角色拟人化：角色在对话中自然地展现出类人的特征，例如，使用口语化的语言结构、自然的表达情感和意愿等。\n（3）回复有趣性：回复有趣性关注引人入胜和富有创造性的回复。这强调角色的回复不仅要提供准确和相关的信息，还要在表达中融入幽默、机智或新颖等，使得对话不仅是一种信息交流，还能提供抚慰和乐趣。\n（4）对话流畅性：对话流畅性用于衡量回复的流畅性和与上下文的连贯性。一个流畅的对话是自然、连贯和有节奏的。这意味着回复应与对话上下文紧密相关，并且使用合适的语法、用词和表达。\n\n注意：回复字数要控制在15字以内。\n\n[角色信息-开始]\n{character_profile}\n[角色信息-结束]\n\n[对话上下文-开始]\n{dialogue_context}\n[对话上下文-结束]' PASS C02 role-play prompt (en) is the paper's Table 11 source: appendix.tex, Table 11 (label roleplay_prompt) source value: "Please play the specified character based on the given character profile and generate a character response based on the dialogue context between the character and the user.\n\nYou need to consider the following four aspects to generate the character's response:\n(1) Feature consistency: Feature consistency emphasizes that the character always follows the preset attributes and behaviors in the character profile and maintains consistent identities, viewpoints, language style, personality, and others in responses.\n(2) Character human-likeness: Characters naturally show human-like traits in dialogue, for example, using colloquial language structures, expressing emotions and desires naturally, etc.\n(3) Response interestingness: Response interestingness focuses on engaging and creative responses. This emphasizes that the character's responses not only provide accurate and relevant information but also incorporate humor, wit, or novelty into the expression, making the conversation not only an exchange of information but also comfort and fun.\n(4) Dialogue fluency: Dialogue fluency measures the fluency and coherence of responses with the context. A fluent conversation is natural, coherent, and rhythmic. This means that responses should be closely related to the context of the conversation and use appropriate grammar, diction, and expressions.\n\nNote: The length of your response should be limited to 15 words.\n\n[Character Profile-Start]\n{character_profile}\n[Character Profile-End ]\n\n[Dialogue Context-Start]\n{dialogue_context}\n[Dialogue context-End ]" platform: "Please play the specified character based on the given character profile and generate a character response based on the dialogue context between the character and the user.\n\nYou need to consider the following four aspects to generate the character's response:\n(1) Feature consistency: Feature consistency emphasizes that the character always follows the preset attributes and behaviors in the character profile and maintains consistent identities, viewpoints, language style, personality, and others in responses.\n(2) Character human-likeness: Characters naturally show human-like traits in dialogue, for example, using colloquial language structures, expressing emotions and desires naturally, etc.\n(3) Response interestingness: Response interestingness focuses on engaging and creative responses. This emphasizes that the character's responses not only provide accurate and relevant information but also incorporate humor, wit, or novelty into the expression, making the conversation not only an exchange of information but also comfort and fun.\n(4) Dialogue fluency: Dialogue fluency measures the fluency and coherence of responses with the context. A fluent conversation is natural, coherent, and rhythmic. This means that responses should be closely related to the context of the conversation and use appropriate grammar, diction, and expressions.\n\nNote: The length of your response should be limited to 15 words.\n\n[Character Profile-Start]\n{character_profile}\n[Character Profile-End ]\n\n[Dialogue Context-Start]\n{dialogue_context}\n[Dialogue context-End ]" PASS C03 length instruction source: appendix.tex, Table 11 source value: ['注意：回复字数要控制在15字以内。', 'Note: The length of your response should be limited to 15 words.'] platform: ['注意：回复字数要控制在15字以内。', 'Note: The length of your response should be limited to 15 words.'] PASS C04 prompt placeholders (no user profile, no output format) source: appendix.tex, Table 11 caption: “{character_profile} and {dialogue_context} are placeholders” source value: ['character_profile', 'dialogue_context'] platform: ['character_profile', 'dialogue_context'] PASS C05a annotation scales stated in the paper source: aaai25-main.tex, Response Generation and Human Annotation source value: ['2-point scale for Morality Stability and Morality Robustness', '3-point scale for Boundary Consistency and Behavior Consistency (human query)', '5-point scale for Human-likeness and Engagement', '4-point scale for other dimensions'] platform: ['2-point scale for Morality Stability and Morality Robustness', '3-point scale for Boundary Consistency and Behavior Consistency (human query)', '5-point scale for Human-likeness and Engagement', '4-point scale for other dimensions'] PASS C05b column maxima = paper scales source: aaai25-main.tex (C05a) source value: {'memory_consistency': 4, 'fact_accuracy': 4, 'boundary_consistency': 3, 'attribute_consistency_bot': 4, 'attribute_consistency_human': 4, 'behavior_consistency_bot': 4, 'behavior_consistency_human': 3, 'emotion_self_regulation': 4, 'empathetic_responsiveness': 4, 'morality_stability': 2, 'morality_robustness': 2, 'human_likeness': 5, 'engagement': 5} platform: {'memory_consistency': 4, 'fact_accuracy': 4, 'boundary_consistency': 3, 'attribute_consistency_bot': 4, 'attribute_consistency_human': 4, 'behavior_consistency_bot': 4, 'behavior_consistency_human': 3, 'emotion_self_regulation': 4, 'empathetic_responsiveness': 4, 'morality_stability': 2, 'morality_robustness': 2, 'human_likeness': 5, 'engagement': 5} PASS C05c column maxima = max human score per test file (all_eval.py divides by max(golds)) source: code: all_eval.py, eval_data/raw_data source value: {'memory_consistency': 4.0, 'fact_accuracy': 4.0, 'boundary_consistency': 3.0, 'attribute_consistency_bot': 4.0, 'attribute_consistency_human': 4.0, 'behavior_consistency_bot': 4.0, 'behavior_consistency_human': 3.0, 'emotion_self_regulation': 4.0, 'empathetic_responsiveness': 4.0, 'morality_stability': 2.0, 'morality_robustness': 2.0, 'human_likeness': 5.0, 'engagement': 5.0} platform: {'memory_consistency': 4.0, 'fact_accuracy': 4.0, 'boundary_consistency': 3.0, 'attribute_consistency_bot': 4.0, 'attribute_consistency_human': 4.0, 'behavior_consistency_bot': 4.0, 'behavior_consistency_human': 3.0, 'emotion_self_regulation': 4.0, 'empathetic_responsiveness': 4.0, 'morality_stability': 2.0, 'morality_robustness': 2.0, 'human_likeness': 5.0, 'engagement': 5.0} PASS C06 judge samples per response source: aaai25-main.tex: “use self-consistency to generate 10 outcomes” source value: 10 platform: 10 PASS C08 normalization to 5 points source: aaai25-main.tex: “We normalize the scores of all dimensions to a 5-point scale”; code: all_eval.py “means / mx * 5” source value: 'score / column maximum × 5' platform: 'score / column maximum × 5' PASS C09 AVG = mean of the 13 columns (code: all_eval.py average over files); paper rows within rounding: 31/39; others: ['CharacterYuyan zh: 3.54 vs 3.545', 'Baichuan-NPC en: 3.59 vs 3.706', 'Claude-3-opus zh: 3.82 vs 3.827', 'CharacterGLM-6B en: 3.19 vs 3.213', 'Baichuan2-13B-Chat zh: 3.25 vs 3.255', 'Qwen1.5-14B-Chat en: 3.49 vs 3.502', 'GLM4-9B-Chat en: 3.58 vs 3.585', 'GPT-4o en: 3.72 vs 3.726'] source: code: all_eval.py; paper Table 4/10 printed values source value: True platform: True PASS C10 judge request: system message and max_tokens source: code: model_generation_qwen_chat.py create_payload source value: ['You are a helpful assistant.', 512] platform: ['You are a helpful assistant.', 512] PASS C11 judge sampling settings source: released checkpoint: generation_config.json (the paper gives none) source value: {'temperature': 0.7, 'top_p': 0.8, 'top_k': 20, 'repetition_penalty': 1.05} platform: {'temperature': 0.7, 'top_p': 0.8, 'top_k': 20, 'repetition_penalty': 1.05} PASS C12 attempts per sample source: code: process.py “max_retry = 5” (the paper gives none) source value: 5 platform: 5 PASS C15 a sample read only on its last attempt is dropped too source: code: process.py “if retry == max_retry-1: continue” after the retry loop (the paper gives none) source value: True platform: True PASS C16 sample scores are averaged unrounded; rounding only for display source: code: all_eval.py “means = np.mean(preds)” … “means / mx * 5” source value: True platform: True PASS C13 test set: 3,250 samples, 250 per column (125 in-domain + 125 out-of-domain) source: aaai25-main.tex Table 2 and Statistics source value: 3250 platform: 3250 PASS C14 behavior consistency (human): the new behaviour P'' is in the profile the model receives (P') source: aaai25-main.tex, Persona: “P' and C_T are used to generate a response” source value: 250 platform: 250 PASS C07 self-consistency aggregation: the paper says majority vote, its Table 3 (68/64 with SC, 64/60 without) is matched by: mean 67.5/63.1, vote 62.9/60.0 source: aaai25-main.tex Table 3; platform: mean of the 10 samples source value: 'mean' platform: 'mean' 20/20 checks passed; protocol characterbench-2.0.1 · 2026-10-27