#### prompt: 52 hits; per file {'Sections/1_Introduction.tex': 1, 'Sections/2_Related_Work.tex': 1, 'Sections/3_Dataset.tex': 2, 'Sections/4_Methodology.tex': 2, 'Sections/Appendix.tex': 25, 'Tables/prompts_agent.tex': 3, 'Tables/prompts_agent_2.tex': 3, 'Tables/prompts_data.tex': 3, 'Tables/prompts_data_2.tex': 3, 'Tables/prompts_data_3.tex': 3, 'Tables/prompts_eval.tex': 3, 'Tables/prompts_eval_2.tex': 3}
Sections/1_Introduction.tex:37: ackgrounds, supporting various purposes including prompting, retrieval, model training and evaluation.
Sections/2_Related_Work.tex:22: hese datasets support various purposes, including prompting, training, retrieval augmentation, and evaluation of RPL
Sections/3_Dataset.tex:84: ed chunking, \ie, during data extraction, we also prompt LLMs to identify truncated plots or trailing content in the
Sections/3_Dataset.tex:100: For technical details, including our prompts, engineering implementation, and handling mechanisms for e
Sections/4_Methodology.tex:105: For detailed prompts and rubrics, please refer to \S\ref{sec:prompts}.
Sections/4_Methodology.tex:105: d prompts and rubrics, please refer to \S\ref{sec:prompts}.
Sections/Appendix.tex:47: each book; that is, they are not included in our prompts or datasets for training or retrieval purposes. Additional
Sections/Appendix.tex:54: g LLMs to extract the complete text of a plot, we prompt LLMs to extract the first and last sentences of each plot.
Sections/Appendix.tex:66: In such cases, we prompt the repairing LLM to truncate the JSON at an appropriate po
Sections/Appendix.tex:146: targets for optimization, while using the system prompt and messages from other characters as inputs. Adjacent inpu
Sections/Appendix.tex:148: ically generate role-playing instructions (system prompts) using varied phrasings, formats, and data types, as shown
Sections/Appendix.tex:148: ats, and data types, as shown in Tables ~\ref{tab:prompts_agent} and ~\ref{tab:prompts_agent_2}.
Sections/Appendix.tex:148: in Tables ~\ref{tab:prompts_agent} and ~\ref{tab:prompts_agent_2}.
Sections/Appendix.tex:187: \subsection{Prompting Strategies for Exisitng RPLA Benchmarks}
Sections/Appendix.tex:189: tep{yuan2024evaluating}, we adapt or refine their prompting strategies as follows:
Sections/Appendix.tex:352: \section{Prompts}
Sections/Appendix.tex:353: \label{sec:prompts}
Sections/Appendix.tex:355: In this section, we list the detailed prompts for:
Sections/Appendix.tex:356: \textit{1)} dataset curation in Tables ~\ref{tab:prompts_data} to ~\ref{tab:prompts_data_3};
Sections/Appendix.tex:356: on in Tables ~\ref{tab:prompts_data} to ~\ref{tab:prompts_data_3};
Sections/Appendix.tex:357: LA and multi-agent simulation in Tables ~\ref{tab:prompts_agent} to ~\ref{tab:prompts_agent_2}, which
Sections/Appendix.tex:357: n in Tables ~\ref{tab:prompts_agent} to ~\ref{tab:prompts_agent_2}, which
Sections/Appendix.tex:359: 3)} Penalty-based LLM Judging in Tables ~\ref{tab:prompts_eval} to ~\ref{tab:prompts_eval_2}.
Sections/Appendix.tex:359: ng in Tables ~\ref{tab:prompts_eval} to ~\ref{tab:prompts_eval_2}.
Sections/Appendix.tex:362: \input{Tables/prompts_data}
Sections/Appendix.tex:364: \input{Tables/prompts_data_2}
Sections/Appendix.tex:366: \input{Tables/prompts_data_3}
Sections/Appendix.tex:368: \input{Tables/prompts_agent}
Sections/Appendix.tex:370: \input{Tables/prompts_agent_2}
Sections/Appendix.tex:372: \input{Tables/prompts_eval}
Sections/Appendix.tex:374: \input{Tables/prompts_eval_2}
Tables/prompts_agent.tex:6: \multicolumn{2}{c}{\textbf{Prompts for RPLAs and Multi-agent Systems}} \\
Tables/prompts_agent.tex:109: \caption{Prompts for RPLAs and multi-agent systems in \method. }
Tables/prompts_agent.tex:110: \label{tab:prompts_agent}
Tables/prompts_agent_2.tex:6: \multicolumn{2}{c}{\textbf{Prompts for RPLAs and Multi-agent Systems}} \\
Tables/prompts_agent_2.tex:54: \caption{Prompts for RPLAs and multi-agent systems in \method. }
Tables/prompts_agent_2.tex:55: \label{tab:prompts_agent_2}
Tables/prompts_data.tex:6: \multicolumn{2}{c}{\textbf{Prompts for Dataset Curation}} \\
Tables/prompts_data.tex:78: \caption{Prompts for dataset construction in \method. }
Tables/prompts_data.tex:79: \label{tab:prompts_data}
Tables/prompts_data_2.tex:6: \multicolumn{2}{c}{\textbf{Prompts for Dataset Curation}} \\
Tables/prompts_data_2.tex:93: \caption{Prompts for dataset construction in \method. }
Tables/prompts_data_2.tex:94: \label{tab:prompts_data_2}
Tables/prompts_data_3.tex:6: \multicolumn{2}{c}{\textbf{Prompts for Dataset Curation}} \\
Tables/prompts_data_3.tex:74: \caption{Prompts for dataset construction in \method. }
Tables/prompts_data_3.tex:75: \label{tab:prompts_data_3}
Tables/prompts_eval.tex:6: \multicolumn{2}{c}{\textbf{Prompts for Penalty-based LLM Critics}} \\
Tables/prompts_eval.tex:90: \caption{Prompts for penalty-based LLM critics in \method. }
Tables/prompts_eval.tex:91: \label{tab:prompts_eval}
Tables/prompts_eval_2.tex:6: \multicolumn{2}{c}{\textbf{Prompts for Penalty-based LLM Critics}} \\
Tables/prompts_eval_2.tex:138: \caption{Prompts for penalty-based LLM critics in \method. }
Tables/prompts_eval_2.tex:139: \label{tab:prompts_eval_2}

#### length: 30 hits; per file {'main.tex': 3, 'Sections/2_Related_Work.tex': 1, 'Sections/4_Methodology.tex': 3, 'Sections/5_Experiments.tex': 1, 'Sections/Appendix.tex': 12, 'Tables/dataset_features.tex': 1, 'Tables/exp_idood.tex': 1, 'Tables/exp_main_cot_full.tex': 1, 'Tables/exp_otherbmks.tex': 1, 'Tables/exp_rag.tex': 1, 'Tables/exp_wr.tex': 1, 'Tables/judge_alignment.tex': 1, 'Tables/main_exp.tex': 1, 'Tables/main_exp_cf3.tex': 1, 'Tables/main_exp_cot.tex': 1}
main.tex:65: \setlength{\belowcaptionskip}{0pt}
main.tex:119: \setlength\parindent{0pt}
main.tex:229: % \setlength{\parskip}{3pt}
Sections/2_Related_Work.tex:29: ver, LLM judges suffer from inherent biases, \eg, length and position bias~\citep{li2024judgesurvey}, and may lack t
Sections/4_Methodology.tex:101: mulations naturally make more flaws, we implement length correction to reduce bias in LLM judging following ~\citet{
Sections/4_Methodology.tex:102: Specifically, we obtain the length corrected score as
Sections/4_Methodology.tex:104: o 1.5 based on statistical analysis in \S\ref{sec:length_correction}.
Sections/5_Experiments.tex:88: ce, \textit{(ii)} scoring rubrics, \textit{(iii)} length correction, and \textit{(iv)} dimension separation.
Sections/Appendix.tex:172: learning rate of \(1 \times 10^{-5}\), a sequence length of 16,384, training for 8 epochs, and a global batch size o
Sections/Appendix.tex:197: \subsection{Length Correction}
Sections/Appendix.tex:198: \label{sec:length_correction}
Sections/Appendix.tex:201: rally accumulate more flaws, we need to implement length correction to reduce length bias in LLM judges, following p
Sections/Appendix.tex:201: we need to implement length correction to reduce length bias in LLM judges, following previous work~\citep{li2024ju
Sections/Appendix.tex:203: We analyze the phenomenon of length bias in penalty-based scoring.
Sections/Appendix.tex:207: As shown in Figure~\ref{fig:length_correction}, we plot the data points for these cases and pe
Sections/Appendix.tex:210: To mitigate this bias, we implement length correction by compensating
Sections/Appendix.tex:212: Specifically, we compute the length-corrected score as \( s = 100-\sum_{f\in\mathcal{F}}v_f + \
Sections/Appendix.tex:219: Linear regression results showing length bias of penalty-based LLM critics in GCA evaluation.
Sections/Appendix.tex:221: \label{fig:length_correction}
Sections/Appendix.tex:384: mechanism and detailed rubrics, problems such as length bias persist. Moreover, LLM Judges may lack the necessary k
Tables/dataset_features.tex:3: \setlength{\tabcolsep}{4pt}
Tables/exp_idood.tex:4: \setlength{\tabcolsep}{2pt}
Tables/exp_main_cot_full.tex:3: \setlength{\tabcolsep}{3pt}
Tables/exp_otherbmks.tex:3: \setlength{\tabcolsep}{4pt}
Tables/exp_rag.tex:3: \setlength{\tabcolsep}{3pt}
Tables/exp_wr.tex:4: \setlength{\tabcolsep}{3pt}
Tables/judge_alignment.tex:11: \quad\quad {\textit{w/o} length correction} & 64.5 \\
Tables/main_exp.tex:3: \setlength{\tabcolsep}{2.5pt}
Tables/main_exp_cf3.tex:4: \setlength{\tabcolsep}{3pt}
Tables/main_exp_cot.tex:3: \setlength{\tabcolsep}{3pt}

#### words: 23 hits; per file {'main.tex': 3, 'Sections/Appendix.tex': 1, 'Tables/case_cerci.tex': 2, 'Tables/case_cerci_2.tex': 2, 'Tables/case_cerci_3.tex': 2, 'Tables/case_sansa.tex': 4, 'Tables/case_sansa_2.tex': 3, 'Tables/prompts_data.tex': 2, 'Tables/prompts_data_2.tex': 3, 'Tables/prompts_eval_2.tex': 1}
main.tex:188: morekeywords={self, clip, exp, mse_loss, uniform_sample, concatenate, lo
main.tex:188: e, concatenate, logsumexp}, % Add keywords here
main.tex:189: keywordstyle=\color{deepblue},
Sections/Appendix.tex:329: conversation extracted from \textit{A Storm of Swords (A Song of Ice and Fire, \#3)} and the corresponding simula
Tables/prompts_data.tex:67: cription of the character before this plot ($~$20 words).",
Tables/prompts_data.tex:68: character development relevant to the plot ($~$30 words).",
Tables/prompts_data_2.tex:14: "topic": "The topic of the conversation (~10 words)",
Tables/prompts_data_2.tex:77: rs with essential context and atmosphere ($<$ 200 words). Include all necessary background information while exclud
Tables/prompts_data_2.tex:80: emotional state before the conversation ($<$ 100 words). Including their feelings, motivations, objectives, and in
Tables/prompts_eval_2.tex:111: * Mechanically repeats one's own words or phrases. More repetitions lead to higher severity (up to

#### characters: 96 hits; per file {'Sections/0_Abstract.tex': 4, 'Sections/1_Introduction.tex': 7, 'Sections/2_Related_Work.tex': 3, 'Sections/3_Dataset.tex': 11, 'Sections/4_Methodology.tex': 9, 'Sections/5_Experiments.tex': 4, 'Sections/7_Conclusion.tex': 2, 'Sections/Appendix.tex': 15, 'Tables/case_cerci.tex': 1, 'Tables/case_sansa.tex': 1, 'Tables/dataset_features.tex': 3, 'Tables/prompts_agent.tex': 2, 'Tables/prompts_agent_2.tex': 9, 'Tables/prompts_data.tex': 6, 'Tables/prompts_data_2.tex': 4, 'Tables/prompts_data_3.tex': 4, 'Tables/prompts_eval.tex': 3, 'Tables/prompts_eval_2.tex': 8}
Sections/0_Abstract.tex:2: However, simulating established characters presents a challenging task for RPLAs, due to the lack of a
Sections/0_Abstract.tex:3: n protocol towards effective RPLAs of established characters.
Sections/0_Abstract.tex:4: The \method dataset covers 17,966 characters from 771 renowned books.
Sections/0_Abstract.tex:6: ng LLMs, where LLMs sequentially portray multiple characters in book scenes.
Sections/1_Introduction.tex:7: versation dialogues and settings, plot summaries, characters' inner thoughts, authentically sourced from renowned books.
Sections/1_Introduction.tex:15: ersonas of various types, including demographics, characters, or daily individuals~\citep{chen2024from}.
Sections/1_Introduction.tex:17: This paper studies \textbf{RPLAs for established characters}, which represent a crucial yet challenging task beyond the
Sections/1_Introduction.tex:18: fically, RPLAs should faithfully align with their characters' complex backgrounds and capture their nuanced personalitie
Sections/1_Introduction.tex:22: ing datasets are limited to dialogues between two characters, and lack necessary dialogue contexts and knowledge in othe
Sections/1_Introduction.tex:38: \textit{ii)} Conversations in \method capture characters' actions and internal thoughts beyond surface-level speech,
Sections/1_Introduction.tex:43: Given a conversation with messages $M$, characters $\mathcal{C}$ and setting $\mathcal{S}$, GCA requires an ac
Sections/2_Related_Work.tex:6: studies develop prototypes of RPLAs for fictional characters~\citep{shao2023character} and multi-agent systems that simu
Sections/2_Related_Work.tex:11: persona types, we focus on RPLAs for established characters, which should faithfully align with their characters’ compl
Sections/2_Related_Work.tex:11: racters, which should faithfully align with their characters’ complex backgrounds and nuanced personalities.
Sections/3_Dataset.tex:5: od dataset, which covers authentic data of 17,966 characters from 771 renowned books.
Sections/3_Dataset.tex:17: tains rich data types on plots, conversations and characters.
Sections/3_Dataset.tex:40: Comprehensive: covering extensive knowledge about characters and plots from the books;
Sections/3_Dataset.tex:45: interconnected elements: plots, conversations and characters.
Sections/3_Dataset.tex:46: ext, summary, conversations in this plot, and key characters' current states and experiences in this plot.
Sections/3_Dataset.tex:47: tual settings including scenario descriptions and characters' motivations.
Sections/3_Dataset.tex:48: \textbf{Characters} are associated with their conversations and plots, based o
Sections/3_Dataset.tex:56: m \textbf{Speech} is for verbal communications of characters.
Sections/3_Dataset.tex:64: s physical changes and reactions from unspecified characters or crowds.
Sections/3_Dataset.tex:76: able~\ref{tab:selected_books}, these books offer characters and narratives with literary significance and widespread re
Sections/3_Dataset.tex:95: e extracted data, we form the knowledge bases for characters in three steps.
Sections/4_Methodology.tex:14: conversation with dialogue messages $M$, involved characters $\mathcal{C}$, and contextual setting $\mathcal{S}$, an act
Sections/4_Methodology.tex:21: a$ and motivation, and profiles of other involved characters, which provide comprehensive context for role-playing.
Sections/4_Methodology.tex:28: \textit{1)} The \method dataset contains massive characters and conversation settings from extensive books.
Sections/4_Methodology.tex:29: We train models on all characters in each conversation, ranging from major characters with de
Sections/4_Methodology.tex:29: aracters in each conversation, ranging from major characters with detailed profiles to minor roles driven only by the co
Sections/4_Methodology.tex:31: data by including or excluding: profiles of other characters, plot summaries, and characters' motivations;
Sections/4_Methodology.tex:31: profiles of other characters, plot summaries, and characters' motivations;
Sections/4_Methodology.tex:32: \textit{3)} We train models both with and without characters' internal thoughts in the extracted dialogues.
Sections/4_Methodology.tex:96: : Assesses whether RPLAs faithfully portray their characters, with rubrics examining language style, knowledge and backg
Sections/5_Experiments.tex:9: We exclude characters' inner thoughts for LLM critics, and set the continue-from
Sections/5_Experiments.tex:40: dels maintain strong performance on out-of-domain characters.
Sections/5_Experiments.tex:59: models' strong capability in nuanced portrayal of characters' personalities and behaviors.
Sections/5_Experiments.tex:124: \textit{1)} Models consistently benefit from characters' retrieved experiences and conversations, especially for \m
Sections/7_Conclusion.tex:2: Towards effective RPLAs for established characters,
Sections/7_Conclusion.tex:6: ng LLMs, where LLMs sequentially portray multiple characters within authentic book scenarios.
Sections/Appendix.tex:10: dialogue data from 771 books and 17,966 distinct characters.
Sections/Appendix.tex:125: Persona Origin & Book characters & LLM-synthesized personas \\
Sections/Appendix.tex:136: serving the authenticity and depth of established characters for high-fidelity role-playing applications.
Sections/Appendix.tex:144: ur models’ ability to generalize to out-of-domain characters and books.
Sections/Appendix.tex:145: very \method conversation, encompassing both main characters and minor roles.
Sections/Appendix.tex:146: e using the system prompt and messages from other characters as inputs. Adjacent inputs are concatenated.
Sections/Appendix.tex:152: \textit{1)} Profiles of other characters in this conversation;
Sections/Appendix.tex:183: sampling strategy to prioritize well-established characters with more persona data. The sampling process consists of th
Sections/Appendix.tex:185: he weight of a conversation as the average of its characters’ weights, including both main characters and side roles. Fi
Sections/Appendix.tex:185: e of its characters’ weights, including both main characters and side roles. Finally, separately for the in-domain and o
Sections/Appendix.tex:254: paragraph{Generalization of \method Models to New Characters}
Sections/Appendix.tex:257: maintain strong performance even on out-of-domain characters.
Sections/Appendix.tex:274: plore three retrieval sources related to specific characters: dialogues from conversations, experiences from plots, and
Sections/Appendix.tex:389: The generated thoughts hardly capture characters' sophisticated thinking processes.
Sections/Appendix.tex:392: ataset may not cover all plots, conversations and characters from the source material. Improving recall is hence an impo
Tables/dataset_features.tex:28: For characters, Num. count characters with profiles, and Expr. denotes str
Tables/dataset_features.tex:28: For characters, Num. count characters with profiles, and Expr. denotes structured character exper
Tables/dataset_features.tex:29: s, and Multi-Chara. denotes involving more than 2 characters. Num. (number), Conv. (conversation), and Summ. (summary) a
Tables/prompts_agent.tex:27: ===Information about the other Characters===
Tables/prompts_agent.tex:80: Here is the your knowledge about the other characters:
Tables/prompts_agent_2.tex:11: provide the environmental feedback: Based on the characters' interactions, dialogues, and actions, describe the resulti
Tables/prompts_agent_2.tex:15: - Reactions of background characters or crowds
Tables/prompts_agent_2.tex:21: oid dictating the actions or dialogue of the main characters (including \{major\_characters\}).
Tables/prompts_agent_2.tex:21: alogue of the main characters (including \{major\_characters\}).
Tables/prompts_agent_2.tex:25: - You may include actions and reactions of minor characters or crowds, as long as they're not main characters (includin
Tables/prompts_agent_2.tex:25: characters or crowds, as long as they're not main characters (including \{major\_characters\}).
Tables/prompts_agent_2.tex:25: s they're not main characters (including \{major\_characters\}).
Tables/prompts_agent_2.tex:29: - Respond to subtle cues in the characters' interactions to create a dynamic, reactive environment.
Tables/prompts_agent_2.tex:42: al feedback. Choose a name from this list: \{all\_characters\}. If it's unclear who should act next, output "random". If
Tables/prompts_data.tex:17: re its prominence from 1 to 100, and list the key characters and their roles, thoughts and actions in it.
Tables/prompts_data.tex:19: nd topic of the conversations. Then, list the key characters with their names, descriptions and thoughts at this point.
Tables/prompts_data.tex:21: re the conversations are faithful to the plot and characters. They should be based on the original conversations in the
Tables/prompts_data.tex:27: en generating thoughts, you should think from the characters' perspectives, analyzing the internal thoughts behind their
Tables/prompts_data.tex:29: as "Environment". The information should exclude characters' active thoughts, observations, and actions.
Tables/prompts_data.tex:64: \quad "key\_characters": [
Tables/prompts_data_2.tex:15: "key\_characters": [
Tables/prompts_data_2.tex:43: 3. In the OUTPUT, use characters' full names, omitting any titles.
Tables/prompts_data_2.tex:63: sation from \{book\}, enhance the scene setup and characters' thoughts to create a comprehensive foundation for dramatic
Tables/prompts_data_2.tex:79: \quad "key\_characters": [\{ "name": "Character name",
Tables/prompts_data_3.tex:16: 3. In the OUTPUT, use characters' full names, omitting any titles.
Tables/prompts_data_3.tex:41: ess, your task is to: i) generate a list of named characters with their official names (in \{language\}); ii) For each n
Tables/prompts_data_3.tex:48: "named\_characters": [
Tables/prompts_data_3.tex:49: The list of named characters with their official names. Each character should appear onl
Tables/prompts_eval.tex:15: * Profiles of the main characters, including {major\_characters}.
Tables/prompts_eval.tex:15: rofiles of the main characters, including {major\_characters}.
Tables/prompts_eval.tex:21: not spoken aloud and are thus invisible to other characters.
Tables/prompts_eval_2.tex:13: How human-like and natural the characters behave
Tables/prompts_eval_2.tex:55: How well the characters match their established profiles from the book
Tables/prompts_eval_2.tex:63: (Only apply to the main characters: {major\_characters})
Tables/prompts_eval_2.tex:63: (Only apply to the main characters: {major\_characters})
Tables/prompts_eval_2.tex:67: ssions, and tone that are not appropriate for the characters' traits or social/educational background
Tables/prompts_eval_2.tex:89: * Interacts inappropriately with other characters regarding their background, relationship and social status
Tables/prompts_eval_2.tex:122: Whether the storyline and characters' reactions in the simulated conversation align well with th
Tables/prompts_eval_2.tex:132: * Characters' reactions (emotions, attitudes, behaviors) in the simulate

#### format: 47 hits; per file {'main.tex': 3, 'Sections/2_Related_Work.tex': 1, 'Sections/3_Dataset.tex': 2, 'Sections/4_Methodology.tex': 3, 'Sections/Appendix.tex': 10, 'Tables/case_critics.tex': 3, 'Tables/prompts_agent.tex': 2, 'Tables/prompts_data.tex': 5, 'Tables/prompts_data_2.tex': 9, 'Tables/prompts_data_3.tex': 6, 'Tables/prompts_eval.tex': 1, 'Tables/prompts_eval_2.tex': 2}
main.tex:236: \Crefformat{equation}{#2Eq.\;(#1)#3}
main.tex:238: \Crefformat{figure}{#2Figure #1#3}
main.tex:239: \Crefformat{assumption}{#2Assumption #1#3}
Sections/2_Related_Work.tex:15: riences~\citep{li2023chatharuhi} and multimodal information~\citep{dai2024mmrolecomprehensiveframeworkdeveloping}, \
Sections/3_Dataset.tex:41: )} Orthogonal: carrying distinct, complementary information with little redundancy;
Sections/3_Dataset.tex:59: houghts should be invisible to others, forming information asymmetry~\citep{zhou2024sotopia}.
Sections/4_Methodology.tex:30: incorporate role-playing instructions in diverse formats through instruction templates of varying formats.
Sections/4_Methodology.tex:30: formats through instruction templates of varying formats.
Sections/4_Methodology.tex:58: instructed to output in the speech-action-thought format.
Sections/Appendix.tex:59: we instruct LLMs to output extracted data in JSON format.
Sections/Appendix.tex:60: be unparseable or do not conform to the specified format (e.g., missing required keys).
Sections/Appendix.tex:71: ribute this to the LLM’s tendency to distribute information across different data fields when extracting multiple ki
Sections/Appendix.tex:71: t data fields when extracting multiple kinds of information simultaneously, rather than repeating it in different da
Sections/Appendix.tex:72: For instance, if certain information is already mentioned in the plot summary, it might be om
Sections/Appendix.tex:143: hod dataset into training samples in the Sharegpt format.
Sections/Appendix.tex:148: ructions (system prompts) using varied phrasings, formats, and data types, as shown in Tables ~\ref{tab:prompts_agen
Sections/Appendix.tex:149: ns entirely in natural language, as well as those formatted with special symbols (such as \#\#\#, ===), randomly sam
Sections/Appendix.tex:149: such as \#\#\#, ===), randomly sampling different formats and various expressions for the same semantics.
Sections/Appendix.tex:193: r questions, thus failing to provide sufficient information for personality assessment.
Tables/prompts_agent.tex:27: ===Information about the other Characters===
Tables/prompts_agent.tex:39: ===Relevant Background Information===
Tables/prompts_data.tex:23: uld be complete, covering the key dialogues and information. Each conversation should contain at least 10 utterances
Tables/prompts_data.tex:29: \quad v) Additionally, describe environmental information (such as scenes, atmosphere, sudden events, etc.) of the
Tables/prompts_data.tex:29: "character" field is set as "Environment". The information should exclude characters' active thoughts, observations
Tables/prompts_data.tex:35: ===Output Format===
Tables/prompts_data.tex:37: Please provide the output in the following JSON format:
Tables/prompts_data_2.tex:18: their attitudes, feelings, motivations, goals, information to convey or topics to be discussed",
Tables/prompts_data_2.tex:39: 1. Adhere strictly to the specified output JSON format.
Tables/prompts_data_2.tex:67: erform the scene. Focus on essential background information, while excluding future details to be portrayed in the c
Tables/prompts_data_2.tex:69: s, objectives, topics they want to discuss, and information they want to convey. Align with their established charac
Tables/prompts_data_2.tex:71: ===Output Format===
Tables/prompts_data_2.tex:73: Please provide the output in the following JSON format:
Tables/prompts_data_2.tex:77: <$ 200 words). Include all necessary background information while excluding future information to be revealed in the
Tables/prompts_data_2.tex:77: y background information while excluding future information to be revealed in the conversation.",
Tables/prompts_data_2.tex:80: ng their feelings, motivations, objectives, and information they want to convey or discuss." ...
Tables/prompts_data_3.tex:12: 1. Adhere strictly to the specified output JSON format.
Tables/prompts_data_3.tex:43: ===Output Format===
Tables/prompts_data_3.tex:45: Please provide the output in the following JSON format:
Tables/prompts_data_3.tex:62: The profile should be written in a concise yet informative style, similar to what one might find in a comprehensive
Tables/prompts_data_3.tex:62: guide, in {language}. Focus on the most crucial information that gives readers a clear understanding of the characte
Tables/prompts_data_3.tex:64: ting knowledge of the character or the provided information, without fabricating or inferring any inaccurate or unce
Tables/prompts_eval.tex:65: Provide your evaluation in JSON format:
Tables/prompts_eval_2.tex:74: * Includes future information beyond the character's current stage
Tables/prompts_eval_2.tex:109: eats others' viewpoints or previously mentioned information

#### JSON: 14 hits; per file {'Sections/Appendix.tex': 6, 'Tables/prompts_data.tex': 1, 'Tables/prompts_data_2.tex': 3, 'Tables/prompts_data_3.tex': 3, 'Tables/prompts_eval.tex': 1}
Sections/Appendix.tex:59: ion, we instruct LLMs to output extracted data in JSON format.
Sections/Appendix.tex:60: However, LLM-generated JSON strings may sometimes be unparseable or do not conform to t
Sections/Appendix.tex:63: cted, we invoke LLMs and attempt to parse a valid JSON object.
Sections/Appendix.tex:64: rsing fails, we employ LLMs to repair the invalid JSON string and retry.
Sections/Appendix.tex:65: failures occur because the LLM attempts to output JSON that exceeds the maximum token limit, resulting in truncati
Sections/Appendix.tex:66: ases, we prompt the repairing LLM to truncate the JSON at an appropriate point.
Tables/prompts_data.tex:37: Please provide the output in the following JSON format:
Tables/prompts_data_2.tex:39: 1. Adhere strictly to the specified output JSON format.
Tables/prompts_data_2.tex:60: \{json.dumps(truncated\_plots, ensure\_ascii=False, indent=2) if t
Tables/prompts_data_2.tex:73: Please provide the output in the following JSON format:
Tables/prompts_data_3.tex:12: 1. Adhere strictly to the specified output JSON format.
Tables/prompts_data_3.tex:36: {json.dumps(truncated\_plots, ensure\_ascii=False, indent=2) if t
Tables/prompts_data_3.tex:45: Please provide the output in the following JSON format:
Tables/prompts_eval.tex:65: Provide your evaluation in JSON format:

#### system: 16 hits; per file {'Sections/1_Introduction.tex': 1, 'Sections/2_Related_Work.tex': 1, 'Sections/3_Dataset.tex': 3, 'Sections/4_Methodology.tex': 1, 'Sections/5_Experiments.tex': 2, 'Sections/Appendix.tex': 4, 'Tables/prompts_agent.tex': 2, 'Tables/prompts_agent_2.tex': 2}
Sections/1_Introduction.tex:48: \textit{1)} create a multi-agent system to simulate a conversation $\bar{M}$, where the actor LLM
Sections/2_Related_Work.tex:6: racters~\citep{shao2023character} and multi-agent systems that simulate human society~\citep{Park2023GenerativeAgent
Sections/3_Dataset.tex:22: we build a multi-agent system for conversation simulation given the same scenario, and as
Sections/3_Dataset.tex:57: ammed to trigger downstream events in multi-agent systems.
Sections/3_Dataset.tex:69: We curate the \method dataset through a systematic LLM-based pipeline that transforms book content into hi
Sections/4_Methodology.tex:53: we build a multi-agent system to simulate a conversation $\bar{M}$, in the same setting a
Sections/5_Experiments.tex:49: on-cf3} shows experiment results when multi-agent systems start from the first $k=3$ original messages.
Sections/5_Experiments.tex:122: The retrieval system is based on FAISS~\citep{douze2024faiss} with BGE-M3~\citep
Sections/Appendix.tex:93: rsonaHub~\citep{chan2024personahub}. We provide a systematic comparison between \method and PersonaHub below.
Sections/Appendix.tex:146: ages as targets for optimization, while using the system prompt and messages from other characters as inputs. Adjace
Sections/Appendix.tex:148: e dynamically generate role-playing instructions (system prompts) using varied phrasings, formats, and data types, a
Sections/Appendix.tex:264: shows experiment results when multi-agent systems continue conversations from the first $k=3$ original messa
Tables/prompts_agent.tex:6: n{2}{c}{\textbf{Prompts for RPLAs and Multi-agent Systems}} \\
Tables/prompts_agent.tex:109: \caption{Prompts for RPLAs and multi-agent systems in \method. }
Tables/prompts_agent_2.tex:6: n{2}{c}{\textbf{Prompts for RPLAs and Multi-agent Systems}} \\
Tables/prompts_agent_2.tex:54: \caption{Prompts for RPLAs and multi-agent systems in \method. }

#### temperature: 0 hits; per file {}

#### top_p: 0 hits; per file {}

#### top_k: 0 hits; per file {}

#### max_tokens: 0 hits; per file {}

#### sampling: 5 hits; per file {'Sections/Appendix.tex': 5}
Sections/Appendix.tex:149: h special symbols (such as \#\#\#, ===), randomly sampling different formats and various expressions for the same sema
Sections/Appendix.tex:179: \subsection{Test Set Sampling}
Sections/Appendix.tex:183: We employ a weighted sampling strategy to prioritize well-established characters with mor
Sections/Appendix.tex:183: stablished characters with more persona data. The sampling process consists of the following steps:
Sections/Appendix.tex:185: all conversations by weight and perform weighted sampling from the top half of conversations with higher weights.

#### self-consistency: 0 hits; per file {}

#### majority: 0 hits; per file {}

#### mean: 10 hits; per file {'Sections/Appendix.tex': 1, 'Tables/case_cerci_2.tex': 1, 'Tables/case_cerci_4.tex': 1, 'Tables/case_critics.tex': 2, 'Tables/case_sansa.tex': 2, 'Tables/main_exp_cot.tex': 1, 'Tables/prompts_eval_2.tex': 1, 'Tables/top_books.tex': 1}
Sections/Appendix.tex:208: -1.5909 \times \text{rounds} + 59.0617 \), which means that for each additional round in the simulation, the scor
Tables/main_exp_cot.tex:21: I.T. and Mot. mean inner thoughts and motivations, respectively. }
Tables/prompts_eval_2.tex:105: * Shows unnatural progression or lacks meaningful developments
Tables/top_books.tex:47: \textbf{79}. \textit{{A Prayer for Owen Meany}} & \textbf{80}. \textit{{Clockwork Angel (The Infernal De

#### vote: 0 hits; per file {}

#### normalize: 0 hits; per file {}

#### scale: 5 hits; per file {'Sections/2_Related_Work.tex': 2, 'Sections/5_Experiments.tex': 2, 'Sections/Appendix.tex': 1}
Sections/2_Related_Work.tex:19: roved quality, but are expensive and difficult to scale.
Sections/2_Related_Work.tex:20: orts for individual sources and are hence hard to scale as well.
Sections/5_Experiments.tex:36: hod 8B similarly outperforms models of comparable scale.
Sections/5_Experiments.tex:72: LLMs on 20 samples each, using a 1 to 10 scoring scale, given the background context and original conversations.
Sections/Appendix.tex:165: al works, thereby acquiring data on a much larger scale.

#### average: 16 hits; per file {'Sections/5_Experiments.tex': 6, 'Sections/Appendix.tex': 3, 'Tables/exp_idood.tex': 2, 'Tables/exp_main_cot_full.tex': 1, 'Tables/exp_rag.tex': 1, 'Tables/main_exp.tex': 1, 'Tables/main_exp_cf3.tex': 1, 'Tables/main_exp_cot.tex': 1}
Sections/5_Experiments.tex:13: t LLM-judged scores for each dimension, and their average as the overall score.
Sections/5_Experiments.tex:34: ts shown in Table~\ref{tab:model-comparison} are averaged across three runs, from which we observe that:
Sections/5_Experiments.tex:38: d 70B demonstrate superior performance, achieving average scores above 57\%;
Sections/5_Experiments.tex:51: For example, the average score gap between Qwen-2 72B and 7B decreases from 11.5\% (
Sections/5_Experiments.tex:73: 15 minutes to evaluate 7 models for one sample on average.
Sections/5_Experiments.tex:74: We report models' average scores and win rate against other models.
Sections/Appendix.tex:12: On average, each conversation consists of approximately 13.2 utterance
Sections/Appendix.tex:185: we calculate the weight of a conversation as the average of its characters’ weights, including both main characters
Sections/Appendix.tex:301: sults demonstrate that GCA simulation achieves an average score of 59.2\%, significantly outperforming HollmWood's 50
Tables/exp_idood.tex:12: } & \multirow{2}{*}{\makecell{\scriptsize \textbf{Average}\\ \scriptsize \textbf{Score}}} & \multirow{2}{*}{\makecel
Tables/exp_idood.tex:12: } & \multirow{2}{*}{\makecell{\scriptsize \textbf{Average}\\ \scriptsize \textbf{Score}}} \\
Tables/exp_main_cot_full.tex:8: bf{Quality}}} & \multirow{2}{*}{\makecell{\textbf{Average}\\\textbf{Score}}} & \multirow{2}{*}{\makecell{\textbf{BLEU
Tables/exp_rag.tex:8: bf{Quality}}} & \multirow{2}{*}{\makecell{\textbf{Average}\\\textbf{Score}}} & \multirow{2}{*}{\makecell{\textbf{BLEU
Tables/main_exp.tex:8: bf{Quality}}} & \multirow{2}{*}{\makecell{\textbf{Average}\\\textbf{Score}}} & \multirow{2}{*}{\makecell{\textbf{BLEU
Tables/main_exp_cf3.tex:9: bf{Quality}}} & \multirow{2}{*}{\makecell{\textbf{Average}\\\textbf{Score}}} & \multirow{2}{*}{\makecell{\textbf{BLEU
Tables/main_exp_cot.tex:20: Ablation study results (average scores) on \method Test.

#### AVG: 0 hits; per file {}

#### filter: 0 hits; per file {}

#### exclude: 5 hits; per file {'Sections/5_Experiments.tex': 2, 'Sections/Appendix.tex': 2, 'Tables/prompts_data.tex': 1}
Sections/5_Experiments.tex:9: We exclude characters' inner thoughts for LLM critics, and set the con
Sections/5_Experiments.tex:39: separately for test splits from books included or excluded for \method training.
Sections/Appendix.tex:151: Each sample may include (50\%) or exclude (50\%) the following elements :
Sections/Appendix.tex:256: of LLMs on test splits from books included in and excluded from \method training.
Tables/prompts_data.tex:29: d is set as "Environment". The information should exclude characters' active thoughts, observations, and actions.

#### retry: 2 hits; per file {'Sections/Appendix.tex': 2}
Sections/Appendix.tex:62: Towards this challenge, we adopt a repair-and-retry strategy to improve extraction success rate.
Sections/Appendix.tex:64: employ LLMs to repair the invalid JSON string and retry.

#### fail: 6 hits; per file {'Sections/Appendix.tex': 5, 'Tables/prompts_eval_2.tex': 1}
Sections/Appendix.tex:64: If parsing fails, we employ LLMs to repair the invalid JSON string and retr
Sections/Appendix.tex:65: Some failures occur because the LLM attempts to output JSON that exce
Sections/Appendix.tex:67: If it still fail, we restart the entire process from the beginning, making u
Sections/Appendix.tex:70: luding scenarios and character motivations, often fail to provide a comprehensive context.
Sections/Appendix.tex:193: oo short or may decline to answer questions, thus failing to provide sufficient information for personality asses
Tables/prompts_eval_2.tex:72: * Fails to demonstrate character-specific knowledge, background or

#### translation: 0 hits; per file {}

#### version: 4 hits; per file {'Sections/5_Experiments.tex': 4}
Sections/5_Experiments.tex:20: Doubao-pro~\footnote{https://team.doubao.com/en/ Version:241215}, Step-1-Flash and Step-2~\footnote{https://www.step
Sections/5_Experiments.tex:20: ash and Step-2~\footnote{https://www.stepfun.com/ Version: 241111 for Step-1-Flash, 241223 for Step-2 (internal).}, G
Sections/5_Experiments.tex:20: ~\footnote{https://openai.com/index/hello-gpt-4o/ Version:240806}, Gemini-1.5-Pro~\footnote{https://deepmind.google/t
Sections/5_Experiments.tex:20: e-3.5-Sonnet~\footnote{https://www.anthropic.com/ Version: 20240307 for Claude-3-Haiku, 20240620 for Claude-3.5-Sonne

#### date: 2 hits; per file {'Sections/5_Experiments.tex': 1, 'Sections/Appendix.tex': 1}
Sections/5_Experiments.tex:68: to further validate the performance of \method models and the reliability of LL
Sections/Appendix.tex:273: We validate the effectiveness of \method’s comprehensive data types for

#### seed: 0 hits; per file {}