sha256 eb175779d1d5adf99c893ff1f82de9b18ada912d3f385b3f5419195afdfb374a
scenes 200
id/ood sizes 100 100
fields only in test_set vs test_data {'tag': 200, 'i_p': 200} differing common values 0 {}
top-level fields ['book', 'character_profiles', 'dialogues', 'i_c', 'i_p', 'key_characters', 'major_characters', 'plot', 'scenario', 'speaking_characters_w_env', 'tag', 'topic', 'weight']
tag Counter({'id': 100, 'ood': 100})
books 101
id books 69 ood books 32 both 0
{'id': 69, 'ood': 32}
speaking per scene 1 5 total 479 mean 2.395
Environment listed 130 appended 70
lines 9 32 median 12.0 >20 7
key_characters entries 456 with thought 456 nonempty thought 456 with motivation field 0
speaking with motivation 451 without 28 speaking without profile text 13
major 1 5
scenes where first listed speaker is Environment 4
held_out type dict len 2
held-out books 2 overlap with test books 0
scenes in overlap books 0
example scenes 3
example in 200 0 example books in test 0
exp/simulation/test_set_gpt-4o.json scenes 3 in 200 0 speech msgs 38 prefixed at start 38 any line 38 same as example 3
exp/simulation/test_set_gpt-4o-mini_rag=expr3_conv1.json scenes 3 in 200 0 speech msgs 46 prefixed at start 21 any line 46 same as example 3
exp/simulation/test_set_gpt-4o_rag=expr3_conv1.json scenes 3 in 200 0 speech msgs 42 prefixed at start 31 any line 42 same as example 3
eval keys ['scores', 'cases'] cases 3 {'Storyline Consistency': 66.1667, 'Anthropomorphism': 52.8333, 'Character Fidelity': 51.1667, 'Storyline Quality': 79.5, 'avg': 62.4167, 'bleu': 0.0253, 'rouge_l': 0.0839}
dimension scores 12 {'5S+1.5R': 12, 'S+1.5R': 0, '5S': 0, '5S+1.5all': 0, '5S+1.5all_noclip': 0, 'S_unclipped_ok': 0} out of bounds before clip 0 R,all per case [(11, 14), (5, 8), (13, 16)]
rouge [0.07246376532209046, 0.1059001475702016, 0.0732700110099804] bleu [0.021624838733610124, 0.023715767261473588, 0.030483934319011176]
eval sim prefix 38 38