"""CharacterEval audit bundle: runs every check behind the reproduction report with a log per step, and assembles what a
third party needs to check and re-run them, for publication with the report (web/public/audit/charactereval/<date>/).

Steps (each logged with its command, start and end time, exit status and full output):
  01 spec_check     the platform's protocol module against the official code (26 checks; V01–V04, V08, M01–M03)
  02 paper          Table 4 read again from the LaTeX source; the platform's published.ts must be unchanged by it
  03 table_checks   the paper's own numbers: counts, correlation table, 90 dimension averages, marks, back-testing
  04 rm_probes      the 26 reward-model probes rebuilt from the official repository; their SHA-256 must equal the one
                    charactereval_rm_server_check.py requires
  05 release_checks the released files: formats, the reward-model training data, the frozen outputs, weighted averages
  07 anchor_diagnostics  (with ANCHOR) the anchor's cell differences against sample sizes, implied per-score noise,
                    pairwise orders and the cells beyond 0.005, recomputed from the published scores
  08 rm_numerics_verify  (with ANCHOR) the operator's numerical-sensitivity result (V07a) checked on our side: its inputs
                    against the repository, its root-mean-square changes recomputed, its bfloat16 scores against the anchor's

Published (no test-set text, no profile text, no model reply; the data stay with their authors):
  code/      the scripts and the platform source they execute, as run
  results/   spec_check.txt, table_checks.json, probes_manifest.json (one SHA-256 per probe text, no text); with ANCHOR,
             anchor/ (anchor.json and the 40,160 scores, as numbers indexed by row of the released files) and
             anchor_diagnostics.json; charactereval_rm_numerics_check.json (the operator's V07a result, host name left out)
             and rm_numerics_verify.json
  reviews/   the three independent reviews of the paper, the code and the released files (Chinese), as written, except
             that every line quoting test-set text, a profile or a released model output is replaced by a marker
  logs/      one log per step
  SHA256SUMS every file of the bundle

    python3 -I tools/experiments/charactereval_audit_bundle.py REPO EPRINT REVIEWS OUT [E2E [ANCHOR]]

REPO: morecry/CharacterEval at c3d44a6; EPRINT: the arXiv v2 e-print directory (acl_latex.tex); REVIEWS: the directory
holding charactereval-pass{1,2,3}-*.md; OUT: the bundle directory; E2E: the directory of the local end-to-end check
(check_ce_run.py, mock_server.py), whose scripts are published without their outputs, which were not kept; ANCHOR: the work
directory of charactereval_anchor.py (anchor.json, scores_<name>.jsonl).
"""
import datetime
import hashlib
import html
import json
import os
import platform
import shutil
import subprocess
import sys

ROOT = os.path.abspath(os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", ".."))
CODE = [
    "tools/experiments/charactereval_reference.py",
    "tools/experiments/charactereval_spec_check.py",
    "tools/experiments/charactereval_protocol_dump.ts",
    "tools/experiments/charactereval_paper.py",
    "tools/experiments/charactereval_table_checks.py",
    "tools/experiments/charactereval_release_checks.py",
    "tools/experiments/charactereval_rm_probes.py",
    "tools/experiments/charactereval_rm_server_check.py",
    "tools/experiments/charactereval_anchor.py",
    "tools/experiments/charactereval_anchor_diagnostics.py",
    "tools/experiments/charactereval_rm_numerics_verify.py",
    "tools/experiments/charactereval_rm_numerics_check.py",
    "tools/experiments/charactereval_audit_bundle.py",
    "worker/src/benchmarks/charactereval/index.ts",
    "worker/src/benchmarks/charactereval/meta.ts",
    "worker/src/benchmarks/charactereval/protocol.ts",
    "worker/src/benchmarks/charactereval/published.ts",
    "worker/src/benchmarks/charactereval/strings.gen.ts",
    "worker/src/llm/rewards.ts",
    "worker/src/engine/runner.ts",
    "worker/src/engine/results.ts",
    "worker/src/scoring.ts",
    "docs/复现规范/CharacterEval-核查矩阵.md",
    "docs/复现规范/BaichuanCharRM-部署校验说明.md",
    "tools/experiments/charactereval_third_party_search.md",
]
REVIEWS = {
    "charactereval-pass1-paper-first.md": "Review 1: from the paper to the code.",
    "charactereval-pass2-code-first.md": "Review 2: from the code to the paper.",
    "charactereval-pass3-numbers.md": "Review 3: from the released files and the paper's numbers.",
}
PLACES = {}


def sha256(path):
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(1 << 20), b""):
            h.update(chunk)
    return h.hexdigest()


def public(text):
    for path, label in sorted(PLACES.items(), key=lambda x: -len(x[0])):
        text = text.replace(path, label)
    return text


def dataset_strings(repo):
    """Strings of the authors' data that must not be republished: dialogue utterances, profile values, released outputs."""
    out = set()
    for x in json.load(open(os.path.join(repo, "data/test_data.jsonl"), encoding="utf-8")):
        for line in x["context"].split("\n"):
            text = line.split("：", 1)[-1].strip()
            if len(text) >= 6:
                out.add(text[:24])
    def walk(v):
        if isinstance(v, str):
            if len(v) >= 10:
                out.add(v[:24])
        elif isinstance(v, dict):
            for w in v.values():
                walk(w)
        elif isinstance(v, list):
            for w in v:
                walk(w)
    walk(json.load(open(os.path.join(repo, "data/character_profiles.json"), encoding="utf-8")))
    for name in os.listdir(os.path.join(repo, "results")):
        if name.startswith("generation_trans_"):
            for x in json.load(open(os.path.join(repo, "results", name), encoding="utf-8")):
                if x["model_output"] and len(x["model_output"]) >= 8:
                    out.add(x["model_output"][:24])
    return out


def redact(text, strings):
    lines, n = [], 0
    for line in text.split("\n"):
        if any(s in line for s in strings):
            lines.append("[line removed: it quoted the authors' test set, a profile or a released model output]")
            n += 1
        else:
            lines.append(line)
    return "\n".join(lines), n


def run(logs, n, name, cmd):
    start = datetime.datetime.now(datetime.timezone.utc)
    r = subprocess.run(cmd, cwd=ROOT, capture_output=True, text=True)
    end = datetime.datetime.now(datetime.timezone.utc)
    with open(os.path.join(logs, f"{n:02d}-{name}.log"), "w", encoding="utf-8") as f:
        f.write(public(f"$ {' '.join(['python3 -I' if c == sys.executable else c for c in cmd if c != '-I'])}\n# started {start.isoformat(timespec='seconds')}  ended {end.isoformat(timespec='seconds')}  exit {r.returncode}\n\n"))
        f.write(public(r.stdout))
        if r.stderr.strip():
            f.write(public("\n# stderr\n" + "\n".join(l for l in r.stderr.splitlines() if "ExperimentalWarning" not in l and "MODULE_TYPELESS" not in l)))
    print(f"{n:02d} {name}: exit {r.returncode}", flush=True)
    if r.returncode != 0:
        raise SystemExit(f"step {name} failed; see its log")
    return r.stdout


def main(repo, eprint, reviews, out, e2e=None, anchor_work=None):
    repo, eprint, reviews, out = (os.path.abspath(p) for p in (repo, eprint, reviews, out))
    if e2e:
        PLACES[os.path.abspath(e2e)] = "<e2e>"
    tmp = os.path.join(os.path.dirname(out), f".work-{os.getpid()}")
    PLACES.update({tmp: "<work>", out: "<bundle>", repo: "<repo>", eprint: "<eprint>", reviews: "<reviews>", ROOT: "<platform>", os.path.expanduser("~"): "<home>"})
    for d in ("code", "results", "reviews", "logs"):
        os.makedirs(os.path.join(out, d), exist_ok=True)
    logs, res = os.path.join(out, "logs"), os.path.join(out, "results")
    os.makedirs(tmp, exist_ok=True)
    py = [sys.executable, "-I"]
    with open(os.path.join(logs, "00-environment.log"), "w") as f:
        f.write(f"generated {datetime.datetime.now(datetime.timezone.utc).isoformat(timespec='seconds')}\n")
        f.write(f"python {platform.python_version()}  node {subprocess.run(['node', '--version'], capture_output=True, text=True).stdout.strip()}  {platform.platform()}\n")
        f.write(f"repository commit {subprocess.run(['git', '-C', repo, 'rev-parse', 'HEAD'], capture_output=True, text=True).stdout.strip()}\n")
        f.write(f"e-print acl_latex.tex sha256 {sha256(os.path.join(eprint, 'acl_latex.tex'))}\n")
        f.write(f"platform commit {subprocess.run(['git', '-C', ROOT, 'rev-parse', 'HEAD'], capture_output=True, text=True).stdout.strip()}\n")

    shutil.copy(os.path.join(logs, "00-environment.log"), os.path.join(tmp, "env.log"))
    spec = run(logs, 1, "spec_check", py + ["tools/experiments/charactereval_spec_check.py", repo])
    open(os.path.join(res, "spec_check.txt"), "w").write(public(spec))
    published = [os.path.join(ROOT, s, "src/benchmarks/charactereval/published.ts") for s in ("worker", "server")]
    before = [sha256(p) for p in published]
    run(logs, 2, "paper", py + ["tools/experiments/charactereval_paper.py", eprint])
    after = [sha256(p) for p in published]
    with open(os.path.join(logs, "02-paper.log"), "a") as f:
        f.write(f"\npublished.ts (worker, server) SHA-256 before {before}, after {after}: {'unchanged' if before == after else 'CHANGED'}\n")
    if before != after:
        raise SystemExit("published.ts changed: the paper extraction no longer reproduces the leaderboard data")
    run(logs, 3, "table_checks", py + ["tools/experiments/charactereval_table_checks.py", eprint, os.path.join(res, "table_checks.json")])
    probes = os.path.join(tmp, "probes.json")
    run(logs, 4, "rm_probes", py + ["tools/experiments/charactereval_rm_probes.py", repo, probes])
    sys.path.insert(0, os.path.join(ROOT, "tools/experiments"))
    import charactereval_rm_server_check as chk  # noqa: E402
    got = sha256(probes)
    with open(os.path.join(logs, "04-rm_probes.log"), "a") as f:
        f.write(f"\nprobe file SHA-256 {got}; required by charactereval_rm_server_check.py: {chk.PROBES_SHA256}: {'equal' if got == chk.PROBES_SHA256 else 'DIFFERENT'}\n")
    if got != chk.PROBES_SHA256:
        raise SystemExit("the rebuilt probes differ from the ones the server check requires")
    manifest = [{k: p[k] for k in ("n", "kind", "id", "metric_en")} | {"chars": len(p["text"]), "text_sha256": hashlib.sha256(p["text"].encode()).hexdigest()}
                for p in json.load(open(probes, encoding="utf-8"))["probes"]]
    json.dump({"probes_sha256": got, "probes": manifest}, open(os.path.join(res, "probes_manifest.json"), "w"), indent=1)
    if anchor_work:
        os.makedirs(os.path.join(res, "anchor"), exist_ok=True)
        for name in sorted(os.listdir(anchor_work)):
            if name == "anchor.json" or (name.startswith("scores_") and name.endswith(".jsonl")):
                shutil.copy(os.path.join(anchor_work, name), os.path.join(res, "anchor", name))
        published_anchor = os.path.join(ROOT, "tools/experiments/charactereval_anchor.result.json")
        if sha256(published_anchor) != sha256(os.path.join(res, "anchor", "anchor.json")):
            raise SystemExit("the anchor work directory does not hold the committed anchor result")
    operator = os.path.join(ROOT, "tools/experiments/charactereval_rm_server_check.result.json")
    if os.path.exists(operator):
        shutil.copy(operator, os.path.join(res, "charactereval_rm_server_check.json"))
    numerics = os.path.join(ROOT, "tools/experiments/charactereval_rm_numerics_check.result.json")
    if os.path.exists(numerics):
        shutil.copy(numerics, os.path.join(res, "charactereval_rm_numerics_check.json"))
    run(logs, 5, "release_checks", py + ["tools/experiments/charactereval_release_checks.py", repo, eprint, os.path.join(res, "release_checks.json")])
    if anchor_work:
        run(logs, 7, "anchor_diagnostics", py + ["tools/experiments/charactereval_anchor_diagnostics.py", repo, os.path.join(res, "anchor"),
                                                 os.path.join(res, "anchor_diagnostics.json")])
        if os.path.exists(numerics):
            run(logs, 8, "rm_numerics_verify", py + ["tools/experiments/charactereval_rm_numerics_verify.py", repo,
                                                     os.path.join(res, "charactereval_rm_numerics_check.json"), os.path.join(res, "anchor"),
                                                     os.path.join(res, "rm_numerics_verify.json")])

    for rel in CODE:
        dst = os.path.join(out, "code", rel)
        os.makedirs(os.path.dirname(dst), exist_ok=True)
        shutil.copy(os.path.join(ROOT, rel), dst)
    if e2e:
        os.makedirs(os.path.join(out, "code/e2e"), exist_ok=True)
        for name in ("check_ce_run.py", "mock_server.py"):
            open(os.path.join(out, "code/e2e", name), "w").write(public(open(os.path.join(e2e, name)).read()))
    strings = dataset_strings(repo)
    removed = {}
    for name in REVIEWS:
        text, removed[name] = redact(public(open(os.path.join(reviews, name), encoding="utf-8").read()), strings)
        open(os.path.join(out, "reviews", name), "w", encoding="utf-8").write(text)
    with open(os.path.join(logs, "06-redaction.log"), "w") as f:
        f.write(f"lines removed from the reviews because they quote the authors' data ({len(strings):,} distinct strings checked): {removed}\n")
    shutil.rmtree(tmp)

    files = sorted(os.path.relpath(os.path.join(d, f), out) for d, _, fs in os.walk(out) for f in fs if f not in ("SHA256SUMS", "index.html"))
    with open(os.path.join(out, "SHA256SUMS"), "w") as f:
        for rel in files:
            f.write(f"{sha256(os.path.join(out, rel))}  {rel}\n")
    describe = {"code": "The check scripts and the platform source they execute, exactly as run.",
                "code/e2e/check_ce_run.py": "Local end-to-end check of a CharacterEval run against the official logic (its output was not kept).",
                "code/e2e/mock_server.py": "The mock model and reward-model endpoint used by the local end-to-end check.", "logs": "One log per step: command, start and end time, exit status and full output.",
                "results/spec_check.txt": "The 26 checks of the protocol module against the official code.", "results/table_checks.json": "The paper's own numbers, checked mechanically.",
                "results/charactereval_rm_server_check.json": "The operator's deployment check of the reward model (V06), as returned on 2026-10-28, with the host name and endpoint address left out.",
                "results/anchor/anchor.json": "The anchor (V07, 2026-10-28 19:05 UTC): the platform's endpoint on the 26 probes and on the five released outputs (5 x 8,032 inputs), against the paper's Table 4. Verdict: within one step.",
                "results/anchor": "The anchor's scores: one line per reward-model input, {i: row of the released file, score}; numbers only.",
                "results/anchor_diagnostics.json": "The anchor's differences against sample sizes, the per-score noise they imply, pairwise orders, and the cells beyond 0.005.",
                "code/tools/experiments/charactereval_rm_numerics_check.py": "The numerical-sensitivity check (V07a), run by the reward model's operator; criteria fixed before any result.",
                "results/charactereval_rm_numerics_check.json": "The operator's numerical-sensitivity check (V07a), as returned on 2026-10-29 (11:19 UTC), with the host name left out: 1,200 scores in bfloat16, strict bfloat16 and float32. Verdict: numerics suffice.",
                "results/rm_numerics_verify.json": "V07a checked on our side: inputs against the repository, root-mean-square changes recomputed, bfloat16 scores against the anchor's.",
                "results/release_checks.json": "The released files: formats, reward-model training data, frozen outputs, weighted averages.",
                "results/probes_manifest.json": "The 26 reward-model probes: sample, metric, length and SHA-256 of each text (the texts are rebuilt from the repository).",
                "code/tools/experiments/charactereval_third_party_search.md": "The search for later work reporting CharacterEval scores (Section 9): queries, sources, what was verified."}
    describe.update({f"reviews/{k}": v for k, v in REVIEWS.items()})
    desc = lambda rel: next((describe[k] for k in ("/".join(rel.split("/")[:i]) for i in range(rel.count("/") + 1, 0, -1)) if k in describe), "")
    rows = "".join(f'<tr><td><a href="{html.escape(rel)}">{html.escape(rel)}</a></td><td>{os.path.getsize(os.path.join(out, rel)):,}</td><td><code>{sha256(os.path.join(out, rel))[:16]}…</code></td><td>{html.escape(desc(rel))}</td></tr>' for rel in files)
    page = f"""<!doctype html><html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>CharacterEval reproduction audit bundle · SociaBench</title>
<meta name="description" content="Re-runnable checks of the SociaBench CharacterEval reproduction: scripts, results, independent reviews and execution logs.">
<style>body{{font:15px/1.55 system-ui,sans-serif;max-width:1100px;margin:32px auto;padding:0 16px;color:#111;background:#fff}}
table{{border-collapse:collapse;width:100%;font-size:13px}}td,th{{border-bottom:1px solid #ddd;padding:5px 8px;text-align:left;vertical-align:top}}
code,pre{{font:12.5px ui-monospace,monospace;background:#f4f4f4;padding:1px 4px}}pre{{padding:10px;overflow-x:auto}}
@media (prefers-color-scheme:dark){{body{{color:#eee;background:#111}}code,pre{{background:#222}}td,th{{border-color:#333}}a{{color:#8ab4ff}}}}</style></head><body>
<h1>CharacterEval reproduction audit bundle</h1>
<p>Everything behind Sections 2, 3, 5 and 6 of the <a href="/b/charactereval/methodology">CharacterEval reproduction report</a>: the check scripts
and the platform source as run, their results and logs, and the three independent reviews of the paper, the code and the released files. Nothing
here contains test-set text, a character profile or a model's reply: the data stay with their authors (github.com/morecry/CharacterEval);
lines of the reviews that quoted them are replaced by a marker (logs/06-redaction.log).</p>
<h2>Check without running anything</h2>
<pre>shasum -a 256 -c SHA256SUMS      # in a copy of this directory</pre>
<h2>Re-run the checks</h2>
<pre>git clone https://github.com/morecry/CharacterEval repo &amp;&amp; git -C repo checkout c3d44a6
curl -L -o eprint.tar.gz https://arxiv.org/e-print/2401.01275v2 &amp;&amp; mkdir eprint &amp;&amp; tar xzf eprint.tar.gz -C eprint
python3 -I code/tools/experiments/charactereval_spec_check.py repo          # needs node and the platform source tree
python3 -I code/tools/experiments/charactereval_table_checks.py eprint
python3 -I code/tools/experiments/charactereval_release_checks.py repo eprint
python3 -I code/tools/experiments/charactereval_rm_probes.py repo probes.json
python3 -I code/tools/experiments/charactereval_anchor_diagnostics.py repo results/anchor
python3 -I code/tools/experiments/charactereval_rm_numerics_verify.py repo results/charactereval_rm_numerics_check.json results/anchor</pre>
<p>The reward-model deployment check (<code>charactereval_rm_server_check.py</code>) runs on the server that serves the model; the operator's
result is <a href="results/charactereval_rm_server_check.json">results/charactereval_rm_server_check.json</a> (all criteria met on 2026-10-28).
The anchor (<code>charactereval_anchor.py</code>) scored the five released outputs through the platform's endpoint; its result and scores are in
<a href="results/anchor/">results/anchor/</a>. Its verdict under the criteria fixed in advance is <i>within one step</i>. The numerical-sensitivity check that examines the cause
(<code>charactereval_rm_numerics_check.py</code>, V07a), run by the operator on 2026-10-29, finds that numerics alone move single scores by a
root-mean-square of 0.095 (strict bfloat16 accumulation) and 0.262 (float32), above the 0.06 fixed as sufficient to explain the anchor; its
bfloat16 scores equal the anchor's for all 1,200 inputs (<a href="results/charactereval_rm_numerics_check.json">result</a>,
<a href="results/rm_numerics_verify.json">our check</a>).</p>
<h2>Files</h2><table><tr><th>File</th><th>Bytes</th><th>SHA-256</th><th>Content</th></tr>{rows}</table>
</body></html>
"""
    open(os.path.join(out, "index.html"), "w").write(page)
    print(f"{len(files)} files")


if __name__ == "__main__":
    main(*sys.argv[1:7])