{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2608-08160","title":"Can LLM Agents Stick to the Script? A Benchmark for Long-Horizon Consistency in Interactive Narratives","arxiv_id":"2608.08160","date":"2026-08-08","proceeding":"ICML","authors":["Yingpeng Ma","Jianhao Yan","Bei Shi","Ka Hou Kam","Runnan Wang","Xuebo Liu","Yulong Chen","Yue Zhang","Derek F. Wong"],"abstract":"The rapid advancement of Large Language Models (LLMs) is revolutionizing AI for Games by enabling open-ended and fluid interactive storytelling. However, existing research has largely overlooked the critical challenge of maintaining long-horizon logical consistency and narrative integrity against unconstrained user interventions. To address this, we formulate this challenge as Narrative Commitment Preservation (NCP), and take interactive narrative as our testbed. We introduce NCP-Bench, a benchmark of 100 narrative environments derived from movie synopses. Each environment includes a structured narrative specification (trajectory, commitments, and initial facts) that we can automatically check throughout the interaction between the player agent and the narrator agent. Experiments across state-of-the-art LLMs reveal a substantial long-horizon consistency gap: high linguistic quality does not guarantee commitment preservation; even strong models frequently generate logically conflicting content under adversarial interventions, with the best-performing model (GPT-5.2) achieving only 42% survival rate after 20 turns and fact conflict rates ranging from 40% to 68% across models, and only isolated runs satisfying all achievement commitments within the 100-turn limit.","url_abs":"https://arxiv.org/abs/2608.08160","url_pdf":"https://arxiv.org/pdf/2608.08160","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2608.08160","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2608.08160"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/yingpengma/NCP-Bench","reach":null}],"summary":{"ran":23,"ran_draft_wrong":45,"ran_honours":3,"ran_violates":1,"unverified":18},"by_repo_kind":{"found_in_text":{"samples":90,"ran":72,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6baefa8c03c8073b","entry":"Commitment","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6baefa8c03c8073b"}},{"code_sha256_prefix":"477d617508a01ed1","entry":"CommitmentAssessment","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"477d617508a01ed1"}},{"code_sha256_prefix":"225c1e1d6aa8ef52","entry":"CommitmentCheckRequest","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"225c1e1d6aa8ef52"}},{"code_sha256_prefix":"9275d276c18d1265","entry":"Conflict","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9275d276c18d1265"}},{"code_sha256_prefix":"7e4e25bf99b30656","entry":"ConflictDecision","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e4e25bf99b30656"}},{"code_sha256_prefix":"403039263cff2dee","entry":"EpisodeCheckpoint","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"403039263cff2dee"}},{"code_sha256_prefix":"26e59d268abd3a42","entry":"EpisodeContext","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"26e59d268abd3a42"}},{"code_sha256_prefix":"d0aa116f6e67740f","entry":"EpisodeSession","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d0aa116f6e67740f"}},{"code_sha256_prefix":"a4d578af7dc4366a","entry":"EpisodeState","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a4d578af7dc4366a"}},{"code_sha256_prefix":"cf5ff3d591cc004d","entry":"EpisodeTransition","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cf5ff3d591cc004d"}},{"code_sha256_prefix":"a397eb49bee1be3b","entry":"EvaluationPrompts","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a397eb49bee1be3b"}},{"code_sha256_prefix":"0b6ddc7118f4a20c","entry":"Fact","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0b6ddc7118f4a20c"}},{"code_sha256_prefix":"cf34436e377a3111","entry":"FactUpdate","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cf34436e377a3111"}},{"code_sha256_prefix":"dc74c41ca23b2db8","entry":"NarratorRequest","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dc74c41ca23b2db8"}},{"code_sha256_prefix":"5c2ee2ba5d267cdf","entry":"NarratorResponse","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5c2ee2ba5d267cdf"}},{"code_sha256_prefix":"ef12c1f61aaa23e9","entry":"OpeningRequest","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ef12c1f61aaa23e9"}},{"code_sha256_prefix":"d9d593f13f3d399f","entry":"PendingFactUpdates","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d9d593f13f3d399f"}},{"code_sha256_prefix":"48a18311c6707329","entry":"StoryTurn","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"48a18311c6707329"}},{"code_sha256_prefix":"3c29c0dff5302cdc","entry":"TrajectoryAssessment","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c29c0dff5302cdc"}},{"code_sha256_prefix":"05547bdfc564737a","entry":"TrajectoryCheckRequest","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"05547bdfc564737a"}},{"code_sha256_prefix":"a4a0ea3a436605c0","entry":"TrajectoryNode","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a4a0ea3a436605c0"}},{"code_sha256_prefix":"ae6e086b9871bf9c","entry":"TurnAuditContext","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ae6e086b9871bf9c"}},{"code_sha256_prefix":"96ded29e4b2b6edb","entry":"TurnEvaluation","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"96ded29e4b2b6edb"}},{"code_sha256_prefix":"87a83c6c64d04d33","entry":"_apply_commitment_assessments","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"87a83c6c64d04d33"}},{"code_sha256_prefix":"881b3ff52990ad4f","entry":"_apply_trajectory_assessments","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"881b3ff52990ad4f"}},{"code_sha256_prefix":"03e084c6b85cff45","entry":"_build_audit_context","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"03e084c6b85cff45"}},{"code_sha256_prefix":"3d68030ecaab9187","entry":"_conflict_items","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3d68030ecaab9187"}},{"code_sha256_prefix":"0b571550ec2f8bd1","entry":"_conflict_variables","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0b571550ec2f8bd1"}},{"code_sha256_prefix":"e78fa6a936db28b3","entry":"_conflicts_for_review","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e78fa6a936db28b3"}},{"code_sha256_prefix":"bb147fbdb564ab08","entry":"_current_node_id","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bb147fbdb564ab08"}},{"code_sha256_prefix":"f48d2b516188201f","entry":"_current_trajectory_node","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f48d2b516188201f"}},{"code_sha256_prefix":"4699f523e541d887","entry":"_first_pending_node_index","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4699f523e541d887"}},{"code_sha256_prefix":"77812a1aa0267b64","entry":"_format_active_facts","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"77812a1aa0267b64"}},{"code_sha256_prefix":"39b6c97dcbe721ff","entry":"_format_checker_commitments","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"39b6c97dcbe721ff"}},{"code_sha256_prefix":"ddbf1846483032a8","entry":"_format_checker_facts","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ddbf1846483032a8"}},{"code_sha256_prefix":"031f59d0efdc193a","entry":"_format_checker_history","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"031f59d0efdc193a"}},{"code_sha256_prefix":"308388e14916159e","entry":"_format_checker_node","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"308388e14916159e"}},{"code_sha256_prefix":"ad2949a66ba46899","entry":"_format_checker_trajectory","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ad2949a66ba46899"}},{"code_sha256_prefix":"9d6c46c34b72e542","entry":"_format_commitments","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9d6c46c34b72e542"}},{"code_sha256_prefix":"8ac58d2d21e6dd5f","entry":"_format_current_node","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8ac58d2d21e6dd5f"}},{"code_sha256_prefix":"4eabff48b3f6c214","entry":"_format_fact_update","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4eabff48b3f6c214"}},{"code_sha256_prefix":"ab51c15d8e068455","entry":"_format_facts","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ab51c15d8e068455"}},{"code_sha256_prefix":"a496e6b2203db33d","entry":"_format_history","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a496e6b2203db33d"}},{"code_sha256_prefix":"6374f68069f53487","entry":"_format_node","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6374f68069f53487"}},{"code_sha256_prefix":"b9cc93c37c18d98c","entry":"_format_trajectory","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b9cc93c37c18d98c"}},{"code_sha256_prefix":"bd7ac87f8e14dbfc","entry":"_next_fact_id","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd7ac87f8e14dbfc"}},{"code_sha256_prefix":"f4dc8710f789a065","entry":"_next_turn_id","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f4dc8710f789a065"}},{"code_sha256_prefix":"e7de95262c944927","entry":"_normalize_facts","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7de95262c944927"}},{"code_sha256_prefix":"35093190a7dd3b9b","entry":"_optional_string","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"35093190a7dd3b9b"}},{"code_sha256_prefix":"48d4ada89ebbfef0","entry":"_parse_commitment_statuses","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"48d4ada89ebbfef0"}},{"code_sha256_prefix":"a9f899432b8e62db","entry":"_parse_conflict","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a9f899432b8e62db"}},{"code_sha256_prefix":"676a716e20a2dbf6","entry":"_parse_conflict_decision","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"676a716e20a2dbf6"}},{"code_sha256_prefix":"800e659cf94b9784","entry":"_parse_conflict_payload","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"800e659cf94b9784"}},{"code_sha256_prefix":"f83b8661a7ad2296","entry":"_parse_conflict_review_payload","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f83b8661a7ad2296"}},{"code_sha256_prefix":"936423ca29b5af3b","entry":"_parse_fact_update","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"936423ca29b5af3b"}},{"code_sha256_prefix":"daee147005f7e47a","entry":"_parse_fact_updates","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"daee147005f7e47a"}},{"code_sha256_prefix":"91130ae0675a62a6","entry":"_parse_trajectory_response","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"91130ae0675a62a6"}},{"code_sha256_prefix":"b631a97caeadc467","entry":"_required_nonnegative_int","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b631a97caeadc467"}},{"code_sha256_prefix":"48b4c9e809d45ecd","entry":"_required_string","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"48b4c9e809d45ecd"}},{"code_sha256_prefix":"cc56c9f0cdcf1c00","entry":"_required_string_list","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cc56c9f0cdcf1c00"}},{"code_sha256_prefix":"fdd1c66fde7c0117","entry":"_string_list","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fdd1c66fde7c0117"}},{"code_sha256_prefix":"00f6e69066ce946e","entry":"active_facts","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"00f6e69066ce946e"}},{"code_sha256_prefix":"21f2de0b99d91a74","entry":"all_achievement_commitments_resolved","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"21f2de0b99d91a74"}},{"code_sha256_prefix":"166c401d938d993b","entry":"apply_fact_updates","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"166c401d938d993b"}},{"code_sha256_prefix":"ee27fb0345c48b9e","entry":"call_with_retries","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ee27fb0345c48b9e"}},{"code_sha256_prefix":"680abdbff4c5f360","entry":"initialize_episode_state","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"680abdbff4c5f360"}},{"code_sha256_prefix":"8cd4a00832b05c68","entry":"parse_json_object","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8cd4a00832b05c68"}},{"code_sha256_prefix":"51f2305ee1b5ff12","entry":"render_conflict_check","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"51f2305ee1b5ff12"}},{"code_sha256_prefix":"4efbd7e33da0c1a6","entry":"render_conflict_double_check","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4efbd7e33da0c1a6"}},{"code_sha256_prefix":"5e5fa92d8ea0338b","entry":"render_fact_update","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5e5fa92d8ea0338b"}},{"code_sha256_prefix":"f938f6e8359bf88e","entry":"render_status_check","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f938f6e8359bf88e"}},{"code_sha256_prefix":"4dac9b8189adc047","entry":"render_trajectory_check","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4dac9b8189adc047"}},{"code_sha256_prefix":"eac5519024e14bb5","entry":"AuditorClient","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eac5519024e14bb5"}},{"code_sha256_prefix":"12af0219d50bab23","entry":"CommitmentChecker","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"12af0219d50bab23"}},{"code_sha256_prefix":"9b7d37cf3b082cce","entry":"EpisodeRunner","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9b7d37cf3b082cce"}},{"code_sha256_prefix":"d8072777741ab0c2","entry":"EpisodeTermination","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d8072777741ab0c2"}},{"code_sha256_prefix":"5e5e2bef2c3fb7bd","entry":"EpisodeTrace","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5e5e2bef2c3fb7bd"}},{"code_sha256_prefix":"59b2101d4d593b4f","entry":"EpisodeTurnResult","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"59b2101d4d593b4f"}},{"code_sha256_prefix":"85bb618d6063baf6","entry":"ModelCallError","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"85bb618d6063baf6"}},{"code_sha256_prefix":"db95fd949e2fe427","entry":"ModelOutputError","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"db95fd949e2fe427"}},{"code_sha256_prefix":"65670dfb60b8bed3","entry":"Narrator","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"65670dfb60b8bed3"}},{"code_sha256_prefix":"c65c208c583165f6","entry":"OpeningAuditClient","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c65c208c583165f6"}},{"code_sha256_prefix":"9b8e90476ed709e8","entry":"OpeningEvaluationError","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9b8e90476ed709e8"}},{"code_sha256_prefix":"fa59174576ea3ade","entry":"OpeningEvaluator","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fa59174576ea3ade"}},{"code_sha256_prefix":"fda2bd9500e337b2","entry":"TrajectoryChecker","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fda2bd9500e337b2"}},{"code_sha256_prefix":"49bf002b2ae80821","entry":"TurnEvaluator","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"49bf002b2ae80821"}},{"code_sha256_prefix":"b3a5232b359f120d","entry":"_validate_conflict_payload","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b3a5232b359f120d"}},{"code_sha256_prefix":"4b6ef600389c3d83","entry":"apply_turn_outcome","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4b6ef600389c3d83"}},{"code_sha256_prefix":"38f0bcdf3fb430c4","entry":"load_evaluation_prompts","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"38f0bcdf3fb430c4"}},{"code_sha256_prefix":"e37b77125d144e74","entry":"render_opening_fact_extract_prompt","repo":"yingpengma/NCP-Bench","repo_kind":"found_in_text","path":"src/ncpbench/runner.py","file_url":"https://github.com/yingpengma/NCP-Bench/blob/HEAD/src/ncpbench/runner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e37b77125d144e74"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,264 of the 9,581 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9581,"papers_checked":6264},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}