{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-27068","title":"QUACK: Questioning, Understanding, and Auditing Communicated Knowledge in Multimodal Social Deduction Agents","arxiv_id":"2605.27068","date":"2026-05-26","proceeding":null,"authors":["Ye Yuan","Rui Song","Weien Li","Zeyu Li","Haochen Liu","Xiangyu Kong","Changjiang Han","Yonghan Yang","Zichen Zhao","Zixuan Dong","Fuyuan Lyu","Bowei He","Haolun Wu","Jikun Kang","Xue Liu"],"abstract":"Social deduction games have become a popular testbed for probing reasoning, deception, coordination, and belief modeling in Large Language Model (LLM) agents. However, most environments are scored only by game outcomes such as win rates and largely remain to text-only interaction, making it difficult to tell whether an agent's language is actually grounded in what it perceived and did, or to identify the failure modes underlying its behavior. To address this gap, we introduce QUACK, an open-source environment and evaluation framework for auditing the grounding of agent language in multimodal social reasoning. QUACK evaluates agents at three levels: game outcomes, behavioral trajectories, and utterance-level consistency. Its core Statement Verification Pipeline reconstructs each agent's ground-truth trajectory from engine logs and checks every discussion claim against it, automatically flagging spatial hallucination, unsupported accusation, deception collapse, and language-action inconsistency. Evaluating three frontier VLMs in both homogeneous and cross-model adversarial settings, we find that even the strongest agent hallucinates 15.1% of its verifiable spatial claims and makes over half of its accusations without grounded evidence. We release the full engine, evaluation framework, toolkit, and logs at https://github.com/AAAAA-Academia-Attractions/QUACK.","url_abs":"https://arxiv.org/abs/2605.27068","url_pdf":"https://arxiv.org/pdf/2605.27068","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2605.27068","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.27068"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/AAAAA-Academia-Attractions/QUACK","reach":null}],"summary":{"ran":2,"ran_draft_wrong":4,"unverified":8},"by_repo_kind":{"found_in_text":{"samples":14,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"faf4eb1e065b783d","entry":"MeetingMemory","repo":"AAAAA-Academia-Attractions/QUACK","repo_kind":"found_in_text","path":"quack/agents/vlm_agent.py","file_url":"https://github.com/AAAAA-Academia-Attractions/QUACK/blob/HEAD/quack/agents/vlm_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"faf4eb1e065b783d"}},{"code_sha256_prefix":"f1deb9dda270a8a8","entry":"TickMemory","repo":"AAAAA-Academia-Attractions/QUACK","repo_kind":"found_in_text","path":"quack/agents/vlm_agent.py","file_url":"https://github.com/AAAAA-Academia-Attractions/QUACK/blob/HEAD/quack/agents/vlm_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f1deb9dda270a8a8"}},{"code_sha256_prefix":"ca0749dc87268c1e","entry":"build_vlm_messages","repo":"AAAAA-Academia-Attractions/QUACK","repo_kind":"found_in_text","path":"quack/agents/vlm_agent.py","file_url":"https://github.com/AAAAA-Academia-Attractions/QUACK/blob/HEAD/quack/agents/vlm_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ca0749dc87268c1e"}},{"code_sha256_prefix":"b60ec523479ff997","entry":"combine_action_and_say","repo":"AAAAA-Academia-Attractions/QUACK","repo_kind":"found_in_text","path":"quack/agents/vlm_agent.py","file_url":"https://github.com/AAAAA-Academia-Attractions/QUACK/blob/HEAD/quack/agents/vlm_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b60ec523479ff997"}},{"code_sha256_prefix":"02e09e4b09d128ee","entry":"extract_say_clause","repo":"AAAAA-Academia-Attractions/QUACK","repo_kind":"found_in_text","path":"quack/agents/vlm_agent.py","file_url":"https://github.com/AAAAA-Academia-Attractions/QUACK/blob/HEAD/quack/agents/vlm_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"02e09e4b09d128ee"}},{"code_sha256_prefix":"7709ec9d28bc19a0","entry":"image_to_base64","repo":"AAAAA-Academia-Attractions/QUACK","repo_kind":"found_in_text","path":"quack/agents/vlm_agent.py","file_url":"https://github.com/AAAAA-Academia-Attractions/QUACK/blob/HEAD/quack/agents/vlm_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7709ec9d28bc19a0"}},{"code_sha256_prefix":"8aa16d43c7b18b23","entry":"AgentMemory","repo":"AAAAA-Academia-Attractions/QUACK","repo_kind":"found_in_text","path":"quack/agents/vlm_agent.py","file_url":"https://github.com/AAAAA-Academia-Attractions/QUACK/blob/HEAD/quack/agents/vlm_agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8aa16d43c7b18b23"}},{"code_sha256_prefix":"ed456165ba9c9e7f","entry":"BaseAgent","repo":"AAAAA-Academia-Attractions/QUACK","repo_kind":"found_in_text","path":"quack/agents/vlm_agent.py","file_url":"https://github.com/AAAAA-Academia-Attractions/QUACK/blob/HEAD/quack/agents/vlm_agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed456165ba9c9e7f"}},{"code_sha256_prefix":"751c3d293bc78167","entry":"VLMAgent","repo":"AAAAA-Academia-Attractions/QUACK","repo_kind":"found_in_text","path":"quack/agents/vlm_agent.py","file_url":"https://github.com/AAAAA-Academia-Attractions/QUACK/blob/HEAD/quack/agents/vlm_agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"751c3d293bc78167"}},{"code_sha256_prefix":"99965b041f3276a0","entry":"_is_retryable_error","repo":"AAAAA-Academia-Attractions/QUACK","repo_kind":"found_in_text","path":"quack/agents/vlm_agent.py","file_url":"https://github.com/AAAAA-Academia-Attractions/QUACK/blob/HEAD/quack/agents/vlm_agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"99965b041f3276a0"}},{"code_sha256_prefix":"bd8e7f72bfe189e2","entry":"build_action_prompt","repo":"AAAAA-Academia-Attractions/QUACK","repo_kind":"found_in_text","path":"quack/agents/vlm_agent.py","file_url":"https://github.com/AAAAA-Academia-Attractions/QUACK/blob/HEAD/quack/agents/vlm_agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd8e7f72bfe189e2"}},{"code_sha256_prefix":"4e906b62ad7dd69d","entry":"build_discussion_prompt","repo":"AAAAA-Academia-Attractions/QUACK","repo_kind":"found_in_text","path":"quack/agents/vlm_agent.py","file_url":"https://github.com/AAAAA-Academia-Attractions/QUACK/blob/HEAD/quack/agents/vlm_agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4e906b62ad7dd69d"}},{"code_sha256_prefix":"491e0e71f9e48cbf","entry":"build_system_prompt","repo":"AAAAA-Academia-Attractions/QUACK","repo_kind":"found_in_text","path":"quack/agents/vlm_agent.py","file_url":"https://github.com/AAAAA-Academia-Attractions/QUACK/blob/HEAD/quack/agents/vlm_agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"491e0e71f9e48cbf"}},{"code_sha256_prefix":"edbf68303c290906","entry":"build_vote_prompt","repo":"AAAAA-Academia-Attractions/QUACK","repo_kind":"found_in_text","path":"quack/agents/vlm_agent.py","file_url":"https://github.com/AAAAA-Academia-Attractions/QUACK/blob/HEAD/quack/agents/vlm_agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"edbf68303c290906"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,795 of the 9,581 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9581,"papers_checked":6795},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}