{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/interintent-investigating-social-intelligence","title":"InterIntent: Investigating Social Intelligence of LLMs via Intention Understanding in an Interactive Game Context","arxiv_id":"2406.12203","date":"2024-06-18","proceeding":null,"authors":["Ziyi Liu","Abhishek Anand","Pei Zhou","Jen-tse Huang","Jieyu Zhao"],"abstract":"Large language models (LLMs) have demonstrated the potential to mimic human social intelligence. However, most studies focus on simplistic and static self-report or performance-based tests, which limits the depth and validity of the analysis. In this paper, we developed a novel framework, InterIntent, to assess LLMs' social intelligence by mapping their ability to understand and manage intentions in a game setting. We focus on four dimensions of social intelligence: situational awareness, self-regulation, self-awareness, and theory of mind. Each dimension is linked to a specific game task: intention selection, intention following, intention summarization, and intention guessing. Our findings indicate that while LLMs exhibit high proficiency in selecting intentions, achieving an accuracy of 88%, their ability to infer the intentions of others is significantly weaker, trailing human performance by 20%. Additionally, game performance correlates with intention understanding, highlighting the importance of the four components towards success in this game. These findings underline the crucial role of intention understanding in evaluating LLMs' social intelligence and highlight the potential of using social deduction games as a complex testbed to enhance LLM evaluation. InterIntent contributes a structured approach to bridging the evaluation gap in social intelligence within multiplayer games.","url_abs":"https://arxiv.org/abs/2406.12203v3","url_pdf":"https://arxiv.org/pdf/2406.12203v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"interintent-investigating-social-intelligence","repo_url":"https://github.com/uscnlp-lime/inter-intent","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.12203","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.12203"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/uscnlp-lime/inter-intent","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":5},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9b8f8ada028982d8","entry":"convert_list_to_dict","repo":"uscnlp-lime/inter-intent","repo_kind":"official","path":"avalon_evaluation.py","file_url":"https://github.com/uscnlp-lime/inter-intent/blob/HEAD/avalon_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9b8f8ada028982d8"}},{"code_sha256_prefix":"0ce03ffd229bd041","entry":"extract_jsons","repo":"uscnlp-lime/inter-intent","repo_kind":"official","path":"chatarena/utils.py","file_url":"https://github.com/uscnlp-lime/inter-intent/blob/HEAD/chatarena/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0ce03ffd229bd041"}},{"code_sha256_prefix":"30607e5949ec40db","entry":"is_json","repo":"uscnlp-lime/inter-intent","repo_kind":"official","path":"chatarena/utils.py","file_url":"https://github.com/uscnlp-lime/inter-intent/blob/HEAD/chatarena/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"30607e5949ec40db"}},{"code_sha256_prefix":"7ca35c23e77739fa","entry":"is_json_inside","repo":"uscnlp-lime/inter-intent","repo_kind":"official","path":"chatarena/utils.py","file_url":"https://github.com/uscnlp-lime/inter-intent/blob/HEAD/chatarena/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7ca35c23e77739fa"}},{"code_sha256_prefix":"d08b1ac8281f99c3","entry":"read_data","repo":"uscnlp-lime/inter-intent","repo_kind":"official","path":"avalon_evaluation.py","file_url":"https://github.com/uscnlp-lime/inter-intent/blob/HEAD/avalon_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d08b1ac8281f99c3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}