{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tom-rl-reinforcement-learning-unlocks-theory","title":"Do Theory of Mind Benchmarks Need Explicit Human-like Reasoning in Language Models?","arxiv_id":"2504.01698","date":"2025-04-02","proceeding":null,"authors":["Yi-Long Lu","Chunhui Zhang","Jiajun Song","Lifeng Fan","Wei Wang"],"abstract":"Theory of Mind (ToM), the ability to attribute mental states to others, is fundamental for human social intelligence and a critical capability for advanced Artificial Intelligence. Recent advancements in Large Language Models (LLMs) have shown promising performance on ToM benchmarks, raising the question: Do these benchmarks necessitate explicit human-like reasoning processes, or can models succeed through alternative strategies? We investigate this question empirically by applying Reinforcement Learning (RL) and Supervised Fine-Tuning (SFT) to LLMs of varying scales (0.5B to 7B parameters) and evaluating them across multiple ToM datasets. Our results reveal a scale-dependent impact of RL: while RL significantly improves accuracy and fosters high-quality, interpretable, and transferable belief-tracking reasoning in larger models (7B), it leads to \"reasoning collapse\" in smaller models ($\\leq$3B), where high accuracy and generalization ability are achieved via drastically shortened, less meaningful responses. Surprisingly, further SFT achieves competitive and generalizable performance across these benchmarks, often matching or exceeding RL models in accuracy, despite not being explicitly trained to produce structured reasoning traces. These findings highlight a critical discrepancy between benchmark accuracy and the nature of learned reasoning. Our work suggests that current ToM benchmarks may be solvable without requiring the explicit, human-like simulation of mental states they were designed to probe. LLMs, particularly when scale is limited or training signals focus solely on output correctness, may leverage alternative rules effective for benchmark data structures.","url_abs":"https://arxiv.org/abs/2504.01698v3","url_pdf":"https://arxiv.org/pdf/2504.01698v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tom-rl-reinforcement-learning-unlocks-theory","repo_url":"https://github.com/bigai-ai/ToM-RL","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"attribute","task_name":"Attribute"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[{"method_slug":"focus","method_name":"Focus"},{"method_slug":"sft","method_name":"SFT"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2504.01698","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2504.01698"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bigai-ai/ToM-RL","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":2,"ran_honours":1,"ran_fixture":1,"ran_draft_wrong":1,"unverified":10},"by_repo_kind":{"official":{"samples":15,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"360c671d6041ba3b","entry":"create_huggingface_actor","repo":"bigai-ai/ToM-RL","repo_kind":"official","path":"verl/utils/model.py","file_url":"https://github.com/bigai-ai/ToM-RL/blob/HEAD/verl/utils/model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"360c671d6041ba3b"}},{"code_sha256_prefix":"216e28c040173a61","entry":"extract_step","repo":"bigai-ai/ToM-RL","repo_kind":"official","path":"verl/trainer/fsdp_sft_trainer.py","file_url":"https://github.com/bigai-ai/ToM-RL/blob/HEAD/verl/trainer/fsdp_sft_trainer.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"216e28c040173a61"}},{"code_sha256_prefix":"42a7954a8cfa89e0","entry":"is_transformers_version_in_range","repo":"bigai-ai/ToM-RL","repo_kind":"official","path":"verl/models/transformers/monkey_patch.py","file_url":"https://github.com/bigai-ai/ToM-RL/blob/HEAD/verl/models/transformers/monkey_patch.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"42a7954a8cfa89e0"}},{"code_sha256_prefix":"21331a58f93375e4","entry":"union_tensor_dict","repo":"bigai-ai/ToM-RL","repo_kind":"official","path":"verl/protocol.py","file_url":"https://github.com/bigai-ai/ToM-RL/blob/HEAD/verl/protocol.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"DEP_MISSING","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"21331a58f93375e4"}},{"code_sha256_prefix":"25f0ea3f460f6ce1","entry":"unpad_dataproto","repo":"bigai-ai/ToM-RL","repo_kind":"official","path":"verl/protocol.py","file_url":"https://github.com/bigai-ai/ToM-RL/blob/HEAD/verl/protocol.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"25f0ea3f460f6ce1"}},{"code_sha256_prefix":"1fb4adb621eefddc","entry":"apply_monkey_patch","repo":"bigai-ai/ToM-RL","repo_kind":"official","path":"verl/models/transformers/monkey_patch.py","file_url":"https://github.com/bigai-ai/ToM-RL/blob/HEAD/verl/models/transformers/monkey_patch.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1fb4adb621eefddc"}},{"code_sha256_prefix":"1f0645c6a0d2044e","entry":"extract_solution","repo":"bigai-ai/ToM-RL","repo_kind":"official","path":"verl/utils/reward_score/explore_tom.py","file_url":"https://github.com/bigai-ai/ToM-RL/blob/HEAD/verl/utils/reward_score/explore_tom.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1f0645c6a0d2044e"}},{"code_sha256_prefix":"55d752fd12d893c2","entry":"extract_xml_answer","repo":"bigai-ai/ToM-RL","repo_kind":"official","path":"verl/utils/reward_score/explore_tom_2.py","file_url":"https://github.com/bigai-ai/ToM-RL/blob/HEAD/verl/utils/reward_score/explore_tom_2.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"55d752fd12d893c2"}},{"code_sha256_prefix":"19472d4343d0a082","entry":"get_huggingface_actor_config","repo":"bigai-ai/ToM-RL","repo_kind":"official","path":"verl/utils/model.py","file_url":"https://github.com/bigai-ai/ToM-RL/blob/HEAD/verl/utils/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"19472d4343d0a082"}},{"code_sha256_prefix":"919cf310ad4c189a","entry":"get_weight_loader","repo":"bigai-ai/ToM-RL","repo_kind":"official","path":"verl/models/weight_loader_registry.py","file_url":"https://github.com/bigai-ai/ToM-RL/blob/HEAD/verl/models/weight_loader_registry.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"919cf310ad4c189a"}},{"code_sha256_prefix":"1436322b957b9b34","entry":"normalize_answer","repo":"bigai-ai/ToM-RL","repo_kind":"official","path":"verl/utils/reward_score/explore_tom.py","file_url":"https://github.com/bigai-ai/ToM-RL/blob/HEAD/verl/utils/reward_score/explore_tom.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1436322b957b9b34"}},{"code_sha256_prefix":"c5322e5a65a9bea1","entry":"normalize_answer","repo":"bigai-ai/ToM-RL","repo_kind":"official","path":"verl/utils/reward_score/explore_tom_2.py","file_url":"https://github.com/bigai-ai/ToM-RL/blob/HEAD/verl/utils/reward_score/explore_tom_2.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c5322e5a65a9bea1"}},{"code_sha256_prefix":"a5c20c9ff16e297d","entry":"reward_func","repo":"bigai-ai/ToM-RL","repo_kind":"official","path":"verl/utils/reward_score/explore_tom_2.py","file_url":"https://github.com/bigai-ai/ToM-RL/blob/HEAD/verl/utils/reward_score/explore_tom_2.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a5c20c9ff16e297d"}},{"code_sha256_prefix":"3b15e2ac7497c441","entry":"squeeze","repo":"bigai-ai/ToM-RL","repo_kind":"official","path":"verl/utils/model.py","file_url":"https://github.com/bigai-ai/ToM-RL/blob/HEAD/verl/utils/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3b15e2ac7497c441"}},{"code_sha256_prefix":"c53a02ef0eee0abe","entry":"validate_response_structure","repo":"bigai-ai/ToM-RL","repo_kind":"official","path":"verl/utils/reward_score/explore_tom.py","file_url":"https://github.com/bigai-ai/ToM-RL/blob/HEAD/verl/utils/reward_score/explore_tom.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c53a02ef0eee0abe"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}