{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/benchmarking-and-understanding-compositional","title":"Benchmarking and Understanding Compositional Relational Reasoning of LLMs","arxiv_id":"2412.12841","date":"2024-12-17","proceeding":null,"authors":["Ruikang Ni","Da Xiao","Qingye Meng","Xiangyu Li","Shihui Zheng","Hongliang Liang"],"abstract":"Compositional relational reasoning (CRR) is a hallmark of human intelligence, but we lack a clear understanding of whether and how existing transformer large language models (LLMs) can solve CRR tasks. To enable systematic exploration of the CRR capability of LLMs, we first propose a new synthetic benchmark called Generalized Associative Recall (GAR) by integrating and generalizing the essence of several tasks in mechanistic interpretability (MI) study in a unified framework. Evaluation shows that GAR is challenging enough for existing LLMs, revealing their fundamental deficiency in CRR. Meanwhile, it is easy enough for systematic MI study. Then, to understand how LLMs solve GAR tasks, we use attribution patching to discover the core circuits reused by Vicuna-33B across different tasks and a set of vital attention heads. Intervention experiments show that the correct functioning of these heads significantly impacts task performance. Especially, we identify two classes of heads whose activations represent the abstract notion of true and false in GAR tasks respectively. They play a fundamental role in CRR across various models and tasks. The dataset and code are available at https://github.com/Caiyun-AI/GAR.","url_abs":"https://arxiv.org/abs/2412.12841v1","url_pdf":"https://arxiv.org/pdf/2412.12841v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"benchmarking-and-understanding-compositional","repo_url":"https://github.com/caiyun-ai/gar","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"relational-reasoning","task_name":"Relational Reasoning"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"patching","method_name":"Patching"},{"method_slug":"set","method_name":"SET"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.12841","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.12841"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/caiyun-ai/gar","reach":null}],"summary":{"ran_draft_wrong":1,"unverified":4},"by_repo_kind":{"official":{"samples":5,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"b67a3179e12bab14","entry":"create_model_and_optimizer","repo":"caiyun-ai/gar","repo_kind":"official","path":"MLP/code/train_and_test_MLP.py","file_url":"https://github.com/caiyun-ai/gar/blob/HEAD/MLP/code/train_and_test_MLP.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b67a3179e12bab14"}},{"code_sha256_prefix":"5cfede186da6f438","entry":"evaluate","repo":"caiyun-ai/gar","repo_kind":"official","path":"MLP/code/train_and_test_MLP.py","file_url":"https://github.com/caiyun-ai/gar/blob/HEAD/MLP/code/train_and_test_MLP.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5cfede186da6f438"}},{"code_sha256_prefix":"f7bccd2df5ca17cd","entry":"generate","repo":"caiyun-ai/gar","repo_kind":"official","path":"GAR/code/generate_GAR.py","file_url":"https://github.com/caiyun-ai/gar/blob/HEAD/GAR/code/generate_GAR.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f7bccd2df5ca17cd"}},{"code_sha256_prefix":"bfef273261c7e1bf","entry":"replace_keys","repo":"caiyun-ai/gar","repo_kind":"official","path":"GAR/code/generate_GAR.py","file_url":"https://github.com/caiyun-ai/gar/blob/HEAD/GAR/code/generate_GAR.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bfef273261c7e1bf"}},{"code_sha256_prefix":"7e5f981c6c901bda","entry":"train","repo":"caiyun-ai/gar","repo_kind":"official","path":"MLP/code/train_and_test_MLP.py","file_url":"https://github.com/caiyun-ai/gar/blob/HEAD/MLP/code/train_and_test_MLP.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7e5f981c6c901bda"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}