{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/can-large-language-models-learn-independent","title":"Can Large Language Models Learn Independent Causal Mechanisms?","arxiv_id":"2402.02636","date":"2024-02-04","proceeding":null,"authors":["Gaël Gendron","Bao Trung Nguyen","Alex Yuxuan Peng","Michael Witbrock","Gillian Dobbie"],"abstract":"Despite impressive performance on language modelling and complex reasoning tasks, Large Language Models (LLMs) fall short on the same tasks in uncommon settings or with distribution shifts, exhibiting a lack of generalisation ability. By contrast, systems such as causal models, that learn abstract variables and causal relationships, can demonstrate increased robustness against changes in the distribution. One reason for this success is the existence and use of Independent Causal Mechanisms (ICMs) representing high-level concepts that only sparsely interact. In this work, we apply two concepts from causality to learn ICMs within LLMs. We develop a new LLM architecture composed of multiple sparsely interacting language modelling modules. We show that such causal constraints can improve out-of-distribution performance on abstract and causal reasoning tasks. We also investigate the level of independence and domain specialisation and show that LLMs rely on pre-trained partially domain-invariant mechanisms resilient to fine-tuning.","url_abs":"https://arxiv.org/abs/2402.02636v2","url_pdf":"https://arxiv.org/pdf/2402.02636v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"can-large-language-models-learn-independent","repo_url":"https://github.com/strong-ai-lab/modular-lm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2402.02636","atlas_url":"https://app.syntology.ai/?focus=2402.02636","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.02636"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Strong-AI-Lab/Logical-and-abstract-reasoning","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":8,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"90ea10eb1aa031b1","entry":"convert_examples_to_features","repo":"Strong-AI-Lab/Logical-and-abstract-reasoning","repo_kind":"found_in_text","path":"models/utils_multiple_choice.py","file_url":"https://github.com/Strong-AI-Lab/Logical-and-abstract-reasoning/blob/HEAD/models/utils_multiple_choice.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"90ea10eb1aa031b1"}},{"code_sha256_prefix":"266daa7219ccdd1d","entry":"corrupt_bench","repo":"Strong-AI-Lab/Logical-and-abstract-reasoning","repo_kind":"found_in_text","path":"src/evaluate/information_gain.py","file_url":"https://github.com/Strong-AI-Lab/Logical-and-abstract-reasoning/blob/HEAD/src/evaluate/information_gain.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"266daa7219ccdd1d"}},{"code_sha256_prefix":"237c07af276bf442","entry":"corrupt_pvr","repo":"Strong-AI-Lab/Logical-and-abstract-reasoning","repo_kind":"found_in_text","path":"src/evaluate/information_gain.py","file_url":"https://github.com/Strong-AI-Lab/Logical-and-abstract-reasoning/blob/HEAD/src/evaluate/information_gain.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"237c07af276bf442"}},{"code_sha256_prefix":"fbc6ef16f2c20b2a","entry":"get_example_symbolic","repo":"Strong-AI-Lab/Logical-and-abstract-reasoning","repo_kind":"found_in_text","path":"src/evaluate/causal_induction_identification.py","file_url":"https://github.com/Strong-AI-Lab/Logical-and-abstract-reasoning/blob/HEAD/src/evaluate/causal_induction_identification.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fbc6ef16f2c20b2a"}},{"code_sha256_prefix":"a099bb4dec45b409","entry":"get_trial_symbolic","repo":"Strong-AI-Lab/Logical-and-abstract-reasoning","repo_kind":"found_in_text","path":"src/evaluate/causal_induction_identification.py","file_url":"https://github.com/Strong-AI-Lab/Logical-and-abstract-reasoning/blob/HEAD/src/evaluate/causal_induction_identification.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a099bb4dec45b409"}},{"code_sha256_prefix":"0f3995390250c44f","entry":"parse_input_symbolic","repo":"Strong-AI-Lab/Logical-and-abstract-reasoning","repo_kind":"found_in_text","path":"src/evaluate/causal_induction_identification.py","file_url":"https://github.com/Strong-AI-Lab/Logical-and-abstract-reasoning/blob/HEAD/src/evaluate/causal_induction_identification.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f3995390250c44f"}},{"code_sha256_prefix":"0a546b305d274996","entry":"select_field","repo":"Strong-AI-Lab/Logical-and-abstract-reasoning","repo_kind":"found_in_text","path":"models/run_multiple_choice.py","file_url":"https://github.com/Strong-AI-Lab/Logical-and-abstract-reasoning/blob/HEAD/models/run_multiple_choice.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a546b305d274996"}},{"code_sha256_prefix":"3c241ecfe3749a6d","entry":"simple_accuracy","repo":"Strong-AI-Lab/Logical-and-abstract-reasoning","repo_kind":"found_in_text","path":"models/run_multiple_choice.py","file_url":"https://github.com/Strong-AI-Lab/Logical-and-abstract-reasoning/blob/HEAD/models/run_multiple_choice.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c241ecfe3749a6d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}