{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/transformers-as-soft-reasoners-over-language","title":"Transformers as Soft Reasoners over Language","arxiv_id":"2002.05867","date":"2020-02-14","proceeding":null,"authors":["Peter Clark","Oyvind Tafjord","Kyle Richardson"],"abstract":"Beginning with McCarthy's Advice Taker (1959), AI has pursued the goal of providing a system with explicit, general knowledge and having the system reason over that knowledge. However, expressing the knowledge in a formal (logical or probabilistic) representation has been a major obstacle to this research. This paper investigates a modern approach to this problem where the facts and rules are provided as natural language sentences, thus bypassing a formal representation. We train transformers to reason (or emulate reasoning) over these sentences using synthetically generated data. Our models, that we call RuleTakers, provide the first empirical demonstration that this kind of soft reasoning over language is learnable, can achieve high (99%) accuracy, and generalizes to test data requiring substantially deeper chaining than seen during training (95%+ scores). We also demonstrate that the models transfer well to two hand-authored rulebases, and to rulebases paraphrased into more natural language. These findings are significant as it suggests a new role for transformers, namely as limited \"soft theorem provers\" operating over explicit theories in language. This in turn suggests new possibilities for explainability, correctability, and counterfactual reasoning in question-answering.","url_abs":"https://arxiv.org/abs/2002.05867v2","url_pdf":"https://arxiv.org/pdf/2002.05867v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"transformers-as-soft-reasoners-over-language","repo_url":"https://github.com/allenai/language_fragments","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"transformers-as-soft-reasoners-over-language","repo_url":"https://github.com/allenai/ruletaker","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"counterfactual-reasoning","task_name":"Counterfactual Reasoning"},{"task_slug":"general-knowledge","task_name":"General Knowledge"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":null,"task_name":"counterfactual"}],"methods":[{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2002.05867","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2002.05867"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/allenai/language_fragments","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/allenai/ruletaker","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":6},"by_repo_kind":{"listed":{"samples":6,"ran":0,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e594764154e16ef4","entry":"choose_production","repo":"allenai/ruletaker","repo_kind":"listed","path":"theory_generator.py","file_url":"https://github.com/allenai/ruletaker/blob/HEAD/theory_generator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e594764154e16ef4"}},{"code_sha256_prefix":"1e9e9aa4121731e7","entry":"constantize","repo":"allenai/ruletaker","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/allenai/ruletaker/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1e9e9aa4121731e7"}},{"code_sha256_prefix":"2c4d167d4241bd86","entry":"format_argument","repo":"allenai/ruletaker","repo_kind":"listed","path":"theory_label_generator.py","file_url":"https://github.com/allenai/ruletaker/blob/HEAD/theory_label_generator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2c4d167d4241bd86"}},{"code_sha256_prefix":"0f8d42da5094172d","entry":"log_time","repo":"allenai/language_fragments","repo_kind":"listed","path":"language_fragments/base.py","file_url":"https://github.com/allenai/language_fragments/blob/HEAD/language_fragments/base.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0f8d42da5094172d"}},{"code_sha256_prefix":"3afdbc99d8d3d769","entry":"predicatize","repo":"allenai/ruletaker","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/allenai/ruletaker/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3afdbc99d8d3d769"}},{"code_sha256_prefix":"0815e3dfc4a3711b","entry":"variablize","repo":"allenai/ruletaker","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/allenai/ruletaker/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0815e3dfc4a3711b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}