{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-the-paradox-of-learning-to-reason-from","title":"On the Paradox of Learning to Reason from Data","arxiv_id":"2205.11502","date":"2022-05-23","proceeding":null,"authors":["Honghua Zhang","Liunian Harold Li","Tao Meng","Kai-Wei Chang","Guy Van Den Broeck"],"abstract":"Logical reasoning is needed in a wide range of NLP tasks. Can a BERT model be trained end-to-end to solve logical reasoning problems presented in natural language? We attempt to answer this question in a confined problem space where there exists a set of parameters that perfectly simulates logical reasoning. We make observations that seem to contradict each other: BERT attains near-perfect accuracy on in-distribution test examples while failing to generalize to other data distributions over the exact same problem space. Our study provides an explanation for this paradox: instead of learning to emulate the correct reasoning function, BERT has in fact learned statistical features that inherently exist in logical reasoning problems. We also show that it is infeasible to jointly remove statistical features from data, illustrating the difficulty of learning to reason in general. Our result naturally extends to other neural models and unveils the fundamental difference between learning to reason and learning to achieve high performance on NLP benchmarks using statistical features.","url_abs":"https://arxiv.org/abs/2205.11502v2","url_pdf":"https://arxiv.org/pdf/2205.11502v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-the-paradox-of-learning-to-reason-from","repo_url":"https://github.com/joshuacnf/paradox-learning2reason","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"logical-reasoning","task_name":"Logical Reasoning"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2205.11502","atlas_url":"https://app.syntology.ai/?focus=2205.11502","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2205.11502"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/joshuacnf/paradox-learning2reason","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6,"unverified":4},"by_repo_kind":{"official":{"samples":10,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"87d975f07938c853","entry":"gen_word_embedding","repo":"joshuacnf/paradox-learning2reason","repo_kind":"official","path":"logic_bert/evaluate.py","file_url":"https://github.com/joshuacnf/paradox-learning2reason/blob/HEAD/logic_bert/evaluate.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"87d975f07938c853"}},{"code_sha256_prefix":"650ed7bb447fd433","entry":"limit_examples","repo":"joshuacnf/paradox-learning2reason","repo_kind":"official","path":"dataset.py","file_url":"https://github.com/joshuacnf/paradox-learning2reason/blob/HEAD/dataset.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"650ed7bb447fd433"}},{"code_sha256_prefix":"226b198cd6845403","entry":"read_vocab","repo":"joshuacnf/paradox-learning2reason","repo_kind":"official","path":"logic_bert/evaluate.py","file_url":"https://github.com/joshuacnf/paradox-learning2reason/blob/HEAD/logic_bert/evaluate.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"226b198cd6845403"}},{"code_sha256_prefix":"ec26eefbcd67387b","entry":"read_vocab","repo":"joshuacnf/paradox-learning2reason","repo_kind":"official","path":"sample/sample.py","file_url":"https://github.com/joshuacnf/paradox-learning2reason/blob/HEAD/sample/sample.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec26eefbcd67387b"}},{"code_sha256_prefix":"a83da76495b09b3a","entry":"sample_one_rule","repo":"joshuacnf/paradox-learning2reason","repo_kind":"official","path":"sample/sample.py","file_url":"https://github.com/joshuacnf/paradox-learning2reason/blob/HEAD/sample/sample.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a83da76495b09b3a"}},{"code_sha256_prefix":"40f5be912c82ef27","entry":"sample_rule_priority","repo":"joshuacnf/paradox-learning2reason","repo_kind":"official","path":"sample/sample.py","file_url":"https://github.com/joshuacnf/paradox-learning2reason/blob/HEAD/sample/sample.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"40f5be912c82ef27"}},{"code_sha256_prefix":"2b95e75e05ea5abf","entry":"all_gather","repo":"joshuacnf/paradox-learning2reason","repo_kind":"official","path":"dist.py","file_url":"https://github.com/joshuacnf/paradox-learning2reason/blob/HEAD/dist.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2b95e75e05ea5abf"}},{"code_sha256_prefix":"1fc9bf9a870979fe","entry":"evaluate","repo":"joshuacnf/paradox-learning2reason","repo_kind":"official","path":"finetune_simplified.py","file_url":"https://github.com/joshuacnf/paradox-learning2reason/blob/HEAD/finetune_simplified.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1fc9bf9a870979fe"}},{"code_sha256_prefix":"63d79b6707c1288e","entry":"gen_position_embedding","repo":"joshuacnf/paradox-learning2reason","repo_kind":"official","path":"logic_bert/evaluate.py","file_url":"https://github.com/joshuacnf/paradox-learning2reason/blob/HEAD/logic_bert/evaluate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"63d79b6707c1288e"}},{"code_sha256_prefix":"a38ed1aa8d59134e","entry":"reduce_dict","repo":"joshuacnf/paradox-learning2reason","repo_kind":"official","path":"dist.py","file_url":"https://github.com/joshuacnf/paradox-learning2reason/blob/HEAD/dist.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a38ed1aa8d59134e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}