{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/probing-linguistic-information-for-logical","title":"Probing Linguistic Information For Logical Inference In Pre-trained Language Models","arxiv_id":"2112.01753","date":"2021-12-03","proceeding":null,"authors":["Zeming Chen","Qiyue Gao"],"abstract":"Progress in pre-trained language models has led to a surge of impressive results on downstream tasks for natural language understanding. Recent work on probing pre-trained language models uncovered a wide range of linguistic properties encoded in their contextualized representations. However, it is unclear whether they encode semantic knowledge that is crucial to symbolic inference methods. We propose a methodology for probing linguistic information for logical inference in pre-trained language model representations. Our probing datasets cover a list of linguistic phenomena required by major symbolic inference systems. We find that (i) pre-trained language models do encode several types of linguistic information for inference, but there are also some types of information that are weakly encoded, (ii) language models can effectively learn missing linguistic information through fine-tuning. Overall, our findings provide insights into which aspects of linguistic information for logical inference do language models and their pre-training procedures capture. Moreover, we have demonstrated language models' potential as semantic and background knowledge bases for supporting symbolic inference methods.","url_abs":"https://arxiv.org/abs/2112.01753v2","url_pdf":"https://arxiv.org/pdf/2112.01753v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"probing-linguistic-information-for-logical","repo_url":"https://github.com/eric11eca/inference-information-probing","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"natural-language-understanding","task_name":"Natural Language Understanding"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2112.01753","atlas_url":"https://app.syntology.ai/?focus=2112.01753","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2112.01753"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/eric11eca/inference-information-probing","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":9,"ran_honours":1,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":14,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e445a3b7d6a58154","entry":"batched_index_select","repo":"eric11eca/inference-information-probing","repo_kind":"official","path":"jiant/ext/allennlp.py","file_url":"https://github.com/eric11eca/inference-information-probing/blob/HEAD/jiant/ext/allennlp.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e445a3b7d6a58154"}},{"code_sha256_prefix":"df8b734fdb77932a","entry":"count_freq","repo":"eric11eca/inference-information-probing","repo_kind":"official","path":"analyze.py","file_url":"https://github.com/eric11eca/inference-information-probing/blob/HEAD/analyze.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"df8b734fdb77932a"}},{"code_sha256_prefix":"349a73ed4ded5853","entry":"evaluate","repo":"eric11eca/inference-information-probing","repo_kind":"official","path":"inform_prob/trainer.py","file_url":"https://github.com/eric11eca/inference-information-probing/blob/HEAD/inform_prob/trainer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"349a73ed4ded5853"}},{"code_sha256_prefix":"e6bd0ba803f299b8","entry":"file_len","repo":"eric11eca/inference-information-probing","repo_kind":"official","path":"inform_prob/util.py","file_url":"https://github.com/eric11eca/inference-information-probing/blob/HEAD/inform_prob/util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e6bd0ba803f299b8"}},{"code_sha256_prefix":"8f7619b2828926ea","entry":"flat_collate_fn","repo":"eric11eca/inference-information-probing","repo_kind":"official","path":"jiant/tasks/core.py","file_url":"https://github.com/eric11eca/inference-information-probing/blob/HEAD/jiant/tasks/core.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8f7619b2828926ea"}},{"code_sha256_prefix":"59cfe607c72c35ba","entry":"get_fasttext","repo":"eric11eca/inference-information-probing","repo_kind":"official","path":"inform_prob/process.py","file_url":"https://github.com/eric11eca/inference-information-probing/blob/HEAD/inform_prob/process.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"59cfe607c72c35ba"}},{"code_sha256_prefix":"16cde5ff8ae0a586","entry":"masked_softmax","repo":"eric11eca/inference-information-probing","repo_kind":"official","path":"jiant/ext/allennlp.py","file_url":"https://github.com/eric11eca/inference-information-probing/blob/HEAD/jiant/ext/allennlp.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"16cde5ff8ae0a586"}},{"code_sha256_prefix":"61ffc5549a75fe3c","entry":"metadata_collate_fn","repo":"eric11eca/inference-information-probing","repo_kind":"official","path":"jiant/tasks/core.py","file_url":"https://github.com/eric11eca/inference-information-probing/blob/HEAD/jiant/tasks/core.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"61ffc5549a75fe3c"}},{"code_sha256_prefix":"0b2cc230b2f4d63a","entry":"read_data","repo":"eric11eca/inference-information-probing","repo_kind":"official","path":"inform_prob/util.py","file_url":"https://github.com/eric11eca/inference-information-probing/blob/HEAD/inform_prob/util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0b2cc230b2f4d63a"}},{"code_sha256_prefix":"e17e83a5650fd73c","entry":"resolve_is_lower_case","repo":"eric11eca/inference-information-probing","repo_kind":"official","path":"jiant/shared/model_resolution.py","file_url":"https://github.com/eric11eca/inference-information-probing/blob/HEAD/jiant/shared/model_resolution.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e17e83a5650fd73c"}},{"code_sha256_prefix":"36c8a80131f2e20a","entry":"resolve_warmup_steps","repo":"eric11eca/inference-information-probing","repo_kind":"official","path":"jiant/shared/model_setup.py","file_url":"https://github.com/eric11eca/inference-information-probing/blob/HEAD/jiant/shared/model_setup.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"36c8a80131f2e20a"}},{"code_sha256_prefix":"ad20a9d93a38e9cd","entry":"eval_all","repo":"eric11eca/inference-information-probing","repo_kind":"official","path":"inform_prob/trainer.py","file_url":"https://github.com/eric11eca/inference-information-probing/blob/HEAD/inform_prob/trainer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ad20a9d93a38e9cd"}},{"code_sha256_prefix":"fa314e7bf8386566","entry":"get_type_token_ratio","repo":"eric11eca/inference-information-probing","repo_kind":"official","path":"analyze.py","file_url":"https://github.com/eric11eca/inference-information-probing/blob/HEAD/analyze.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fa314e7bf8386566"}},{"code_sha256_prefix":"2f417d486bcad523","entry":"weighted_sum","repo":"eric11eca/inference-information-probing","repo_kind":"official","path":"jiant/ext/allennlp.py","file_url":"https://github.com/eric11eca/inference-information-probing/blob/HEAD/jiant/ext/allennlp.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2f417d486bcad523"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}