{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-the-inductive-bias-of-masked-language","title":"On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies","arxiv_id":"2104.05694","date":"2021-04-12","proceeding":"NAACL 2021 4","authors":["Tianyi Zhang","Tatsunori Hashimoto"],"abstract":"We study how masking and predicting tokens in an unsupervised fashion can give rise to linguistic structures and downstream performance gains. Recent theories have suggested that pretrained language models acquire useful inductive biases through masks that implicitly act as cloze reductions for downstream tasks. While appealing, we show that the success of the random masking strategy used in practice cannot be explained by such cloze-like masks alone. We construct cloze-like masks using task-specific lexicons for three different classification datasets and show that the majority of pretrained performance gains come from generic masks that are not associated with the lexicon. To explain the empirical success of these generic masks, we demonstrate a correspondence between the Masked Language Model (MLM) objective and existing methods for learning statistical dependencies in graphical models. Using this, we derive a method for extracting these learned statistical dependencies in MLMs and show that these dependencies encode useful inductive biases in the form of syntactic structures. In an unsupervised parsing evaluation, simply forming a minimum spanning tree on the implied statistical dependence structure outperforms a classic method for unsupervised parsing (58.74 vs. 55.91 UUAS).","url_abs":"https://arxiv.org/abs/2104.05694v1","url_pdf":"https://arxiv.org/pdf/2104.05694v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-the-inductive-bias-of-masked-language","repo_url":"https://github.com/tatsu-lab/mlm_inductive_bias","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"inductive-bias","task_name":"Inductive Bias"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"masked-language-modeling","task_name":"Masked Language Modeling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2104.05694","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.05694"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tatsu-lab/mlm_inductive_bias","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/tatsu-lab/mlm_","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}}],"summary":{"ran_draft_wrong":1,"ran_fixture":1,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e0145f15e5d4874c","entry":"load_conll_dataset","repo":"tatsu-lab/mlm_inductive_bias","repo_kind":"official","path":"unsupervised_parsing/mutual_info_eval.py","file_url":"https://github.com/tatsu-lab/mlm_inductive_bias/blob/HEAD/unsupervised_parsing/mutual_info_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e0145f15e5d4874c"}},{"code_sha256_prefix":"a83be86ee13daa09","entry":"prims_matrix_to_edges","repo":"tatsu-lab/mlm_inductive_bias","repo_kind":"official","path":"unsupervised_parsing/mutual_info_eval.py","file_url":"https://github.com/tatsu-lab/mlm_inductive_bias/blob/HEAD/unsupervised_parsing/mutual_info_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a83be86ee13daa09"}},{"code_sha256_prefix":"a95ce902b14c23f9","entry":"report_uuas_and_tikz","repo":"tatsu-lab/mlm_inductive_bias","repo_kind":"official","path":"unsupervised_parsing/mutual_info_eval.py","file_url":"https://github.com/tatsu-lab/mlm_inductive_bias/blob/HEAD/unsupervised_parsing/mutual_info_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a95ce902b14c23f9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}