{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/open-sesame-getting-inside-berts-linguistic","title":"Open Sesame: Getting Inside BERT's Linguistic Knowledge","arxiv_id":"1906.01698","date":"2019-06-04","proceeding":"WS 2019 8","authors":["Yongjie Lin","Yi Chern Tan","Robert Frank"],"abstract":"How and to what extent does BERT encode syntactically-sensitive hierarchical information or positionally-sensitive linear information? Recent work has shown that contextual representations like BERT perform well on tasks that require sensitivity to linguistic structure. We present here two studies which aim to provide a better understanding of the nature of BERT's representations. The first of these focuses on the identification of structurally-defined elements using diagnostic classifiers, while the second explores BERT's representation of subject-verb agreement and anaphor-antecedent dependencies through a quantitative assessment of self-attention vectors. In both cases, we find that BERT encodes positional information about word tokens well on its lower layers, but switches to a hierarchically-oriented encoding on higher layers. We conclude then that BERT's representations do indeed model linguistically relevant aspects of hierarchical structure, though they do not appear to show the sharp sensitivity to hierarchical structure that is found in human processing of reflexive anaphora.","url_abs":"https://arxiv.org/abs/1906.01698v1","url_pdf":"https://arxiv.org/pdf/1906.01698v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"open-sesame-getting-inside-berts-linguistic","repo_url":"https://github.com/yongjie-lin/bert-opensesame","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"diagnostic","task_name":"Diagnostic"},{"task_slug":"sensitivity","task_name":"Sensitivity"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1906.01698","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1906.01698"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yongjie-lin/bert-opensesame","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":3,"ran_honours":4,"unverified":11},"by_repo_kind":{"official":{"samples":18,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0f786c407fb1ee4c","entry":"swish","repo":"yongjie-lin/bert-opensesame","repo_kind":"official","path":"pytorch_pretrained_bert/modeling.py","file_url":"https://github.com/yongjie-lin/bert-opensesame/blob/HEAD/pytorch_pretrained_bert/modeling.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0f786c407fb1ee4c"}},{"code_sha256_prefix":"8d23fbe2b99b840b","entry":"gelu","repo":"yongjie-lin/bert-opensesame","repo_kind":"official","path":"bertviz/bertviz/pytorch_pretrained_bert/modeling_gpt2.py","file_url":"https://github.com/yongjie-lin/bert-opensesame/blob/HEAD/bertviz/bertviz/pytorch_pretrained_bert/modeling_gpt2.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8d23fbe2b99b840b"}},{"code_sha256_prefix":"fdc64f4c72036ae4","entry":"gelu","repo":"yongjie-lin/bert-opensesame","repo_kind":"official","path":"bertviz/bertviz/pytorch_pretrained_bert/modeling.py","file_url":"https://github.com/yongjie-lin/bert-opensesame/blob/HEAD/bertviz/bertviz/pytorch_pretrained_bert/modeling.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fdc64f4c72036ae4"}},{"code_sha256_prefix":"40e9fee2e0b7e278","entry":"gelu","repo":"yongjie-lin/bert-opensesame","repo_kind":"official","path":"pytorch_pretrained_bert/modeling.py","file_url":"https://github.com/yongjie-lin/bert-opensesame/blob/HEAD/pytorch_pretrained_bert/modeling.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"40e9fee2e0b7e278"}},{"code_sha256_prefix":"af64ec220e8bcdbc","entry":"url_to_filename","repo":"yongjie-lin/bert-opensesame","repo_kind":"official","path":"pytorch_pretrained_bert/file_utils.py","file_url":"https://github.com/yongjie-lin/bert-opensesame/blob/HEAD/pytorch_pretrained_bert/file_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"af64ec220e8bcdbc"}},{"code_sha256_prefix":"e7d542062316094a","entry":"warmup_constant","repo":"yongjie-lin/bert-opensesame","repo_kind":"official","path":"pytorch_pretrained_bert/optimization.py","file_url":"https://github.com/yongjie-lin/bert-opensesame/blob/HEAD/pytorch_pretrained_bert/optimization.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e7d542062316094a"}},{"code_sha256_prefix":"c58d57224530d17e","entry":"warmup_linear","repo":"yongjie-lin/bert-opensesame","repo_kind":"official","path":"pytorch_pretrained_bert/optimization.py","file_url":"https://github.com/yongjie-lin/bert-opensesame/blob/HEAD/pytorch_pretrained_bert/optimization.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c58d57224530d17e"}},{"code_sha256_prefix":"776ea2acae7131ec","entry":"augment_data_JJ","repo":"yongjie-lin/bert-opensesame","repo_kind":"official","path":"pytorch_pretrained_bert/dd_utils.py","file_url":"https://github.com/yongjie-lin/bert-opensesame/blob/HEAD/pytorch_pretrained_bert/dd_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"776ea2acae7131ec"}},{"code_sha256_prefix":"349d780dd4a89a37","entry":"cached_path","repo":"yongjie-lin/bert-opensesame","repo_kind":"official","path":"pytorch_pretrained_bert/file_utils.py","file_url":"https://github.com/yongjie-lin/bert-opensesame/blob/HEAD/pytorch_pretrained_bert/file_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"349d780dd4a89a37"}},{"code_sha256_prefix":"db0ac56aaf6e35e6","entry":"filename_to_url","repo":"yongjie-lin/bert-opensesame","repo_kind":"official","path":"pytorch_pretrained_bert/file_utils.py","file_url":"https://github.com/yongjie-lin/bert-opensesame/blob/HEAD/pytorch_pretrained_bert/file_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"db0ac56aaf6e35e6"}},{"code_sha256_prefix":"baa5766f4566aafc","entry":"load_tf_weights_in_bert","repo":"yongjie-lin/bert-opensesame","repo_kind":"official","path":"bertviz/bertviz/pytorch_pretrained_bert/modeling.py","file_url":"https://github.com/yongjie-lin/bert-opensesame/blob/HEAD/bertviz/bertviz/pytorch_pretrained_bert/modeling.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"baa5766f4566aafc"}},{"code_sha256_prefix":"a62604f618a8cc82","entry":"load_tf_weights_in_gpt2","repo":"yongjie-lin/bert-opensesame","repo_kind":"official","path":"bertviz/bertviz/pytorch_pretrained_bert/modeling_gpt2.py","file_url":"https://github.com/yongjie-lin/bert-opensesame/blob/HEAD/bertviz/bertviz/pytorch_pretrained_bert/modeling_gpt2.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a62604f618a8cc82"}},{"code_sha256_prefix":"46f305675277e757","entry":"load_tf_weights_in_openai_gpt","repo":"yongjie-lin/bert-opensesame","repo_kind":"official","path":"bertviz/bertviz/pytorch_pretrained_bert/modeling_openai.py","file_url":"https://github.com/yongjie-lin/bert-opensesame/blob/HEAD/bertviz/bertviz/pytorch_pretrained_bert/modeling_openai.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"46f305675277e757"}},{"code_sha256_prefix":"fb95c4b13cdf89a2","entry":"load_vocab","repo":"yongjie-lin/bert-opensesame","repo_kind":"official","path":"pytorch_pretrained_bert/tokenization.py","file_url":"https://github.com/yongjie-lin/bert-opensesame/blob/HEAD/pytorch_pretrained_bert/tokenization.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fb95c4b13cdf89a2"}},{"code_sha256_prefix":"5c1d1e5a7edda4cd","entry":"normalize_string","repo":"yongjie-lin/bert-opensesame","repo_kind":"official","path":"pytorch_pretrained_bert/dd_utils.py","file_url":"https://github.com/yongjie-lin/bert-opensesame/blob/HEAD/pytorch_pretrained_bert/dd_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5c1d1e5a7edda4cd"}},{"code_sha256_prefix":"83145131533aa4db","entry":"prep_batch","repo":"yongjie-lin/bert-opensesame","repo_kind":"official","path":"pytorch_pretrained_bert/dd_utils.py","file_url":"https://github.com/yongjie-lin/bert-opensesame/blob/HEAD/pytorch_pretrained_bert/dd_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"83145131533aa4db"}},{"code_sha256_prefix":"35f7cddf90dd05d4","entry":"warmup_cosine","repo":"yongjie-lin/bert-opensesame","repo_kind":"official","path":"pytorch_pretrained_bert/optimization.py","file_url":"https://github.com/yongjie-lin/bert-opensesame/blob/HEAD/pytorch_pretrained_bert/optimization.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"35f7cddf90dd05d4"}},{"code_sha256_prefix":"da7295883cd7da14","entry":"whitespace_tokenize","repo":"yongjie-lin/bert-opensesame","repo_kind":"official","path":"pytorch_pretrained_bert/tokenization.py","file_url":"https://github.com/yongjie-lin/bert-opensesame/blob/HEAD/pytorch_pretrained_bert/tokenization.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"da7295883cd7da14"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}