{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/transformer-visualization-via-dictionary","title":"Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors","arxiv_id":"2103.15949","date":"2021-03-29","proceeding":"NAACL (DeeLIO) 2021 6","authors":["Zeyu Yun","Yubei Chen","Bruno A Olshausen","Yann Lecun"],"abstract":"Transformer networks have revolutionized NLP representation learning since they were introduced. Though a great effort has been made to explain the representation in transformers, it is widely recognized that our understanding is not sufficient. One important reason is that there lack enough visualization tools for detailed analysis. In this paper, we propose to use dictionary learning to open up these \"black boxes\" as linear superpositions of transformer factors. Through visualization, we demonstrate the hierarchical semantic structures captured by the transformer factors, e.g., word-level polysemy disambiguation, sentence-level pattern formation, and long-range dependency. While some of these patterns confirm the conventional prior linguistic knowledge, the rest are relatively unexpected, which may provide new insights. We hope this visualization tool can bring further knowledge and a better understanding of how transformer networks work. The code is available at https://github.com/zeyuyun1/TransformerVis","url_abs":"https://arxiv.org/abs/2103.15949v2","url_pdf":"https://arxiv.org/pdf/2103.15949v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"transformer-visualization-via-dictionary","repo_url":"https://github.com/zeyuyun1/transformervis","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"dictionary-learning","task_name":"Dictionary Learning"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"sentence","task_name":"Sentence"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2103.15949","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2103.15949"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zeyuyun1/transformervis","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":5},"by_repo_kind":{"official":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"71cca0db54dd1b7d","entry":"FISTA","repo":"zeyuyun1/transformervis","repo_kind":"official","path":"sparsify_PyTorch.py","file_url":"https://github.com/zeyuyun1/transformervis/blob/HEAD/sparsify_PyTorch.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"71cca0db54dd1b7d"}},{"code_sha256_prefix":"a0124421c75783a1","entry":"ISTA_PN","repo":"zeyuyun1/transformervis","repo_kind":"official","path":"sparsify_PyTorch.py","file_url":"https://github.com/zeyuyun1/transformervis/blob/HEAD/sparsify_PyTorch.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a0124421c75783a1"}},{"code_sha256_prefix":"5dfcb40ff95bc046","entry":"cstr","repo":"zeyuyun1/transformervis","repo_kind":"official","path":"util.py","file_url":"https://github.com/zeyuyun1/transformervis/blob/HEAD/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5dfcb40ff95bc046"}},{"code_sha256_prefix":"afa11e466aa93385","entry":"merge_two","repo":"zeyuyun1/transformervis","repo_kind":"official","path":"util.py","file_url":"https://github.com/zeyuyun1/transformervis/blob/HEAD/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"afa11e466aa93385"}},{"code_sha256_prefix":"c17a2c30daeaebd0","entry":"quadraticBasisUpdate","repo":"zeyuyun1/transformervis","repo_kind":"official","path":"sparsify_PyTorch.py","file_url":"https://github.com/zeyuyun1/transformervis/blob/HEAD/sparsify_PyTorch.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c17a2c30daeaebd0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}