{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pushdown-layers-encoding-recursive-structure","title":"Pushdown Layers: Encoding Recursive Structure in Transformer Language Models","arxiv_id":"2310.19089","date":"2023-10-29","proceeding":null,"authors":["Shikhar Murty","Pratyusha Sharma","Jacob Andreas","Christopher D. Manning"],"abstract":"Recursion is a prominent feature of human language, and fundamentally challenging for self-attention due to the lack of an explicit recursive-state tracking mechanism. Consequently, Transformer language models poorly capture long-tail recursive structure and exhibit sample-inefficient syntactic generalization. This work introduces Pushdown Layers, a new self-attention layer that models recursive state via a stack tape that tracks estimated depths of every token in an incremental parse of the observed prefix. Transformer LMs with Pushdown Layers are syntactic language models that autoregressively and synchronously update this stack tape as they predict new tokens, in turn using the stack tape to softly modulate attention over tokens -- for instance, learning to \"skip\" over closed constituents. When trained on a corpus of strings annotated with silver constituency parses, Transformers equipped with Pushdown Layers achieve dramatically better and 3-5x more sample-efficient syntactic generalization, while maintaining similar perplexities. Pushdown Layers are a drop-in replacement for standard self-attention. We illustrate this by finetuning GPT2-medium with Pushdown Layers on an automatically parsed WikiText-103, leading to improvements on several GLUE text classification tasks.","url_abs":"https://arxiv.org/abs/2310.19089v1","url_pdf":"https://arxiv.org/pdf/2310.19089v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pushdown-layers-encoding-recursive-structure","repo_url":"https://github.com/murtyshikhar/pushdown-layers","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"text-classification","task_name":"Text Classification"},{"task_slug":"text-classification-1","task_name":"text-classification"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.19089","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.19089"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/murtyshikhar/pushdown-layers","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/MurtyShikhar/Pushdown-Layers","reach":{"status":"ok"}}],"summary":{"ran":13,"ran_honours":1,"unverified":9},"by_repo_kind":{"official":{"samples":23,"ran":14,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":23,"samples":[{"code_sha256_prefix":"551d6401ece551f3","entry":"AttentionMask","repo":"MurtyShikhar/Pushdown-Layers","repo_kind":"official","path":"models/pushdown_transformer_lm.py","file_url":"https://github.com/MurtyShikhar/Pushdown-Layers/blob/HEAD/models/pushdown_transformer_lm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"551d6401ece551f3"}},{"code_sha256_prefix":"a7a732117e3ad716","entry":"DotDict","repo":"MurtyShikhar/Pushdown-Layers","repo_kind":"official","path":"models/pushdown_transformer_lm.py","file_url":"https://github.com/MurtyShikhar/Pushdown-Layers/blob/HEAD/models/pushdown_transformer_lm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a7a732117e3ad716"}},{"code_sha256_prefix":"6ea6715c721ed9a7","entry":"MultiHeadAttentionBase","repo":"MurtyShikhar/Pushdown-Layers","repo_kind":"official","path":"models/pushdown_transformer_lm.py","file_url":"https://github.com/MurtyShikhar/Pushdown-Layers/blob/HEAD/models/pushdown_transformer_lm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6ea6715c721ed9a7"}},{"code_sha256_prefix":"545b1940b792fa22","entry":"PositionalEncoding","repo":"MurtyShikhar/Pushdown-Layers","repo_kind":"official","path":"models/pushdown_transformer_lm.py","file_url":"https://github.com/MurtyShikhar/Pushdown-Layers/blob/HEAD/models/pushdown_transformer_lm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"545b1940b792fa22"}},{"code_sha256_prefix":"59316687afde843b","entry":"RelativeAttentionBase","repo":"MurtyShikhar/Pushdown-Layers","repo_kind":"official","path":"models/pushdown_transformer_lm.py","file_url":"https://github.com/MurtyShikhar/Pushdown-Layers/blob/HEAD/models/pushdown_transformer_lm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"59316687afde843b"}},{"code_sha256_prefix":"5612e13b2384e5b0","entry":"TiedEmbedding","repo":"MurtyShikhar/Pushdown-Layers","repo_kind":"official","path":"models/pushdown_transformer_lm.py","file_url":"https://github.com/MurtyShikhar/Pushdown-Layers/blob/HEAD/models/pushdown_transformer_lm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5612e13b2384e5b0"}},{"code_sha256_prefix":"217b95213cfe1f2b","entry":"TransformerEncoder","repo":"MurtyShikhar/Pushdown-Layers","repo_kind":"official","path":"models/pushdown_transformer_lm.py","file_url":"https://github.com/MurtyShikhar/Pushdown-Layers/blob/HEAD/models/pushdown_transformer_lm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"217b95213cfe1f2b"}},{"code_sha256_prefix":"cd78addc129539b3","entry":"TransformerResult","repo":"MurtyShikhar/Pushdown-Layers","repo_kind":"official","path":"models/pushdown_transformer_lm.py","file_url":"https://github.com/MurtyShikhar/Pushdown-Layers/blob/HEAD/models/pushdown_transformer_lm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cd78addc129539b3"}},{"code_sha256_prefix":"70622ae82f44b450","entry":"compute_per_token_logprob","repo":"murtyshikhar/pushdown-layers","repo_kind":"official","path":"pushdown_util.py","file_url":"https://github.com/murtyshikhar/pushdown-layers/blob/HEAD/pushdown_util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"70622ae82f44b450"}},{"code_sha256_prefix":"dcf24b95d9f14aa9","entry":"compute_stack_logprobs","repo":"murtyshikhar/pushdown-layers","repo_kind":"official","path":"pushdown_util.py","file_url":"https://github.com/murtyshikhar/pushdown-layers/blob/HEAD/pushdown_util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dcf24b95d9f14aa9"}},{"code_sha256_prefix":"45d4819a42863712","entry":"get_parsing_accuracy","repo":"murtyshikhar/pushdown-layers","repo_kind":"official","path":"eval_utils/eval_pushdown_model.py","file_url":"https://github.com/murtyshikhar/pushdown-layers/blob/HEAD/eval_utils/eval_pushdown_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"45d4819a42863712"}},{"code_sha256_prefix":"6b331f505ce51944","entry":"logsumexp","repo":"murtyshikhar/pushdown-layers","repo_kind":"official","path":"beam_search_util.py","file_url":"https://github.com/murtyshikhar/pushdown-layers/blob/HEAD/beam_search_util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6b331f505ce51944"}},{"code_sha256_prefix":"f24ed9058bec8d35","entry":"sinusoidal_pos_embedding","repo":"MurtyShikhar/Pushdown-Layers","repo_kind":"official","path":"models/pushdown_transformer_lm.py","file_url":"https://github.com/MurtyShikhar/Pushdown-Layers/blob/HEAD/models/pushdown_transformer_lm.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f24ed9058bec8d35"}},{"code_sha256_prefix":"d6b9018a0880eafd","entry":"sinusoidal_pos_embedding","repo":"murtyshikhar/pushdown-layers","repo_kind":"official","path":"layers/positional_encoding.py","file_url":"https://github.com/murtyshikhar/pushdown-layers/blob/HEAD/layers/positional_encoding.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d6b9018a0880eafd"}},{"code_sha256_prefix":"040a0db1fa164ee2","entry":"PushdownAttention","repo":"MurtyShikhar/Pushdown-Layers","repo_kind":"official","path":"models/pushdown_transformer_lm.py","file_url":"https://github.com/MurtyShikhar/Pushdown-Layers/blob/HEAD/models/pushdown_transformer_lm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"040a0db1fa164ee2"}},{"code_sha256_prefix":"1d6a2fb2322500d7","entry":"PushdownLM","repo":"MurtyShikhar/Pushdown-Layers","repo_kind":"official","path":"models/pushdown_transformer_lm.py","file_url":"https://github.com/MurtyShikhar/Pushdown-Layers/blob/HEAD/models/pushdown_transformer_lm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1d6a2fb2322500d7"}},{"code_sha256_prefix":"ea75be5e2691ee1e","entry":"StackPredictor","repo":"MurtyShikhar/Pushdown-Layers","repo_kind":"official","path":"models/pushdown_transformer_lm.py","file_url":"https://github.com/MurtyShikhar/Pushdown-Layers/blob/HEAD/models/pushdown_transformer_lm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ea75be5e2691ee1e"}},{"code_sha256_prefix":"e1e85555ec42de79","entry":"Transformer","repo":"MurtyShikhar/Pushdown-Layers","repo_kind":"official","path":"models/pushdown_transformer_lm.py","file_url":"https://github.com/MurtyShikhar/Pushdown-Layers/blob/HEAD/models/pushdown_transformer_lm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e1e85555ec42de79"}},{"code_sha256_prefix":"cd05f27177c97848","entry":"TransformerEncoderLayer","repo":"MurtyShikhar/Pushdown-Layers","repo_kind":"official","path":"models/pushdown_transformer_lm.py","file_url":"https://github.com/MurtyShikhar/Pushdown-Layers/blob/HEAD/models/pushdown_transformer_lm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cd05f27177c97848"}},{"code_sha256_prefix":"5d6f6fc2cd657722","entry":"TransformerEncoderWithLayer","repo":"MurtyShikhar/Pushdown-Layers","repo_kind":"official","path":"models/pushdown_transformer_lm.py","file_url":"https://github.com/MurtyShikhar/Pushdown-Layers/blob/HEAD/models/pushdown_transformer_lm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5d6f6fc2cd657722"}},{"code_sha256_prefix":"9f7833997a6a7918","entry":"TransformerEncoderWithLayer","repo":"murtyshikhar/pushdown-layers","repo_kind":"official","path":"layers/transformer/transformer.py","file_url":"https://github.com/murtyshikhar/pushdown-layers/blob/HEAD/layers/transformer/transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9f7833997a6a7918"}},{"code_sha256_prefix":"199613ff3592d354","entry":"TransformerLM","repo":"MurtyShikhar/Pushdown-Layers","repo_kind":"official","path":"models/pushdown_transformer_lm.py","file_url":"https://github.com/MurtyShikhar/Pushdown-Layers/blob/HEAD/models/pushdown_transformer_lm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"199613ff3592d354"}},{"code_sha256_prefix":"de36bbda5931ac11","entry":"cross_entropy","repo":"murtyshikhar/pushdown-layers","repo_kind":"official","path":"layers/cross_entropy_label_smoothing.py","file_url":"https://github.com/murtyshikhar/pushdown-layers/blob/HEAD/layers/cross_entropy_label_smoothing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"de36bbda5931ac11"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}