{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-hierarchical-structures-with","title":"Learning Hierarchical Structures with Differentiable Nondeterministic Stacks","arxiv_id":"2109.01982","date":"2021-09-05","proceeding":"ICLR 2022 4","authors":["Brian DuSell","David Chiang"],"abstract":"Learning hierarchical structures in sequential data -- from simple algorithmic patterns to natural language -- in a reliable, generalizable way remains a challenging problem for neural language models. Past work has shown that recurrent neural networks (RNNs) struggle to generalize on held-out algorithmic or syntactic patterns without supervision or some inductive bias. To remedy this, many papers have explored augmenting RNNs with various differentiable stacks, by analogy with finite automata and pushdown automata (PDAs). In this paper, we improve the performance of our recently proposed Nondeterministic Stack RNN (NS-RNN), which uses a differentiable data structure that simulates a nondeterministic PDA, with two important changes. First, the model now assigns unnormalized positive weights instead of probabilities to stack actions, and we provide an analysis of why this improves training. Second, the model can directly observe the state of the underlying PDA. Our model achieves lower cross-entropy than all previous stack RNNs on five context-free language modeling tasks (within 0.05 nats of the information-theoretic lower bound), including a task on which the NS-RNN previously failed to outperform a deterministic stack RNN baseline. Finally, we propose a restricted version of the NS-RNN that incrementally processes infinitely long sequences, and we present language modeling results on the Penn Treebank.","url_abs":"https://arxiv.org/abs/2109.01982v3","url_pdf":"https://arxiv.org/pdf/2109.01982v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-hierarchical-structures-with","repo_url":"https://github.com/bdusell/nondeterministic-stack-rnn","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"inductive-bias","task_name":"Inductive Bias"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2109.01982","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2109.01982"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/bdusell/nondeterministic-stack-rnn","reach":null}],"summary":{"ran_honours":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"64a2db52cb99b8d7","entry":"ensure_not_negative","repo":"bdusell/nondeterministic-stack-rnn","repo_kind":"official","path":"src/stack_rnn_models/nondeterministic_stack.py","file_url":"https://github.com/bdusell/nondeterministic-stack-rnn/blob/HEAD/src/stack_rnn_models/nondeterministic_stack.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":2,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"64a2db52cb99b8d7"}},{"code_sha256_prefix":"cd507999d35f69ca","entry":"alpha_j_index","repo":"bdusell/nondeterministic-stack-rnn","repo_kind":"official","path":"src/stack_rnn_models/nondeterministic_stack.py","file_url":"https://github.com/bdusell/nondeterministic-stack-rnn/blob/HEAD/src/stack_rnn_models/nondeterministic_stack.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cd507999d35f69ca"}},{"code_sha256_prefix":"d17f9b2a1bbe7c9a","entry":"gamma_i_index","repo":"bdusell/nondeterministic-stack-rnn","repo_kind":"official","path":"src/stack_rnn_models/nondeterministic_stack.py","file_url":"https://github.com/bdusell/nondeterministic-stack-rnn/blob/HEAD/src/stack_rnn_models/nondeterministic_stack.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d17f9b2a1bbe7c9a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}