{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/discovering-non-monotonic-autoregressive","title":"Discovering Non-monotonic Autoregressive Orderings with Variational Inference","arxiv_id":"2110.15797","date":"2021-10-27","proceeding":null,"authors":["Xuanlin Li","Brandon Trabucco","Dong Huk Park","Michael Luo","Sheng Shen","Trevor Darrell","Yang Gao"],"abstract":"The predominant approach for language modeling is to process sequences from left to right, but this eliminates a source of information: the order by which the sequence was generated. One strategy to recover this information is to decode both the content and ordering of tokens. Existing approaches supervise content and ordering by designing problem-specific loss functions and pre-training with an ordering pre-selected. Other recent works use iterative search to discover problem-specific orderings for training, but suffer from high time complexity and cannot be efficiently parallelized. We address these limitations with an unsupervised parallelizable learner that discovers high-quality generation orders purely from training data -- no domain knowledge required. The learner contains an encoder network and decoder language model that perform variational inference with autoregressive orders (represented as permutation matrices) as latent variables. The corresponding ELBO is not differentiable, so we develop a practical algorithm for end-to-end optimization using policy gradients. We implement the encoder as a Transformer with non-causal attention that outputs permutations in one forward pass. Permutations then serve as target generation orders for training an insertion-based Transformer language model. Empirical results in language modeling tasks demonstrate that our method is context-aware and discovers orderings that are competitive with or even better than fixed orders.","url_abs":"https://arxiv.org/abs/2110.15797v1","url_pdf":"https://arxiv.org/pdf/2110.15797v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"discovering-non-monotonic-autoregressive","repo_url":"https://github.com/xuanlinli17/autoregressive_inference","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"text-summarization","task_name":"Text Summarization"},{"task_slug":"variational-inference","task_name":"Variational Inference"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"variational-inference","method_name":"Variational Inference"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2110.15797","atlas_url":"https://app.syntology.ai/?focus=2110.15797","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2110.15797"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xuanlinli17/autoregressive_inference","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":6},"by_repo_kind":{"official":{"samples":7,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f1dd530f5c706f2f","entry":"levenshtein","repo":"xuanlinli17/autoregressive_inference","repo_kind":"official","path":"voi/algorithms/levenshtein.py","file_url":"https://github.com/xuanlinli17/autoregressive_inference/blob/HEAD/voi/algorithms/levenshtein.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f1dd530f5c706f2f"}},{"code_sha256_prefix":"082c009a23cb13d4","entry":"adaptive_search","repo":"xuanlinli17/autoregressive_inference","repo_kind":"official","path":"voi/algorithms/adaptive_search.py","file_url":"https://github.com/xuanlinli17/autoregressive_inference/blob/HEAD/voi/algorithms/adaptive_search.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"082c009a23cb13d4"}},{"code_sha256_prefix":"ec90483fc2b722f6","entry":"beam_search","repo":"xuanlinli17/autoregressive_inference","repo_kind":"official","path":"voi/algorithms/beam_search.py","file_url":"https://github.com/xuanlinli17/autoregressive_inference/blob/HEAD/voi/algorithms/beam_search.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec90483fc2b722f6"}},{"code_sha256_prefix":"6f6c9fbef829da96","entry":"greedy_search","repo":"xuanlinli17/autoregressive_inference","repo_kind":"official","path":"voi/algorithms/greedy_search.py","file_url":"https://github.com/xuanlinli17/autoregressive_inference/blob/HEAD/voi/algorithms/greedy_search.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6f6c9fbef829da96"}},{"code_sha256_prefix":"6d0af24619eaf852","entry":"nucleus_sampling","repo":"xuanlinli17/autoregressive_inference","repo_kind":"official","path":"voi/algorithms/nucleus_sampling.py","file_url":"https://github.com/xuanlinli17/autoregressive_inference/blob/HEAD/voi/algorithms/nucleus_sampling.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6d0af24619eaf852"}},{"code_sha256_prefix":"4c81b400a2cb458f","entry":"prepare_batch_for_lm_captioning","repo":"xuanlinli17/autoregressive_inference","repo_kind":"official","path":"voi/core/batch_prepare_utils.py","file_url":"https://github.com/xuanlinli17/autoregressive_inference/blob/HEAD/voi/core/batch_prepare_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4c81b400a2cb458f"}},{"code_sha256_prefix":"899639890dfcac04","entry":"prepare_batch_for_lm_wmt","repo":"xuanlinli17/autoregressive_inference","repo_kind":"official","path":"voi/core/batch_prepare_utils.py","file_url":"https://github.com/xuanlinli17/autoregressive_inference/blob/HEAD/voi/core/batch_prepare_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"899639890dfcac04"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}