{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/discovering-autoregressive-orderings-with","title":"Discovering Autoregressive Orderings with Variational Inference","arxiv_id":null,"date":"2021-01-01","proceeding":"ICLR 2021 1","authors":["Xuanlin Li","Brandon Trabucco","Dong Huk Park","Michael Luo","Sheng Shen","Trevor Darrell","Yang Gao"],"abstract":"The predominant approach for language modeling is to encode a sequence of tokens from left to right, but this eliminates a source of information: the order by which the sequence was naturally generated. One strategy to recover this information is to decode both the content and location of tokens. Prior work supervises content and location with hand-designed loss functions or bootstraps from a predefined ordering. These approaches require domain-specific insight. We address this limitation with an unsupervised learner that discovers high-quality autoregressive orders without domain-specific prior. Our learner is a neural network that performs variational inference with the autoregressive order as a latent variable. The corresponding ELBO is not differentiable, so we develop a practical algorithm for end-to-end optimization using policy gradients. Strong empirical results with our solution on image captioning and code generation suggest that our algorithm is capable of discovering various autoregressive orders for different sequences that are competitive with or better than fixed orders.","url_abs":"https://openreview.net/forum?id=jP1vTH3inC","url_pdf":"https://openreview.net/pdf?id=jP1vTH3inC","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"discovering-autoregressive-orderings-with","repo_url":"https://github.com/xuanlinli17/autoregressive_inference","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"variational-inference","task_name":"Variational Inference"}],"methods":[{"method_slug":"variational-inference","method_name":"Variational Inference"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}