{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/extensions-and-limitations-of-the-neural-gpu","title":"Extensions and Limitations of the Neural GPU","arxiv_id":"1611.00736","date":"2016-11-02","proceeding":null,"authors":["Eric Price","Wojciech Zaremba","Ilya Sutskever"],"abstract":"The Neural GPU is a recent model that can learn algorithms such as\nmulti-digit binary addition and binary multiplication in a way that generalizes\nto inputs of arbitrary length. We show that there are two simple ways of\nimproving the performance of the Neural GPU: by carefully designing a\ncurriculum, and by increasing model size. The latter requires a memory\nefficient implementation, as a naive implementation of the Neural GPU is memory\nintensive. We find that these techniques increase the set of algorithmic\nproblems that can be solved by the Neural GPU: we have been able to learn to\nperform all the arithmetic operations (and generalize to arbitrarily long\nnumbers) when the arguments are given in the decimal representation (which,\nsurprisingly, has not been possible before). We have also been able to train\nthe Neural GPU to evaluate long arithmetic expressions with multiple operands\nthat require respecting the precedence order of the operands, although these\nhave succeeded only in their binary representation, and not with perfect\naccuracy.\n  In addition, we gain insight into the Neural GPU by investigating its failure\nmodes. We find that Neural GPUs that correctly generalize to arbitrarily long\nnumbers still fail to compute the correct answer on highly-symmetric, atypical\ninputs: for example, a Neural GPU that achieves near-perfect generalization on\ndecimal multiplication of up to 100-digit long numbers can fail on\n$000000\\dots002 \\times 000000\\dots002$ while succeeding at $2 \\times 2$. These\nfailure modes are reminiscent of adversarial examples.","url_abs":"http://arxiv.org/abs/1611.00736v2","url_pdf":"http://arxiv.org/pdf/1611.00736v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"extensions-and-limitations-of-the-neural-gpu","repo_url":"https://github.com/openai/ecprice-neural-gpu","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":null,"task_name":"GPU"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1611.00736","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}