{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/neural-gpus-learn-algorithms","title":"Neural GPUs Learn Algorithms","arxiv_id":"1511.08228","date":"2015-11-25","proceeding":null,"authors":["Łukasz Kaiser","Ilya Sutskever"],"abstract":"Learning an algorithm from examples is a fundamental problem that has been\nwidely studied. Recently it has been addressed using neural networks, in\nparticular by Neural Turing Machines (NTMs). These are fully differentiable\ncomputers that use backpropagation to learn their own programming. Despite\ntheir appeal NTMs have a weakness that is caused by their sequential nature:\nthey are not parallel and are are hard to train due to their large depth when\nunfolded.\n  We present a neural network architecture to address this problem: the Neural\nGPU. It is based on a type of convolutional gated recurrent unit and, like the\nNTM, is computationally universal. Unlike the NTM, the Neural GPU is highly\nparallel which makes it easier to train and efficient to run.\n  An essential property of algorithms is their ability to handle inputs of\narbitrary size. We show that the Neural GPU can be trained on short instances\nof an algorithmic task and successfully generalize to long instances. We\nverified it on a number of tasks including long addition and long\nmultiplication of numbers represented in binary. We train the Neural GPU on\nnumbers with upto 20 bits and observe no errors whatsoever while testing it,\neven on much longer numbers.\n  To achieve these results we introduce a technique for training deep recurrent\nnetworks: parameter sharing relaxation. We also found a small amount of dropout\nand gradient noise to have a large positive effect on learning and\ngeneralization.","url_abs":"http://arxiv.org/abs/1511.08228v3","url_pdf":"http://arxiv.org/pdf/1511.08228v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"neural-gpus-learn-algorithms","repo_url":"https://github.com/tensorflow/models","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":null},{"paper_slug":"neural-gpus-learn-algorithms","repo_url":"https://github.com/openai/ecprice-neural-gpu","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"neural-gpus-learn-algorithms","repo_url":"https://github.com/openai/neural-gpu","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"neural-gpus-learn-algorithms","repo_url":"https://github.com/tensorflow/models/tree/master/research/neural_gpu","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"neural-gpus-learn-algorithms","repo_url":"https://github.com/zhuyawen/neural-gpu","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}}],"tasks":[{"task_slug":null,"task_name":"GPU"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1511.08228","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}