{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/optimizing-performance-of-recurrent-neural","title":"Optimizing Performance of Recurrent Neural Networks on GPUs","arxiv_id":"1604.01946","date":"2016-04-07","proceeding":null,"authors":["Jeremy Appleyard","Tomas Kocisky","Phil Blunsom"],"abstract":"As recurrent neural networks become larger and deeper, training times for\nsingle networks are rising into weeks or even months. As such there is a\nsignificant incentive to improve the performance and scalability of these\nnetworks. While GPUs have become the hardware of choice for training and\ndeploying recurrent models, the implementations employed often make use of only\nbasic optimizations for these architectures. In this article we demonstrate\nthat by exposing parallelism between operations within the network, an order of\nmagnitude speedup across a range of network sizes can be achieved over a naive\nimplementation. We describe three stages of optimization that have been\nincorporated into the fifth release of NVIDIA's cuDNN: firstly optimizing a\nsingle cell, secondly a single layer, and thirdly the entire network.","url_abs":"http://arxiv.org/abs/1604.01946v1","url_pdf":"http://arxiv.org/pdf/1604.01946v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"optimizing-performance-of-recurrent-neural","repo_url":"https://github.com/parallel-forall/code-samples","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1604.01946","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}