{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/decoupled-neural-interfaces-using-synthetic","title":"Decoupled Neural Interfaces using Synthetic Gradients","arxiv_id":"1608.05343","date":"2016-08-18","proceeding":"ICML 2017 8","authors":["Max Jaderberg","Wojciech Marian Czarnecki","Simon Osindero","Oriol Vinyals","Alex Graves","David Silver","Koray Kavukcuoglu"],"abstract":"Training directed neural networks typically requires forward-propagating data\nthrough a computation graph, followed by backpropagating error signal, to\nproduce weight updates. All layers, or more generally, modules, of the network\nare therefore locked, in the sense that they must wait for the remainder of the\nnetwork to execute forwards and propagate error backwards before they can be\nupdated. In this work we break this constraint by decoupling modules by\nintroducing a model of the future computation of the network graph. These\nmodels predict what the result of the modelled subgraph will produce using only\nlocal information. In particular we focus on modelling error gradients: by\nusing the modelled synthetic gradient in place of true backpropagated error\ngradients we decouple subgraphs, and can update them independently and\nasynchronously i.e. we realise decoupled neural interfaces. We show results for\nfeed-forward models, where every layer is trained asynchronously, recurrent\nneural networks (RNNs) where predicting one's future gradient extends the time\nover which the RNN can effectively model, and also a hierarchical RNN system\nwith ticking at different timescales. Finally, we demonstrate that in addition\nto predicting gradients, the same framework can be used to predict inputs,\nresulting in models which are decoupled in both the forward and backwards pass\n-- amounting to independent networks which co-learn such that they can be\ncomposed into a single functioning corporation.","url_abs":"http://arxiv.org/abs/1608.05343v2","url_pdf":"http://arxiv.org/pdf/1608.05343v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"decoupled-neural-interfaces-using-synthetic","repo_url":"https://github.com/TheoryDev/Deep-neural-network-training-optimisation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"decoupled-neural-interfaces-using-synthetic","repo_url":"https://github.com/koz4k/dni-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"decoupled-neural-interfaces-using-synthetic","repo_url":"https://github.com/mishig25/synthetic-gradients-keras","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"decoupled-neural-interfaces-using-synthetic","repo_url":"https://github.com/mtanghu/DNI-RNN","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"decoupled-neural-interfaces-using-synthetic","repo_url":"https://github.com/unconst/MACH","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1608.05343","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}