{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/neural-networks-with-differentiable-structure","title":"Neural networks with differentiable structure","arxiv_id":"1606.06216","date":"2016-06-20","proceeding":null,"authors":["Thomas Miconi"],"abstract":"While gradient descent has proven highly successful in learning connection\nweights for neural networks, the actual structure of these networks is usually\ndetermined by hand, or by other optimization algorithms. Here we describe a\nsimple method to make network structure differentiable, and therefore\naccessible to gradient descent. We test this method on recurrent neural\nnetworks applied to simple sequence prediction problems. Starting with initial\nnetworks containing only one node, the method automatically builds networks\nthat successfully solve the tasks. The number of nodes in the final network\ncorrelates with task difficulty. The method can dynamically increase network\nsize in response to an abrupt complexification in the task; however, reduction\nin network size in response to task simplification is not evident for\nreasonable meta-parameters. The method does not penalize network performance\nfor these test tasks: variable-size networks actually reach better performance\nthan fixed-size networks of higher, lower or identical size. We conclude by\ndiscussing how this method could be applied to more complex networks, such as\nfeedforward layered networks, or multiple-area networks of arbitrary shape.","url_abs":"http://arxiv.org/abs/1606.06216v3","url_pdf":"http://arxiv.org/pdf/1606.06216v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"neural-networks-with-differentiable-structure","repo_url":"https://github.com/ThomasMiconi/DiffRNN","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1606.06216","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}