{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/low-rank-passthrough-neural-networks","title":"Low-rank passthrough neural networks","arxiv_id":"1603.03116","date":"2016-03-10","proceeding":"WS 2018 7","authors":["Antonio Valerio Miceli Barone"],"abstract":"Various common deep learning architectures, such as LSTMs, GRUs, Resnets and\nHighway Networks, employ state passthrough connections that support training\nwith high feed-forward depth or recurrence over many time steps. These\n\"Passthrough Networks\" architectures also enable the decoupling of the network\nstate size from the number of parameters of the network, a possibility has been\nstudied by \\newcite{Sak2014} with their low-rank parametrization of the LSTM.\nIn this work we extend this line of research, proposing effective, low-rank and\nlow-rank plus diagonal matrix parametrizations for Passthrough Networks which\nexploit this decoupling property, reducing the data complexity and memory\nrequirements of the network while preserving its memory capacity. This is\nparticularly beneficial in low-resource settings as it supports expressive\nmodels with a compact parametrization less susceptible to overfitting. We\npresent competitive experimental results on several tasks, including language\nmodeling and a near state of the art result on sequential randomly-permuted\nMNIST classification, a hard task on natural data.","url_abs":"http://arxiv.org/abs/1603.03116v3","url_pdf":"http://arxiv.org/pdf/1603.03116v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"low-rank-passthrough-neural-networks","repo_url":"https://github.com/Avmb/lowrank-gru","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"low-rank-passthrough-neural-networks","repo_url":"https://github.com/Avmb/lowrank-highwaynetwork","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"low-rank-passthrough-neural-networks","repo_url":"https://github.com/Avmb/dl4mt-lm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null},{"paper_slug":"low-rank-passthrough-neural-networks","repo_url":"https://github.com/Avmb/lowrank-lstm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"torch","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":null,"task_name":"Permuted-MNIST"}],"methods":[{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}