{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-sparse-transformations-through","title":"Learning sparse transformations through backpropagation","arxiv_id":"1810.09184","date":"2018-10-22","proceeding":null,"authors":["Peter Bloem"],"abstract":"Many transformations in deep learning architectures are sparsely connected.\nWhen such transformations cannot be designed by hand, they can be learned, even\nthrough plain backpropagation, for instance in attention mechanisms. However,\nduring learning, such sparse structures are often represented in a dense form,\nas we do not know beforehand which elements will eventually become non-zero. We\nintroduce the adaptive, sparse hyperlayer, a method for learning a sparse\ntransformation, paramatrized sparsely: as index-tuples with associated values.\nTo overcome the lack of gradients from such a discrete structure, we introduce\na method of randomly sampling connections, and backpropagating over the\nrandomly wired computation graph. To show that this approach allows us to train\na model to competitive performance on real data, we use it to build two\narchitectures. First, an attention mechanism for visual classification. Second,\nwe implement a method for differentiable sorting: specifically, learning to\nsort unlabeled MNIST digits, given only the correct order.","url_abs":"http://arxiv.org/abs/1810.09184v1","url_pdf":"http://arxiv.org/pdf/1810.09184v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-sparse-transformations-through","repo_url":"https://github.com/MaestroGraph/sparse-hyper","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"learning-sparse-transformations-through","repo_url":"https://github.com/MaestroGraph/quicksort","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}