{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/egru-event-based-gru-for-activity-sparse","title":"Efficient recurrent architectures through activity sparsity and sparse back-propagation through time","arxiv_id":"2206.06178","date":"2022-06-13","proceeding":null,"authors":["Anand Subramoney","Khaleelulla Khan Nazeer","Mark Schöne","Christian Mayr","David Kappel"],"abstract":"Recurrent neural networks (RNNs) are well suited for solving sequence tasks in resource-constrained systems due to their expressivity and low computational requirements. However, there is still a need to bridge the gap between what RNNs are capable of in terms of efficiency and performance and real-world application requirements. The memory and computational requirements arising from propagating the activations of all the neurons at every time step to every connected neuron, together with the sequential dependence of activations, contribute to the inefficiency of training and using RNNs. We propose a solution inspired by biological neuron dynamics that makes the communication between RNN units sparse and discrete. This makes the backward pass with backpropagation through time (BPTT) computationally sparse and efficient as well. We base our model on the gated recurrent unit (GRU), extending it with units that emit discrete events for communication triggered by a threshold so that no information is communicated to other units in the absence of events. We show theoretically that the communication between units, and hence the computation required for both the forward and backward passes, scales with the number of events in the network. Our model achieves efficiency without compromising task performance, demonstrating competitive performance compared to state-of-the-art recurrent network models in real-world tasks, including language modeling. The dynamic activity sparsity mechanism also makes our model well suited for novel energy-efficient neuromorphic hardware. Code is available at https://github.com/KhaleelKhan/EvNN/.","url_abs":"https://arxiv.org/abs/2206.06178v3","url_pdf":"https://arxiv.org/pdf/2206.06178v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"egru-event-based-gru-for-activity-sparse","repo_url":"https://github.com/khaleelkhan/evnn","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"GPL-3.0"}}],"tasks":[{"task_slug":"gesture-recognition","task_name":"Gesture Recognition"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"sequential-image-classification","task_name":"Sequential Image Classification"},{"task_slug":"time-series","task_name":"Time Series Analysis"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"base","method_name":"BASE"},{"method_slug":"gru","method_name":"GRU"},{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/gesture-recognition-on-dvs128-gesture","task":"Gesture Recognition","dataset":"DVS128 Gesture","model":"EGRU","rank_in_archive_order":3,"of":14,"metrics":{"Accuracy (%)":"97.8"},"uses_additional_data":true},{"leaderboard":"/sota/language-modelling-on-wikitext-2","task":"Language Modelling","dataset":"WikiText-2","model":"EGRU","rank_in_archive_order":34,"of":38,"metrics":{"Test perplexity":"68.9"},"uses_additional_data":false},{"leaderboard":"/sota/sequential-image-classification-on-sequential","task":"Sequential Image Classification","dataset":"Sequential MNIST","model":"EGRU","rank_in_archive_order":23,"of":30,"metrics":{"Permuted Accuracy":"95.1%","Unpermuted Accuracy":"98.3%"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2206.06178","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}