{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-elm-neuron-an-efficient-and-expressive","title":"The Expressive Leaky Memory Neuron: an Efficient and Expressive Phenomenological Neuron Model Can Solve Long-Horizon Tasks","arxiv_id":"2306.16922","date":"2023-06-14","proceeding":null,"authors":["Aaron Spieler","Nasim Rahaman","Georg Martius","Bernhard Schölkopf","Anna Levina"],"abstract":"Biological cortical neurons are remarkably sophisticated computational devices, temporally integrating their vast synaptic input over an intricate dendritic tree, subject to complex, nonlinearly interacting internal biological processes. A recent study proposed to characterize this complexity by fitting accurate surrogate models to replicate the input-output relationship of a detailed biophysical cortical pyramidal neuron model and discovered it needed temporal convolutional networks (TCN) with millions of parameters. Requiring these many parameters, however, could stem from a misalignment between the inductive biases of the TCN and cortical neuron's computations. In light of this, and to explore the computational implications of leaky memory units and nonlinear dendritic processing, we introduce the Expressive Leaky Memory (ELM) neuron model, a biologically inspired phenomenological model of a cortical neuron. Remarkably, by exploiting such slowly decaying memory-like hidden states and two-layered nonlinear integration of synaptic input, our ELM neuron can accurately match the aforementioned input-output relationship with under ten thousand trainable parameters. To further assess the computational ramifications of our neuron design, we evaluate it on various tasks with demanding temporal structures, including the Long Range Arena (LRA) datasets, as well as a novel neuromorphic dataset based on the Spiking Heidelberg Digits dataset (SHD-Adding). Leveraging a larger number of memory units with sufficiently long timescales, and correspondingly sophisticated synaptic integration, the ELM neuron displays substantial long-range processing capabilities, reliably outperforming the classic Transformer or Chrono-LSTM architectures on LRA, and even solving the Pathfinder-X task with over 70% accuracy (16k context length).","url_abs":"https://arxiv.org/abs/2306.16922v3","url_pdf":"https://arxiv.org/pdf/2306.16922v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-elm-neuron-an-efficient-and-expressive","repo_url":"https://github.com/AaronSpieler/elmneuron","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"16k","task_name":"16k"},{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"long-range-modeling","task_name":"Long-range modeling"},{"task_slug":null,"task_name":"Pathfinder"},{"task_slug":null,"task_name":"Pathfinder-X"},{"task_slug":"time-series-1","task_name":"Time Series"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[{"slug":"shd-adding","name":"SHD - Adding","full_name":"Spiking Heidelberg Digits - Adding"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/classification-on-shd-adding","task":"Classification","dataset":"SHD - Adding","model":"ELM Neuron","rank_in_archive_order":1,"of":3,"metrics":{"Accuracy (%)":"82"},"uses_additional_data":false},{"leaderboard":"/sota/classification-on-shd-adding","task":"Classification","dataset":"SHD - Adding","model":"LSTM","rank_in_archive_order":2,"of":3,"metrics":{"Accuracy (%)":"10"},"uses_additional_data":false},{"leaderboard":"/sota/classification-on-shd-adding","task":"Classification","dataset":"SHD - Adding","model":"LIF-SNN","rank_in_archive_order":3,"of":3,"metrics":{"Accuracy (%)":"FAIL"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2306.16922","atlas_url":"https://app.syntology.ai/?focus=2306.16922","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.16922"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/AaronSpieler/elmneuron","reach":null}],"summary":{"ran":1,"ran_honours":2,"ran_draft_wrong":3,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2af19698c2b90183","entry":"MLP","repo":"AaronSpieler/elmneuron","repo_kind":"official","path":"src/expressive_leaky_memory_neuron.py","file_url":"https://github.com/AaronSpieler/elmneuron/blob/HEAD/src/expressive_leaky_memory_neuron.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2af19698c2b90183"}},{"code_sha256_prefix":"6e3086b79a2f27d1","entry":"create_interlocking_indices","repo":"AaronSpieler/elmneuron","repo_kind":"official","path":"src/expressive_leaky_memory_neuron.py","file_url":"https://github.com/AaronSpieler/elmneuron/blob/HEAD/src/expressive_leaky_memory_neuron.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6e3086b79a2f27d1"}},{"code_sha256_prefix":"64ff7be08903844e","entry":"create_overlapping_window_indices","repo":"AaronSpieler/elmneuron","repo_kind":"official","path":"src/expressive_leaky_memory_neuron.py","file_url":"https://github.com/AaronSpieler/elmneuron/blob/HEAD/src/expressive_leaky_memory_neuron.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"64ff7be08903844e"}},{"code_sha256_prefix":"f58a44453a26de81","entry":"custom_tanh","repo":"AaronSpieler/elmneuron","repo_kind":"official","path":"src/expressive_leaky_memory_neuron.py","file_url":"https://github.com/AaronSpieler/elmneuron/blob/HEAD/src/expressive_leaky_memory_neuron.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f58a44453a26de81"}},{"code_sha256_prefix":"6aaf7ee986c488fd","entry":"inverse_scaled_sigmoid","repo":"AaronSpieler/elmneuron","repo_kind":"official","path":"src/expressive_leaky_memory_neuron.py","file_url":"https://github.com/AaronSpieler/elmneuron/blob/HEAD/src/expressive_leaky_memory_neuron.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6aaf7ee986c488fd"}},{"code_sha256_prefix":"b1eb6b8362771ad8","entry":"scaled_sigmoid","repo":"AaronSpieler/elmneuron","repo_kind":"official","path":"src/expressive_leaky_memory_neuron.py","file_url":"https://github.com/AaronSpieler/elmneuron/blob/HEAD/src/expressive_leaky_memory_neuron.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1eb6b8362771ad8"}},{"code_sha256_prefix":"e71be8b98d7d0433","entry":"ELM","repo":"AaronSpieler/elmneuron","repo_kind":"official","path":"src/expressive_leaky_memory_neuron.py","file_url":"https://github.com/AaronSpieler/elmneuron/blob/HEAD/src/expressive_leaky_memory_neuron.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e71be8b98d7d0433"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}