{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/spikingbert-distilling-bert-to-train-spiking","title":"SpikingBERT: Distilling BERT to Train Spiking Language Models Using Implicit Differentiation","arxiv_id":"2308.10873","date":"2023-08-21","proceeding":null,"authors":["Malyaban Bal","Abhronil Sengupta"],"abstract":"Large language Models (LLMs), though growing exceedingly powerful, comprises of orders of magnitude less neurons and synapses than the human brain. However, it requires significantly more power/energy to operate. In this work, we propose a novel bio-inspired spiking language model (LM) which aims to reduce the computational cost of conventional LMs by drawing motivation from the synaptic information flow in the brain. In this paper, we demonstrate a framework that leverages the average spiking rate of neurons at equilibrium to train a neuromorphic spiking LM using implicit differentiation technique, thereby overcoming the non-differentiability problem of spiking neural network (SNN) based algorithms without using any type of surrogate gradient. The steady-state convergence of the spiking neurons also allows us to design a spiking attention mechanism, which is critical in developing a scalable spiking LM. Moreover, the convergence of average spiking rate of neurons at equilibrium is utilized to develop a novel ANN-SNN knowledge distillation based technique wherein we use a pre-trained BERT model as \"teacher\" to train our \"student\" spiking architecture. While the primary architecture proposed in this paper is motivated by BERT, the technique can be potentially extended to different kinds of LLMs. Our work is the first one to demonstrate the performance of an operational spiking LM architecture on multiple different tasks in the GLUE benchmark.","url_abs":"https://arxiv.org/abs/2308.10873v3","url_pdf":"https://arxiv.org/pdf/2308.10873v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"spikingbert-distilling-bert-to-train-spiking","repo_url":"https://github.com/neurocomplab-psu/spikingbert","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2308.10873","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.10873"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/neurocomplab-psu/spikingbert","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":4,"ran_fixture":1,"ran":6,"ran_honours":2,"unverified":4},"by_repo_kind":{"official":{"samples":17,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":17,"samples":[{"code_sha256_prefix":"0f786c407fb1ee4c","entry":"swish","repo":"neurocomplab-psu/spikingbert","repo_kind":"official","path":"transformer/modeling.py","file_url":"https://github.com/neurocomplab-psu/spikingbert/blob/HEAD/transformer/modeling.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0f786c407fb1ee4c"}},{"code_sha256_prefix":"cac113ca87b9d9f3","entry":"acc_and_f1","repo":"neurocomplab-psu/spikingbert","repo_kind":"official","path":"spiking_bert_task_distill.py","file_url":"https://github.com/neurocomplab-psu/spikingbert/blob/HEAD/spiking_bert_task_distill.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cac113ca87b9d9f3"}},{"code_sha256_prefix":"3dce841898f3d634","entry":"compute_spectral_norm","repo":"neurocomplab-psu/spikingbert","repo_kind":"official","path":"implicit_bert/modules/optimizations.py","file_url":"https://github.com/neurocomplab-psu/spikingbert/blob/HEAD/implicit_bert/modules/optimizations.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3dce841898f3d634"}},{"code_sha256_prefix":"f2f5b170e5141470","entry":"convert_example_to_features","repo":"neurocomplab-psu/spikingbert","repo_kind":"official","path":"spiking_bert_general_distill.py","file_url":"https://github.com/neurocomplab-psu/spikingbert/blob/HEAD/spiking_bert_general_distill.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f2f5b170e5141470"}},{"code_sha256_prefix":"ca4a60dd9a4dee68","entry":"convert_examples_to_features","repo":"neurocomplab-psu/spikingbert","repo_kind":"official","path":"spiking_bert_task_distill.py","file_url":"https://github.com/neurocomplab-psu/spikingbert/blob/HEAD/spiking_bert_task_distill.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ca4a60dd9a4dee68"}},{"code_sha256_prefix":"6376a70f6dfc27d6","entry":"create_instances_from_document","repo":"neurocomplab-psu/spikingbert","repo_kind":"official","path":"preprocess_training_data.py","file_url":"https://github.com/neurocomplab-psu/spikingbert/blob/HEAD/preprocess_training_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6376a70f6dfc27d6"}},{"code_sha256_prefix":"c1f215df4049f030","entry":"create_masked_lm_predictions","repo":"neurocomplab-psu/spikingbert","repo_kind":"official","path":"preprocess_training_data.py","file_url":"https://github.com/neurocomplab-psu/spikingbert/blob/HEAD/preprocess_training_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c1f215df4049f030"}},{"code_sha256_prefix":"fdc64f4c72036ae4","entry":"gelu","repo":"neurocomplab-psu/spikingbert","repo_kind":"official","path":"transformer/modeling.py","file_url":"https://github.com/neurocomplab-psu/spikingbert/blob/HEAD/transformer/modeling.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fdc64f4c72036ae4"}},{"code_sha256_prefix":"3caf26b311c98658","entry":"get_tasks","repo":"neurocomplab-psu/spikingbert","repo_kind":"official","path":"download_glue_data.py","file_url":"https://github.com/neurocomplab-psu/spikingbert/blob/HEAD/download_glue_data.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3caf26b311c98658"}},{"code_sha256_prefix":"cf5710371bcfb0ab","entry":"line_search","repo":"neurocomplab-psu/spikingbert","repo_kind":"official","path":"implicit_bert/modules/broyden.py","file_url":"https://github.com/neurocomplab-psu/spikingbert/blob/HEAD/implicit_bert/modules/broyden.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cf5710371bcfb0ab"}},{"code_sha256_prefix":"968a8f446ad3c5b1","entry":"rmatvec","repo":"neurocomplab-psu/spikingbert","repo_kind":"official","path":"implicit_bert/modules/broyden.py","file_url":"https://github.com/neurocomplab-psu/spikingbert/blob/HEAD/implicit_bert/modules/broyden.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"968a8f446ad3c5b1"}},{"code_sha256_prefix":"b2b5022f654ee4d3","entry":"scalar_search_armijo","repo":"neurocomplab-psu/spikingbert","repo_kind":"official","path":"implicit_bert/modules/broyden.py","file_url":"https://github.com/neurocomplab-psu/spikingbert/blob/HEAD/implicit_bert/modules/broyden.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b2b5022f654ee4d3"}},{"code_sha256_prefix":"f0fda38cdd52aefb","entry":"weight_norm","repo":"neurocomplab-psu/spikingbert","repo_kind":"official","path":"implicit_bert/modules/optimizations.py","file_url":"https://github.com/neurocomplab-psu/spikingbert/blob/HEAD/implicit_bert/modules/optimizations.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f0fda38cdd52aefb"}},{"code_sha256_prefix":"a8d00bf9b456b3f6","entry":"create_training_file","repo":"neurocomplab-psu/spikingbert","repo_kind":"official","path":"preprocess_training_data.py","file_url":"https://github.com/neurocomplab-psu/spikingbert/blob/HEAD/preprocess_training_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a8d00bf9b456b3f6"}},{"code_sha256_prefix":"0453ed29d1aecc4c","entry":"load_tf_weights_in_bert","repo":"neurocomplab-psu/spikingbert","repo_kind":"official","path":"transformer/modeling.py","file_url":"https://github.com/neurocomplab-psu/spikingbert/blob/HEAD/transformer/modeling.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0453ed29d1aecc4c"}},{"code_sha256_prefix":"3c241ecfe3749a6d","entry":"simple_accuracy","repo":"neurocomplab-psu/spikingbert","repo_kind":"official","path":"spiking_bert_task_distill.py","file_url":"https://github.com/neurocomplab-psu/spikingbert/blob/HEAD/spiking_bert_task_distill.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3c241ecfe3749a6d"}},{"code_sha256_prefix":"e2f5c84554d935f4","entry":"weight_spectral_norm","repo":"neurocomplab-psu/spikingbert","repo_kind":"official","path":"implicit_bert/modules/optimizations.py","file_url":"https://github.com/neurocomplab-psu/spikingbert/blob/HEAD/implicit_bert/modules/optimizations.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e2f5c84554d935f4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}