{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/spikformer-v2-join-the-high-accuracy-club-on","title":"Spikformer V2: Join the High Accuracy Club on ImageNet with an SNN Ticket","arxiv_id":"2401.02020","date":"2024-01-04","proceeding":null,"authors":["Zhaokun Zhou","Kaiwei Che","Wei Fang","Keyu Tian","Yuesheng Zhu","Shuicheng Yan","Yonghong Tian","Li Yuan"],"abstract":"Spiking Neural Networks (SNNs), known for their biologically plausible architecture, face the challenge of limited performance. The self-attention mechanism, which is the cornerstone of the high-performance Transformer and also a biologically inspired structure, is absent in existing SNNs. To this end, we explore the potential of leveraging both self-attention capability and biological properties of SNNs, and propose a novel Spiking Self-Attention (SSA) and Spiking Transformer (Spikformer). The SSA mechanism eliminates the need for softmax and captures the sparse visual feature employing spike-based Query, Key, and Value. This sparse computation without multiplication makes SSA efficient and energy-saving. Further, we develop a Spiking Convolutional Stem (SCS) with supplementary convolutional layers to enhance the architecture of Spikformer. The Spikformer enhanced with the SCS is referred to as Spikformer V2. To train larger and deeper Spikformer V2, we introduce a pioneering exploration of Self-Supervised Learning (SSL) within the SNN. Specifically, we pre-train Spikformer V2 with masking and reconstruction style inspired by the mainstream self-supervised Transformer, and then finetune the Spikformer V2 on the image classification on ImageNet. Extensive experiments show that Spikformer V2 outperforms other previous surrogate training and ANN2SNN methods. An 8-layer Spikformer V2 achieves an accuracy of 80.38% using 4 time steps, and after SSL, a 172M 16-layer Spikformer V2 reaches an accuracy of 81.10% with just 1 time step. To the best of our knowledge, this is the first time that the SNN achieves 80+% accuracy on ImageNet. The code will be available at Spikformer V2.","url_abs":"https://arxiv.org/abs/2401.02020v1","url_pdf":"https://arxiv.org/pdf/2401.02020v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"spikformer-v2-join-the-high-accuracy-club-on","repo_url":"https://github.com/zk-zhou/spikformer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"spikformer-v2-join-the-high-accuracy-club-on","repo_url":"https://github.com/Fancyssc/Spiking-Transformers","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"spikformer-v2-join-the-high-accuracy-club-on","repo_url":"https://github.com/qiwang233/dailydvs-200","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"snn","method_name":"SNN"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2401.02020","atlas_url":"https://app.syntology.ai/?focus=2401.02020","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.02020"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/qiwang233/dailydvs-200","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Fancyssc/Spiking-Transformers","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zk-zhou/spikformer","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":1,"ran":5,"unverified":2},"by_repo_kind":{"official":{"samples":8,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4134b8420b4c48b5","entry":"accuracy","repo":"zk-zhou/spikformer","repo_kind":"official","path":"cifar10dvs/utils.py","file_url":"https://github.com/zk-zhou/spikformer/blob/HEAD/cifar10dvs/utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4134b8420b4c48b5"}},{"code_sha256_prefix":"8eee4204b6767c33","entry":"fast_collate","repo":"zk-zhou/spikformer","repo_kind":"official","path":"cifar10/loader.py","file_url":"https://github.com/zk-zhou/spikformer/blob/HEAD/cifar10/loader.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8eee4204b6767c33"}},{"code_sha256_prefix":"2d7b973a4b160d73","entry":"shear_y","repo":"zk-zhou/spikformer","repo_kind":"official","path":"cifar10/aa_snn.py","file_url":"https://github.com/zk-zhou/spikformer/blob/HEAD/cifar10/aa_snn.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2d7b973a4b160d73"}},{"code_sha256_prefix":"d5a693c429aa8571","entry":"transforms_imagenet_eval","repo":"zk-zhou/spikformer","repo_kind":"official","path":"cifar10/transforms_factory.py","file_url":"https://github.com/zk-zhou/spikformer/blob/HEAD/cifar10/transforms_factory.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d5a693c429aa8571"}},{"code_sha256_prefix":"c82a60e3c9859e18","entry":"transforms_noaug_train","repo":"zk-zhou/spikformer","repo_kind":"official","path":"cifar10/transforms_factory.py","file_url":"https://github.com/zk-zhou/spikformer/blob/HEAD/cifar10/transforms_factory.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c82a60e3c9859e18"}},{"code_sha256_prefix":"68edbf1d7855d1bc","entry":"unpack_len1_tuple","repo":"zk-zhou/spikformer","repo_kind":"official","path":"cifar10dvs/monitor.py","file_url":"https://github.com/zk-zhou/spikformer/blob/HEAD/cifar10dvs/monitor.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"68edbf1d7855d1bc"}},{"code_sha256_prefix":"b31e84df5541213a","entry":"shear_x","repo":"zk-zhou/spikformer","repo_kind":"official","path":"cifar10/aa_snn.py","file_url":"https://github.com/zk-zhou/spikformer/blob/HEAD/cifar10/aa_snn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b31e84df5541213a"}},{"code_sha256_prefix":"61f2a3e08a11e1f5","entry":"translate_x_rel","repo":"zk-zhou/spikformer","repo_kind":"official","path":"cifar10/aa_snn.py","file_url":"https://github.com/zk-zhou/spikformer/blob/HEAD/cifar10/aa_snn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"61f2a3e08a11e1f5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}