{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pixelated-butterfly-simple-and-efficient-1","title":"Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models","arxiv_id":"2112.00029","date":"2021-11-30","proceeding":"ICLR 2022 4","authors":["Tri Dao","Beidi Chen","Kaizhao Liang","Jiaming Yang","Zhao Song","Atri Rudra","Christopher Ré"],"abstract":"Overparameterized neural networks generalize well but are expensive to train. Ideally, one would like to reduce their computational cost while retaining their generalization benefits. Sparse model training is a simple and promising approach to achieve this, but there remain challenges as existing methods struggle with accuracy loss, slow training runtime, or difficulty in sparsifying all model components. The core problem is that searching for a sparsity mask over a discrete set of sparse matrices is difficult and expensive. To address this, our main insight is to optimize over a continuous superset of sparse matrices with a fixed structure known as products of butterfly matrices. As butterfly matrices are not hardware efficient, we propose simple variants of butterfly (block and flat) to take advantage of modern hardware. Our method (Pixelated Butterfly) uses a simple fixed sparsity pattern based on flat block butterfly and low-rank matrices to sparsify most network layers (e.g., attention, MLP). We empirically validate that Pixelated Butterfly is 3x faster than butterfly and speeds up training to achieve favorable accuracy--efficiency tradeoffs. On the ImageNet classification and WikiText-103 language modeling tasks, our sparse models train up to 2.5x faster than the dense MLP-Mixer, Vision Transformer, and GPT-2 medium with no drop in accuracy.","url_abs":"https://arxiv.org/abs/2112.00029v2","url_pdf":"https://arxiv.org/pdf/2112.00029v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pixelated-butterfly-simple-and-efficient-1","repo_url":"https://github.com/HazyResearch/pixelfly","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"discriminative-fine-tuning","method_name":"Discriminative Fine-Tuning"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-2","method_name":"GPT-2"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"mlp-mixer","method_name":"MLP-Mixer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2112.00029","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2112.00029"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/huggingface/pytorch_block_sparse","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/HazyResearch/pixelfly","reach":null}],"summary":{"ran":4,"unverified":2},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1},"found_in_text":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"90b5ab602b92ab5f","entry":"BlockSparseLinearFunction","repo":"huggingface/pytorch_block_sparse","repo_kind":"found_in_text","path":"pytorch_block_sparse/block_sparse_linear.py","file_url":"https://github.com/huggingface/pytorch_block_sparse/blob/HEAD/pytorch_block_sparse/block_sparse_linear.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"90b5ab602b92ab5f"}},{"code_sha256_prefix":"c4d9adf80ea69faf","entry":"BlockSparseMatrix","repo":"huggingface/pytorch_block_sparse","repo_kind":"found_in_text","path":"pytorch_block_sparse/block_sparse_linear.py","file_url":"https://github.com/huggingface/pytorch_block_sparse/blob/HEAD/pytorch_block_sparse/block_sparse_linear.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c4d9adf80ea69faf"}},{"code_sha256_prefix":"2c4fe31e1554158a","entry":"BlockSparseMatrixBase","repo":"huggingface/pytorch_block_sparse","repo_kind":"found_in_text","path":"pytorch_block_sparse/block_sparse_linear.py","file_url":"https://github.com/huggingface/pytorch_block_sparse/blob/HEAD/pytorch_block_sparse/block_sparse_linear.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2c4fe31e1554158a"}},{"code_sha256_prefix":"ed2fbbe21934c400","entry":"BlockSparseMatrixEmulator","repo":"huggingface/pytorch_block_sparse","repo_kind":"found_in_text","path":"pytorch_block_sparse/block_sparse_linear.py","file_url":"https://github.com/huggingface/pytorch_block_sparse/blob/HEAD/pytorch_block_sparse/block_sparse_linear.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ed2fbbe21934c400"}},{"code_sha256_prefix":"bcb76e709020c908","entry":"BlockSparseLinear","repo":"huggingface/pytorch_block_sparse","repo_kind":"found_in_text","path":"pytorch_block_sparse/block_sparse_linear.py","file_url":"https://github.com/huggingface/pytorch_block_sparse/blob/HEAD/pytorch_block_sparse/block_sparse_linear.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"bcb76e709020c908"}},{"code_sha256_prefix":"1b4982391e4c1bf4","entry":"BlockdiagButterflyMultiply","repo":"HazyResearch/pixelfly","repo_kind":"official","path":"src/models/layers/blockdiag_butterfly_multiply.py","file_url":"https://github.com/HazyResearch/pixelfly/blob/HEAD/src/models/layers/blockdiag_butterfly_multiply.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1b4982391e4c1bf4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}