{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/squeezebert-what-can-computer-vision-teach","title":"SqueezeBERT: What can computer vision teach NLP about efficient neural networks?","arxiv_id":"2006.11316","date":"2020-06-19","proceeding":"EMNLP (sustainlp) 2020 11","authors":["Forrest N. Iandola","Albert E. Shaw","Ravi Krishna","Kurt W. Keutzer"],"abstract":"Humans read and write hundreds of billions of messages every day. Further, due to the availability of large datasets, large computing systems, and better neural network models, natural language processing (NLP) technology has made significant strides in understanding, proofreading, and organizing these messages. Thus, there is a significant opportunity to deploy NLP in myriad applications to help web users, social networks, and businesses. In particular, we consider smartphones and other mobile devices as crucial platforms for deploying NLP models at scale. However, today's highly-accurate NLP neural network models such as BERT and RoBERTa are extremely computationally expensive, with BERT-base taking 1.7 seconds to classify a text snippet on a Pixel 3 smartphone. In this work, we observe that methods such as grouped convolutions have yielded significant speedups for computer vision networks, but many of these techniques have not been adopted by NLP neural network designers. We demonstrate how to replace several operations in self-attention layers with grouped convolutions, and we use this technique in a novel network architecture called SqueezeBERT, which runs 4.3x faster than BERT-base on the Pixel 3 while achieving competitive accuracy on the GLUE test set. The SqueezeBERT code will be released.","url_abs":"https://arxiv.org/abs/2006.11316v1","url_pdf":"https://arxiv.org/pdf/2006.11316v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"squeezebert-what-can-computer-vision-teach","repo_url":"https://github.com/huggingface/transformers/blob/main/src/transformers/models/squeezebert/modeling_squeezebert.py","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"jax","reach":null},{"paper_slug":"squeezebert-what-can-computer-vision-teach","repo_url":"https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_squeezebert.py","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"squeezebert-what-can-computer-vision-teach","repo_url":"https://github.com/mindspore-courses/d2l-mindspore/tree/master/chapter_14_natural_language_processing_pretraining","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"squeezebert-what-can-computer-vision-teach","repo_url":"https://github.com/pwc-1/Paper-10/tree/main/squeezebert","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"squeezebert-what-can-computer-vision-teach","repo_url":"https://github.com/renmada/squeezebert-paddle","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"paddle","reach":{"status":"ok"}},{"paper_slug":"squeezebert-what-can-computer-vision-teach","repo_url":"https://github.com/yangyucheng000/University/tree/main/model-3/squeezebert","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"linguistic-acceptability","task_name":"Linguistic Acceptability"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"semantic-textual-similarity","task_name":"Semantic Textual Similarity"},{"task_slug":"sentiment-analysis","task_name":"Sentiment Analysis"},{"task_slug":"text-classification","task_name":"Text Classification"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"}],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"grouped-convolution","method_name":"Grouped Convolution"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"squeezebert","method_name":"SqueezeBERT"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[{"slug":"squeezebert","name":"SqueezeBERT","full_name":"SqueezeBERT"}],"results":[{"leaderboard":"/sota/linguistic-acceptability-on-cola","task":"Linguistic Acceptability","dataset":"CoLA","model":"SqueezeBERT","rank_in_archive_order":39,"of":43,"metrics":{"Accuracy":"46.5%"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-multinli","task":"Natural Language Inference","dataset":"MultiNLI","model":"SqueezeBERT","rank_in_archive_order":42,"of":67,"metrics":{"Matched":"82.0","Mismatched":"81.1"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-qnli","task":"Natural Language Inference","dataset":"QNLI","model":"SqueezeBERT","rank_in_archive_order":37,"of":43,"metrics":{"Accuracy":"90.1%"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-rte","task":"Natural Language Inference","dataset":"RTE","model":"SqueezeBERT","rank_in_archive_order":48,"of":90,"metrics":{"Accuracy":"73.2%"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-wnli","task":"Natural Language Inference","dataset":"WNLI","model":"SqueezeBERT","rank_in_archive_order":20,"of":23,"metrics":{"Accuracy":"65.1"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-quora-question-pairs","task":"Question Answering","dataset":"Quora Question Pairs","model":"SqueezeBERT","rank_in_archive_order":18,"of":19,"metrics":{"Accuracy":"80.3%"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-textual-similarity-on-mrpc","task":"Semantic Textual Similarity","dataset":"MRPC","model":"SqueezeBERT","rank_in_archive_order":24,"of":45,"metrics":{"Accuracy":"87.8%"},"uses_additional_data":false},{"leaderboard":"/sota/sentiment-analysis-on-sst-2-binary","task":"Sentiment Analysis","dataset":"SST-2 Binary classification","model":"SqueezeBERT","rank_in_archive_order":52,"of":87,"metrics":{"Accuracy":"91.4"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2006.11316","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.11316"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/renmada/squeezebert-paddle","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/huggingface/transformers/blob/main/src/transformers/models/squeezebert/modeling_squeezebert.py","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yangyucheng000/University/tree/main/model-3/squeezebert","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mindspore-courses/d2l-mindspore/tree/master/chapter_14_natural_language_processing_pretraining","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_squeezebert.py","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pwc-1/Paper-10/tree/main/squeezebert","reach":null}],"summary":{"unverified":1},"by_repo_kind":{"listed":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"50dbc6491aad4f72","entry":"squared_loss","repo":"mindspore-courses/d2l-mindspore","repo_kind":"listed","path":"d2l/mindspore.py","file_url":"https://github.com/mindspore-courses/d2l-mindspore/blob/HEAD/d2l/mindspore.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"50dbc6491aad4f72"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}