{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lighthubert-lightweight-and-configurable","title":"LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERT","arxiv_id":"2203.15610","date":"2022-03-29","proceeding":null,"authors":["Rui Wang","Qibing Bai","Junyi Ao","Long Zhou","Zhixiang Xiong","Zhihua Wei","Yu Zhang","Tom Ko","Haizhou Li"],"abstract":"Self-supervised speech representation learning has shown promising results in various speech processing tasks. However, the pre-trained models, e.g., HuBERT, are storage-intensive Transformers, limiting their scope of applications under low-resource settings. To this end, we propose LightHuBERT, a once-for-all Transformer compression framework, to find the desired architectures automatically by pruning structured parameters. More precisely, we create a Transformer-based supernet that is nested with thousands of weight-sharing subnets and design a two-stage distillation strategy to leverage the contextualized latent representations from HuBERT. Experiments on automatic speech recognition (ASR) and the SUPERB benchmark show the proposed LightHuBERT enables over $10^9$ architectures concerning the embedding dimension, attention dimension, head number, feed-forward network ratio, and network depth. LightHuBERT outperforms the original HuBERT on ASR and five SUPERB tasks with the HuBERT size, achieves comparable performance to the teacher model in most tasks with a reduction of 29% parameters, and obtains a $3.5\\times$ compression ratio in three SUPERB tasks, e.g., automatic speaker verification, keyword spotting, and intent classification, with a slight accuracy loss. The code and pre-trained models are available at https://github.com/mechanicalsea/lighthubert.","url_abs":"https://arxiv.org/abs/2203.15610v2","url_pdf":"https://arxiv.org/pdf/2203.15610v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"lighthubert-lightweight-and-configurable","repo_url":"https://github.com/mechanicalsea/lighthubert","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"all","task_name":"All"},{"task_slug":"automatic-speech-recognition-2","task_name":"Automatic Speech Recognition"},{"task_slug":"automatic-speech-recognition","task_name":"Automatic Speech Recognition (ASR)"},{"task_slug":"intent-classification","task_name":"Intent Classification"},{"task_slug":"keyword-spotting","task_name":"Keyword Spotting"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"speaker-verification","task_name":"Speaker Verification"},{"task_slug":"speech-recognition","task_name":"Speech Recognition"},{"task_slug":"speech-representation-learning","task_name":"Speech Representation Learning"},{"task_slug":"intent-classification-1","task_name":"intent-classification"},{"task_slug":"speech-recognition-1","task_name":"speech-recognition"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"pruning","method_name":"Pruning"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2203.15610","atlas_url":"https://app.syntology.ai/?focus=2203.15610","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2203.15610"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mechanicalsea/lighthubert","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":8,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"28bed6ccb2f1b89a","entry":"index_put","repo":"mechanicalsea/lighthubert","repo_kind":"official","path":"lighthubert/functional/fairseq_utils.py","file_url":"https://github.com/mechanicalsea/lighthubert/blob/HEAD/lighthubert/functional/fairseq_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"28bed6ccb2f1b89a"}},{"code_sha256_prefix":"29c1031dcedd09de","entry":"is_xla_tensor","repo":"mechanicalsea/lighthubert","repo_kind":"official","path":"lighthubert/functional/fairseq_utils.py","file_url":"https://github.com/mechanicalsea/lighthubert/blob/HEAD/lighthubert/functional/fairseq_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"29c1031dcedd09de"}},{"code_sha256_prefix":"d48916e1aac4a4de","entry":"mask_padding","repo":"mechanicalsea/lighthubert","repo_kind":"official","path":"lighthubert/functional/sliding_attn.py","file_url":"https://github.com/mechanicalsea/lighthubert/blob/HEAD/lighthubert/functional/sliding_attn.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d48916e1aac4a4de"}},{"code_sha256_prefix":"3bbfd682fbeb9a6d","entry":"pad_as_attn_swz","repo":"mechanicalsea/lighthubert","repo_kind":"official","path":"lighthubert/functional/sliding_attn.py","file_url":"https://github.com/mechanicalsea/lighthubert/blob/HEAD/lighthubert/functional/sliding_attn.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3bbfd682fbeb9a6d"}},{"code_sha256_prefix":"50d074f5c3f3407d","entry":"quant_noise","repo":"mechanicalsea/lighthubert","repo_kind":"official","path":"lighthubert/modules/fairseq_modules.py","file_url":"https://github.com/mechanicalsea/lighthubert/blob/HEAD/lighthubert/modules/fairseq_modules.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"50d074f5c3f3407d"}},{"code_sha256_prefix":"6eeb1f787769a05e","entry":"compute_mask_indices","repo":"mechanicalsea/lighthubert","repo_kind":"official","path":"lighthubert/lighthubert.py","file_url":"https://github.com/mechanicalsea/lighthubert/blob/HEAD/lighthubert/lighthubert.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6eeb1f787769a05e"}},{"code_sha256_prefix":"fb33b4608f21a92a","entry":"merge_padding_attm_mask","repo":"mechanicalsea/lighthubert","repo_kind":"official","path":"lighthubert/functional/sliding_attn.py","file_url":"https://github.com/mechanicalsea/lighthubert/blob/HEAD/lighthubert/functional/sliding_attn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fb33b4608f21a92a"}},{"code_sha256_prefix":"b57d92d8a18d12a3","entry":"pad_to_multiple","repo":"mechanicalsea/lighthubert","repo_kind":"official","path":"lighthubert/functional/fairseq_utils.py","file_url":"https://github.com/mechanicalsea/lighthubert/blob/HEAD/lighthubert/functional/fairseq_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b57d92d8a18d12a3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}