Papers › WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing

WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing

26 Oct 2021arXiv:2110.13900archive 2025-07-28

Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, Furu Wei

Self-supervised learning (SSL) achieves great success in speech recognition, while limited exploration has been attempted for other speech processing tasks. As speech signal contains multi-faceted information including speaker identity, paralinguistics, spoken content, etc., learning universal representations for all speech tasks is challenging. To tackle the problem, we propose a new pre-trained model, WavLM, to solve full-stack downstream speech tasks. WavLM jointly learns masked speech prediction and denoising in pre-training. By this means, WavLM does not only keep the speech content modeling capability by the masked speech prediction, but also improves the potential to non-ASR tasks by the speech denoising. In addition, WavLM employs gated relative position bias for the Transformer structure to better capture the sequence ordering of input speech. We also scale up the training dataset from 60k hours to 94k hours. WavLM Large achieves state-of-the-art performance on the SUPERB benchmark, and brings significant improvements for various speech processing tasks on their representative benchmarks. The code and pre-trained models are available at https://aka.ms/wavlm.

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

microsoft/unilm officialmentioned in papermentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training (arXiv:2002.12804; that paper's own run record: 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified), BEiT: BERT Pre-Training of Image Transformers (arXiv:2106.08254; that paper's own run record: 6 ran (of which 2 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 2 where Syntology's instrument failed) · 5 unverified), DeepNet: Scaling Transformers to 1,000 Layers (arXiv:2203.00555; that paper's own run record: 4 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 3 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified) and 1 more (every one in this page's JSON twin); nothing checks that they implement this paper's method. report
cywang97/unispeech mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data (arXiv:2101.07597; that paper's own run record: 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
kyutai-labs/moshi mentioned on GitHubpytorchApache-2.0Syntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Moshi: a speech-text foundation model for real-time dialogue (arXiv:2410.00037; that paper's own run record: 7 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 4 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
microsoft/unispeech mentioned on GitHubpytorchNOASSERTIONSyntology: not harvested report
nyrahealth/crisperwhisper mentioned on GitHubpytorchNOASSERTIONSyntology: not harvested report
olawod/freevc mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech (arXiv:2106.06103; that paper's own run record: 47 ran (of which 12 constructed an object rather than computing a result; 34 with no instrument failure: 2 honoured, 0 violated, 32 with no contract checked; 13 where Syntology's instrument failed) · 30 unverified); nothing checks that they implement this paper's method. report
sanyuan-chen/unispeech mentioned on GitHubpytorchNOASSERTIONSyntology: not harvested report
MS-P3/code7 mindsporeSyntology: not harvested report
pwc-1/Paper-9 mindsporeSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to RandAugment: Practical automated data augmentation with a reduced search space (arXiv:1909.13719; that paper's own run record: 58 ran (of which 1 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 1 violated, 6 with no contract checked; 51 where Syntology's instrument failed) · 7 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

DenoisingSelf-Supervised LearningSpeech DenoisingSpeech Recognitionspeech-recognition

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Speech Recognition CALLHOME En WavLM Large & EEND-vector clustering Word Error Rate (WER) 10.35 #1 of 1 Archive leaderboard report
Speech Recognition LibriSpeech test-clean WavLM Large Word Error Rate (WER) 1.8 #14 of 64 Archive leaderboard report
Speech Recognition LibriSpeech test-other WavLM Large Word Error Rate (WER) 3.2 #8 of 53 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Absolute Position EncodingsAdamAttentionBPEDense ConnectionsDropoutLabel SmoothingLayer NormalizationLinear LayerMulti-Head AttentionPosition-Wise Feed-Forward LayerResidual ConnectionSoftmaxTransformer

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections