Papers › ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and...
ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement
Wei-Ning Hsu, Tal Remez, Bowen Shi, Jacob Donley, Yossi Adi
Prior works on improving speech quality with visual input typically study each type of auditory distortion separately (e.g., separation, inpainting, video-to-speech) and present tailored algorithms. This paper proposes to unify these subjects and study Generalized Speech Enhancement, where the goal is not to reconstruct the exact reference clean signal, but to focus on improving certain aspects of speech. In particular, this paper concerns intelligibility, quality, and video synchronization. We cast the problem as audio-visual speech resynthesis, which is composed of two steps: pseudo audio-visual speech recognition (P-AVSR) and pseudo text-to-speech synthesis (P-TTS). P-AVSR and P-TTS are connected by discrete units derived from a self-supervised speech model. Moreover, we utilize self-supervised audio-visual speech model to initialize P-AVSR. The proposed model is coined ReVISE. ReVISE is the first high-quality model for in-the-wild video-to-speech synthesis and achieves superior performance on all LRS3 audio-visual enhancement tasks with a single model. To demonstrates its applicability in the real world, ReVISE is also evaluated on EasyCom, an audio-visual benchmark collected under challenging acoustic conditions with only 1.6 hours of training data. Similarly, ReVISE greatly suppresses noise and improves quality. Project page: https://wnhsu.github.io/ReVISE.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
No code repository is listed for this paper in the archive or in Syntology's graph.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Speech Enhancement | EasyCom | ReVISE (ch2) | Audio Quality MOS | 4.19 | #2 of 6 | Archive leaderboard | report |
| Speech Enhancement | EasyCom | ReVISE (bf) | Audio Quality MOS | 4.11 | #3 of 6 | Archive leaderboard | report |
| Speech Enhancement | EasyCom | Demucs (ch2) | Audio Quality MOS | 2.95 | #4 of 6 | Archive leaderboard | report |
| Speech Enhancement | EasyCom | Demucs (bf) | Audio Quality MOS | 2.39 | #5 of 6 | Archive leaderboard | report |
| Speech Recognition | EasyCom | ReVISE (bf) | WER (%) | 52.1 | #1 of 5 | Archive leaderboard | report |
| Speech Recognition | EasyCom | ReVISE (ch2) | WER (%) | 55.0 | #2 of 5 | Archive leaderboard | report |
| Speech Recognition | EasyCom | Demucs (bf) | WER (%) | 69.8 | #4 of 5 | Archive leaderboard | report |
| Speech Recognition | EasyCom | Demucs (ch2) | WER (%) | 86.8 | #5 of 5 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections