Browse State-of-the-Art › Audio-Visual Synchronization

Audio-Visual Synchronization

11 papers with code · 0 benchmarks · 4 datasets archive 2025-07-28

AudioComputer Vision

Benchmarks archive 2025-07-28

No benchmark for this task in the archive.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

4 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

No subtask under this task in the archive's task tree.

Papers archive 2025-07-28

archive papers tagged: 32 · with a code link: 11 · where Syntology ran a sample: 3 (3 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument) Syntology

Show: all tagged papersonly where code ran (3 of 32 tagged: 3 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument)

Page 1 of 1: papers 1 to 32 of 32, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.

Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.

  • Our objective is audio-visual synchronization with a focus on 'in-the-wild' videos, such as those on YouTube, where synchronization cues can be sparse.
  • 27 Oct 2022 2 repositories listed
    This paper proposed an MTDVocaLiST model, which is trained by our proposed multimodal Transformer distillation (MTD) loss.
  • 13 Oct 2022 2 repositories listed
    This contrasts with the case of synchronising videos of talking heads, where audio-visual correspondence is dense in both time and space.
  • 6 May 2025 1 repository listed
    They also enable strong performance in Visual Speech Recognition (VSR) with a WER of 20.
  • We propose to synthesize high-quality and synchronized audio, given video and optional text conditions, using a novel multimodal joint training framework MMAudio.
  • Real-time video dubbing that preserves identity consistency while achieving accurate lip synchronization remains a critical challenge.
  • 30 Apr 2024 1 repository listed
    With the rising prevalence of deepfakes, there is a growing interest in developing generalizable detection methods for various types of deepfakes.
  • 10 Apr 2024 1 repository listed
    Recent advancements in audio-visual generative modeling have been propelled by progress in deep learning and the availability of data-rich benchmarks.
  • 22 May 2023 1 repository listed
    To benefit from both facial cue and reference speech, we propose the Target Speaker TalkNet (TS-TalkNet), which leverages a pre-enrolled speaker embedding to complement the audio-visual synchronization cue in detecting…
  • 5 Apr 2022 1 repository listed
    Finally, we use the frozen visual features learned by our lip synchronisation model in the singing voice separation task to outperform a baseline audio-visual model which was trained end-to-end.
  • 5 Oct 2021 1 repository listed
    Modifying the pitch and timing of an audio signal are fundamental audio editing operations with applications in speech manipulation, audio-visual synchronization, and singing voice editing and synthesis.
  • Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generation
    24 Jun 2025 0 repositories listed
    We propose Kling-Foley, a large-scale multimodal Video-to-Audio generation model that synthesizes high-quality audio synchronized with video content.
  • Audio-Sync Video Generation with Multi-Stream Temporal Control
    9 Jun 2025 0 repositories listed
    Audio is inherently temporal and closely synchronized with the visual world, making it a naturally aligned and expressive control signal for controllable video generation (e.g., movies).
  • OmniResponse: Online Multimodal Conversational Response Generation in Dyadic Interactions
    27 May 2025 0 repositories listed
    In this paper, we introduce Online Multimodal Conversational Response Generation (OMCRG), a novel task that aims to online generate synchronized verbal and non-verbal listener feedback, conditioned on the speaker's…
  • DeepAudio-V1:Towards Multi-Modal Multi-Stage End-to-End Video to Speech and Audio Generation
    28 Mar 2025 0 repositories listed
    Currently, high-quality, synchronized audio is synthesized using various multi-modal joint learning frameworks, leveraging video and optional text inputs.
  • UniSync: A Unified Framework for Audio-Visual Synchronization
    20 Mar 2025 0 repositories listed
    Precise audio-visual synchronization in speech videos is crucial for content quality and viewer comprehension.
  • FREAK: Frequency-modulated High-fidelity and Real-time Audio-driven Talking Portrait Synthesis
    6 Mar 2025 0 repositories listed
    Achieving high-fidelity lip-speech synchronization in audio-driven talking portrait synthesis remains challenging.
  • Draw an Audio: Leveraging Multi-Instruction for Video-to-Audio Synthesis
    10 Sep 2024 0 repositories listed
    Foley is a term commonly used in filmmaking, referring to the addition of daily sound effects to silent films or videos to enhance the auditory experience.
  • A Comprehensive Review and Taxonomy of Audio-Visual Synchronization Techniques for Realistic Speech Animation
    24 Jul 2024 0 repositories listed
    In many applications, synchronizing audio with visuals is crucial, such as in creating graphic animations for films or games, translating movie audio into different languages, and developing metaverse applications.
  • RealTalk: Real-time and Realistic Audio-driven Face Generation with 3D Facial Prior-guided Identity Alignment Network
    26 Jun 2024 0 repositories listed
    Person-generic audio-driven face generation is a challenging task in computer vision.
  • CoAVT: A Cognition-Inspired Unified Audio-Visual-Text Pre-Training Model for Multimodal Processing
    22 Jan 2024 0 repositories listed
    There has been a long-standing quest for a unified audio-visual-text model to enable various multimodal understanding tasks, which mimics the listening, seeing and reading process of human beings.
  • Comparative Analysis of Deep-Fake Algorithms
    6 Sep 2023 0 repositories listed
    Due to the widespread use of smartphones with high-quality digital cameras and easy access to a wide range of software apps for recording, editing, and sharing videos and images, as well as the deep learning AI…
  • Audio-driven Talking Face Generation with Stabilized Synchronization Loss
    18 Jul 2023 0 repositories listed
    Talking face generation aims to create realistic videos with accurate lip synchronization and high visual quality, using given audio and reference video while preserving identity and visual characteristics.
  • On the Audio-visual Synchronization for Lip-to-Speech Synthesis
    1 Mar 2023 0 repositories listed
    Most lip-to-speech (LTS) synthesis models are trained and evaluated under the assumption that the audio-video pairs in the dataset are perfectly synchronized.
  • SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memory
    2 Nov 2022 0 repositories listed
    The challenge of talking face generation from speech lies in aligning two different modal information, audio and video, such that the mouth region corresponds to input audio.
  • Rethinking Audio-visual Synchronization for Active Speaker Detection
    21 Jun 2022 0 repositories listed
    Active speaker detection (ASD) systems are important modules for analyzing multi-talker conversations.
  • Looking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separation
    25 Mar 2021 0 repositories listed
    In this paper, we address the problem of separating individual speech signals from videos using audio-visual neural processing.
  • Look, Listen, and Attend: Co-Attention Network for Self-Supervised Audio-Visual Representation Learning
    13 Aug 2020 0 repositories listed
    When watching videos, the occurrence of a visual event is often accompanied by an audio event, e.g., the voice of lip motion, the music of playing instruments.
  • Identity-Preserving Realistic Talking Face Generation
    25 May 2020 0 repositories listed
    Speech-driven facial animation is useful for a variety of applications such as telepresence, chatbots, etc.
  • Realistic Speech-Driven Facial Animation with GANs
    14 Jun 2019 0 repositories listed
    Speech-driven facial animation is the process that automatically synthesizes talking characters based on speech signals.
  • On Attention Modules for Audio-Visual Synchronization
    14 Dec 2018 0 repositories listed
  • FaceDirector: Continuous Control of Facial Performance in Video
    1 Dec 2015 0 repositories listed

Syntology lines on 3 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections