Browse State-of-the-Art › audio-visual learning
audio-visual learning
24 papers with code · 0 benchmarks · 4 datasets archive 2025-07-28
Benchmarks archive 2025-07-28
No benchmark for this task in the archive.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
4 datasets whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
No subtask under this task in the archive's task tree.
Papers archive 2025-07-28
archive papers tagged: 38 · with a code link: 24 · where Syntology ran a sample: 11 (10 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (11 of 38 tagged: 10 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 38 of 38, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
22 May 2023 2 repositories listed Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 5 where Syntology's instrument failed) · 6 unverified (of 11 harvested samples) · 6 pointer-only (licence)In this paper, we propose a novel method utilizing latent diffusion models trained for text-to-image-generation to generate images conditioned on audio recordings.
-
2 May 2025 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 2 harvested samples)Second, we resolve conflicting optimization goals by separating contrastive and reconstruction objectives through dedicated global tokens.
-
1 Jan 2025 1 repository listedLong-term sports assessment is a challenging task in video understanding since it requires judging complex movement variations and action-music coordination.
-
17 Dec 2024 1 repository listed Syntology official (archive's flag): 13 ran · 13 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 2 honoured, 0 violated, 1 with no contract checked; 10 where Syntology's instrument failed) · 1 unverified (of 14 harvested samples) · 14 pointer-only (licence)Specifically, the CMCC module contains two branches: a cross-modal interaction branch and a temporal consistency-gated branch.
-
29 Aug 2024 1 repository listedTo address this issue, we propose a novel audio-visual learning framework which is instantiated with two individual learning schemes: self-supervised predictive learning (SSPL) and semantic-aware contrastive learning…
-
7 Jun 2024 1 repository listedFurthermore, to suppress the background features in each modality from foreground matched audio-visual features, we introduce a robust discriminative foreground mining scheme.
-
14 Mar 2024 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 4 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 0 violated, 4 with no contract checked; 1 where Syntology's instrument failed) · 7 unverified (of 13 harvested samples) · 2 pointer-only (licence)Recent advancements in self-supervised audio-visual representation learning have demonstrated its potential to capture rich and comprehensive representations.
-
29 Nov 2023 1 repository listedThe prevailing use of SVs to spread emotions leads to the necessity of conducting video emotion analysis (VEA) towards SVs.
-
21 Nov 2023 1 repository listedRecent methods mainly focus on learning multi-modal features aligned with class names to enhance the generalization ability to unseen categories.
-
7 Nov 2023 1 repository listed Syntology official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 0 violated, 10 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 10 harvested samples) · 10 pointer-only (licence)Large-scale pre-trained image-text models demonstrate remarkable versatility across diverse tasks, benefiting from their robust representational capabilities and effective multimodal alignment.
-
11 Oct 2023 1 repository listedTo implement the prior knowledge, we first train the audio-visual network, which learns the correspondence between auditory and visual information.
-
19 Sep 2023 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 3 harvested samples) · 3 pointer-only (licence)Audio-visual representation learning aims to develop systems with human-like perception by utilizing correlation between auditory and visual information.
-
11 Sep 2023 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 2 where Syntology's instrument failed) · 2 unverified (of 8 harvested samples)Our CIGN leverages learnable audio-visual class tokens and audio-visual grouping to continually aggregate class-aware features.
-
30 May 2023 1 repository listedThe ability to accurately recognize, localize and separate sound sources is fundamental to any audio-visual perception task.
-
27 May 2023 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 2 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 0 violated, 3 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 6 harvested samples)Audio-visual learning has been a major pillar of multi-modal machine learning, where the community mostly focused on its modality-aligned setting, i. e., the audio and visual modality are both assumed to signal the…
-
6 Apr 2023 1 repository listedWe show empirical results that demonstrate the effectiveness of our benchmark.
-
7 Feb 2023 1 repository listedSpecifically, we explore the effects of pre-trained models on two audio-visual learning scenarios: cross-modal initialization and multi-modal joint learning.
-
29 Jul 2022 1 repository listed Syntology official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 1 honoured, 1 violated, 4 with no contract checked; 4 where Syntology's instrument failed) · 4 unverified (of 14 harvested samples) · 3 pointer-only (licence)Conventional audio-visual models have independent audio and video branches.
-
12 Jul 2022 1 repository listed Syntology official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 1 violated, 6 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 10 harvested samples) · 1 pointer-only (licence)In this paper, we analyze the modality asynchrony and undifferentiated instances phenomena of the multiple instance learning (MIL) procedure, and further investigate its negative impact on weakly-supervised audio-visual…
-
26 Mar 2022 1 repository listed Syntology official: harvested, nothing ran · 0 ran · 4 unverified (of 4 harvested samples)In this paper, we focus on the Audio-Visual Question Answering (AVQA) task, which aims to answer questions regarding different visual objects, sounds, and their associations in videos.
-
8 Nov 2021 1 repository listedIn this paper, we explore self-supervised audio-visual models that learn from instructional videos.
-
22 Apr 2021 1 repository listed Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample)Having access to multi-modal cues (e. g. vision and audio) empowers some cognitive tasks to be done faster compared to learning from a single modality.
-
5 Apr 2021 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)In this paper, we propose to make a systematic study on machines multisensory perception under attacks.
-
12 Jan 2021 1 repository listedAML aims to generate a modality-independent representation for each person in each modality via adversarial learning, while simultaneously learns a robust similarity measure for cross-modality matching via metric…
-
Lightweight Joint Audio-Visual Deepfake Detection via Single-Stream Multi-Modal Learning Framework9 Jun 2025 0 repositories listedDeepfakes are AI-synthesized multimedia data that may be abused for spreading misinformation.
-
Rethinking Audio-Visual Adversarial Vulnerability from Temporal and Modality Perspectives17 Feb 2025 0 repositories listedWhile audio-visual learning equips models with a richer understanding of the real world by leveraging multiple sensory modalities, this integration also introduces new vulnerabilities to adversarial attacks.
-
Unveiling Visual Biases in Audio-Visual Localization Benchmarks25 Aug 2024 0 repositories listedAudio-Visual Source Localization (AVSL) aims to localize the source of sound within a video.
-
Sequential Contrastive Audio-Visual Learning8 Jul 2024 0 repositories listedContrastive learning has emerged as a powerful technique in audio-visual representation learning, leveraging the natural co-occurrence of audio and visual modalities in webscale video datasets.
-
Multi-Input Multi-Output Target-Speaker Voice Activity Detection For Unified, Flexible, and Robust Audio-Visual Speaker Diarization16 Jan 2024 0 repositories listedAudio-visual learning has demonstrated promising results in many classical speech tasks (e.g., speech separation, automatic speech recognition, wake-word spotting).
-
Leveraging Pretrained Image-text Models for Improving Audio-Visual Learning8 Sep 2023 0 repositories listedVisually grounded speech systems learn from paired images and their spoken captions.
-
RealImpact: A Dataset of Impact Sound Fields for Real Objects16 Jun 2023 0 repositories listed
-
12 May 2023 0 repositories listed
-
Object Segmentation with Audio Context4 Jan 2023 0 repositories listed
-
Learning in Audio-visual Context: A Review, Analysis, and New Perspective20 Aug 2022 0 repositories listed
-
Few-Shot Audio-Visual Learning of Environment Acoustics8 Jun 2022 0 repositories listed
-
Telling Left from Right: Learning Spatial Correspondence of Sight and Sound11 Jun 2020 0 repositories listed
-
Deep Audio-Visual Learning: A Survey14 Jan 2020 0 repositories listed
-
Audio-Visual Embedding for Cross-Modal MusicVideo Retrieval through Supervised Deep CCA10 Aug 2019 0 repositories listed
Syntology lines on 13 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections