audio-visual learning

Benchmarks archive 2025-07-28

No benchmark for this task in the archive.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

4 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

No subtask under this task in the archive's task tree.

Most implemented papers archive 2025-07-28

24 shown of 24 papers with code (38 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• AudioToken: Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generation
  22 May 2023 2 repositories listed Syntology ran 4 of 11 samples · 7 unverified
  In this paper, we propose a novel method utilizing latent diffusion models trained for text-to-image-generation to generate images conditioned on audio recordings.
• CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment
  2 May 2025 1 repository listed Syntology ran 1 of 2 samples · 1 unverified
  Second, we resolve conflicting optimization goals by separating contrastive and reconstruction objectives through dedicated global tokens.
• Language-Guided Audio-Visual Learning for Long-Term Sports Assessment
  1 Jan 2025 1 repository listed
  Long-term sports assessment is a challenging task in video understanding since it requires judging complex movement variations and action-music coordination.
• Dense Audio-Visual Event Localization under Cross-Modal Consistency and Multi-Temporal Granularity Collaboration
  17 Dec 2024 1 repository listed Syntology ran 0 of 14 samples · 14 unverified · 14 pointer-only (licence)
  Specifically, the CMCC module contains two branches: a cross-modal interaction branch and a temporal consistency-gated branch.
• Enhancing Sound Source Localization via False Negative Elimination
  29 Aug 2024 1 repository listed
  To address this issue, we propose a novel audio-visual learning framework which is instantiated with two individual learning schemes: self-supervised predictive learning (SSPL) and semantic-aware contrastive learning…
• MA-AVT: Modality Alignment for Parameter-Efficient Audio-Visual Transformers
  7 Jun 2024 1 repository listed
  Furthermore, to suppress the background features in each modality from foreground matched audio-visual features, we introduce a robust discriminative foreground mining scheme.
• EquiAV: Leveraging Equivariance for Audio-Visual Contrastive Learning
  14 Mar 2024 1 repository listed Syntology ran 6 of 13 samples · 7 unverified
  Recent advancements in self-supervised audio-visual representation learning have demonstrated its potential to capture rich and comprehensive representations.
• Towards Emotion Analysis in Short-form Videos: A Large-Scale Dataset and Baseline
  29 Nov 2023 1 repository listed
  The prevailing use of SVs to spread emotions leads to the necessity of conducting video emotion analysis (VEA) towards SVs.
• Boosting Audio-visual Zero-shot Learning with Large Language Models
  21 Nov 2023 1 repository listed
  Recent methods mainly focus on learning multi-modal features aligned with class names to enhance the generalization ability to unseen categories.
• Can CLIP Help Sound Source Localization?
  7 Nov 2023 1 repository listed Syntology ran 10 of 10 samples · 0 unverified · 10 pointer-only (licence)
  Large-scale pre-trained image-text models demonstrate remarkable versatility across diverse tasks, benefiting from their robust representational capabilities and effective multimodal alignment.
• Deep Video Inpainting Guided by Audio-Visual Self-Supervision
  11 Oct 2023 1 repository listed
  To implement the prior knowledge, we first train the audio-visual network, which learns the correspondence between auditory and visual information.
• AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Models
  19 Sep 2023 1 repository listed Syntology ran 2 of 3 samples · 1 unverified · 3 pointer-only (licence)
  Audio-visual representation learning aims to develop systems with human-like perception by utilizing correlation between auditory and visual information.
• Class-Incremental Grouping Network for Continual Audio-Visual Learning
  11 Sep 2023 1 repository listed Syntology ran 6 of 8 samples · 2 unverified
  Our CIGN leverages learnable audio-visual class tokens and audio-visual grouping to continually aggregate class-aware features.
• A Unified Audio-Visual Learning Framework for Localization, Separation, and Recognition
  30 May 2023 1 repository listed
  The ability to accurately recognize, localize and separate sound sources is fundamental to any audio-visual perception task.
• Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parser
  27 May 2023 1 repository listed Syntology ran 6 of 6 samples · 0 unverified
  Audio-visual learning has been a major pillar of multi-modal machine learning, where the community mostly focused on its modality-aligned setting, i.
• Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation
  6 Apr 2023 1 repository listed
  We show empirical results that demonstrate the effectiveness of our benchmark.
• Revisiting Pre-training in Audio-Visual Learning
  7 Feb 2023 1 repository listed
  Specifically, we explore the effects of pre-trained models on two audio-visual learning scenarios: cross-modal initialization and multi-modal joint learning.
• UAVM: Towards Unifying Audio and Visual Models
  29 Jul 2022 1 repository listed Syntology ran 6 of 14 samples · 8 unverified
  Conventional audio-visual models have independent audio and video branches.
• Modality-Aware Contrastive Instance Learning with Self-Distillation for Weakly-Supervised Audio-Visual Violence Detection
  12 Jul 2022 1 repository listed Syntology ran 2 of 10 samples · 8 unverified
  In this paper, we analyze the modality asynchrony and undifferentiated instances phenomena of the multiple instance learning (MIL) procedure, and further investigate its negative impact on weakly-supervised audio-visual…
• Learning to Answer Questions in Dynamic Audio-Visual Scenarios
  26 Mar 2022 1 repository listed Syntology ran 0 of 4 samples · 4 unverified
  In this paper, we focus on the Audio-Visual Question Answering (AVQA) task, which aims to answer questions regarding different visual objects, sounds, and their associations in videos.
• Cascaded Multilingual Audio-Visual Learning from Videos
  8 Nov 2021 1 repository listed
  In this paper, we explore self-supervised audio-visual models that learn from instructional videos.
• Distilling Audio-Visual Knowledge by Compositional Contrastive Learning
  22 Apr 2021 1 repository listed Syntology ran 0 of 1 samples · 1 unverified
  Having access to multi-modal cues (e.
• Can audio-visual integration strengthen robustness under multimodal attacks?
  5 Apr 2021 1 repository listed Syntology ran 1 of 1 samples · 0 unverified
  In this paper, we propose to make a systematic study on machines multisensory perception under attacks.
• Adversarial-Metric Learning for Audio-Visual Cross-Modal Matching
  12 Jan 2021 1 repository listed
  AML aims to generate a modality-independent representation for each person in each modality via adversarial learning, while simultaneously learns a robust similarity measure for cross-modality matching via metric…

Syntology lines on 13 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.