{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/audio-visual-generalised-zero-shot-learning","title":"Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Language","arxiv_id":"2203.03598","date":"2022-03-07","proceeding":"CVPR 2022 1","authors":["Otniel-Bogdan Mercea","Lukas Riesch","A. Sophia Koepke","Zeynep Akata"],"abstract":"Learning to classify video data from classes not included in the training data, i.e. video-based zero-shot learning, is challenging. We conjecture that the natural alignment between the audio and visual modalities in video data provides a rich training signal for learning discriminative multi-modal representations. Focusing on the relatively underexplored task of audio-visual zero-shot learning, we propose to learn multi-modal representations from audio-visual data using cross-modal attention and exploit textual label embeddings for transferring knowledge from seen classes to unseen classes. Taking this one step further, in our generalised audio-visual zero-shot learning setting, we include all the training classes in the test-time search space which act as distractors and increase the difficulty while making the setting more realistic. Due to the lack of a unified benchmark in this domain, we introduce a (generalised) zero-shot learning benchmark on three audio-visual datasets of varying sizes and difficulty, VGGSound, UCF, and ActivityNet, ensuring that the unseen test classes do not appear in the dataset used for supervised training of the backbone deep models. Comparing multiple relevant and recent methods, we demonstrate that our proposed AVCA model achieves state-of-the-art performance on all three datasets. Code and data are available at \\url{https://github.com/ExplainableML/AVCA-GZSL}.","url_abs":"https://arxiv.org/abs/2203.03598v2","url_pdf":"https://arxiv.org/pdf/2203.03598v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"audio-visual-generalised-zero-shot-learning","repo_url":"https://github.com/explainableml/avca-gzsl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"gzsl-video-classification","task_name":"GZSL Video Classification"},{"task_slug":"zsl-video-classification","task_name":"ZSL Video Classification"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/gzsl-video-classification-on-activitynet-gzsl","task":"GZSL Video Classification","dataset":"ActivityNet-GZSL (cls)","model":"AVCA","rank_in_archive_order":4,"of":4,"metrics":{"HM":"9.92"},"uses_additional_data":false},{"leaderboard":"/sota/gzsl-video-classification-on-activitynet-gzsl-1","task":"GZSL Video Classification","dataset":"ActivityNet-GZSL(main)","model":"AVCA","rank_in_archive_order":3,"of":7,"metrics":{"HM":"12.13","ZSL":"9.13"},"uses_additional_data":false},{"leaderboard":"/sota/gzsl-video-classification-on-ucf-gzsl-cls","task":"GZSL Video Classification","dataset":"UCF-GZSL (cls)","model":"AVCA","rank_in_archive_order":4,"of":4,"metrics":{"HM":"41.34","ZSL":"37.72"},"uses_additional_data":false},{"leaderboard":"/sota/gzsl-video-classification-on-ucf-gzsl-main","task":"GZSL Video Classification","dataset":"UCF-GZSL(main)","model":"AVCA","rank_in_archive_order":4,"of":7,"metrics":{"HM":"27.15","ZSL":"20.0"},"uses_additional_data":false},{"leaderboard":"/sota/gzsl-video-classification-on-vggsound-gzsl","task":"GZSL Video Classification","dataset":"VGGSound-GZSL (cls)","model":"AVCA","rank_in_archive_order":4,"of":4,"metrics":{"HM":"8.31","ZSL":"6.91"},"uses_additional_data":false},{"leaderboard":"/sota/gzsl-video-classification-on-vggsound-gzsl-1","task":"GZSL Video Classification","dataset":"VGGSound-GZSL(main)","model":"AVCA","rank_in_archive_order":4,"of":7,"metrics":{"HM":"6.31","ZSL":"6.00"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2203.03598","atlas_url":"https://app.syntology.ai/?focus=2203.03598","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2203.03598"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ExplainableML/AVCA-GZSL","reach":null}],"summary":{"ran":3,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"de7ce81032c8b359","entry":"Attention","repo":"ExplainableML/AVCA-GZSL","repo_kind":"official","path":"src/model_improvements.py","file_url":"https://github.com/ExplainableML/AVCA-GZSL/blob/HEAD/src/model_improvements.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"de7ce81032c8b359"}},{"code_sha256_prefix":"3591b61d48e13ac1","entry":"EmbeddingNet","repo":"ExplainableML/AVCA-GZSL","repo_kind":"official","path":"src/model_improvements.py","file_url":"https://github.com/ExplainableML/AVCA-GZSL/blob/HEAD/src/model_improvements.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3591b61d48e13ac1"}},{"code_sha256_prefix":"3cc123bd4fd93c25","entry":"Transformer","repo":"ExplainableML/AVCA-GZSL","repo_kind":"official","path":"src/model_improvements.py","file_url":"https://github.com/ExplainableML/AVCA-GZSL/blob/HEAD/src/model_improvements.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3cc123bd4fd93c25"}},{"code_sha256_prefix":"c90732558c5d8749","entry":"AVCA","repo":"ExplainableML/AVCA-GZSL","repo_kind":"official","path":"src/model_improvements.py","file_url":"https://github.com/ExplainableML/AVCA-GZSL/blob/HEAD/src/model_improvements.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c90732558c5d8749"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}