{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-tri-modal-embeddings-for-zero-shot","title":"Learning Tri-modal Embeddings for Zero-Shot Soundscape Mapping","arxiv_id":"2309.10667","date":"2023-09-19","proceeding":null,"authors":["Subash Khanal","Srikumar Sastry","Aayush Dhakal","Nathan Jacobs"],"abstract":"We focus on the task of soundscape mapping, which involves predicting the most probable sounds that could be perceived at a particular geographic location. We utilise recent state-of-the-art models to encode geotagged audio, a textual description of the audio, and an overhead image of its capture location using contrastive pre-training. The end result is a shared embedding space for the three modalities, which enables the construction of soundscape maps for any geographic region from textual or audio queries. Using the SoundingEarth dataset, we find that our approach significantly outperforms the existing SOTA, with an improvement of image-to-audio Recall@100 from 0.256 to 0.450. Our code is available at https://github.com/mvrl/geoclap.","url_abs":"https://arxiv.org/abs/2309.10667v1","url_pdf":"https://arxiv.org/pdf/2309.10667v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-tri-modal-embeddings-for-zero-shot","repo_url":"https://github.com/mvrl/geoclap","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"cross-modal-retrieval","task_name":"Cross-Modal Retrieval"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/cross-modal-retrieval-on-soundingearth","task":"Cross-Modal Retrieval","dataset":"SoundingEarth","model":"GeoCLAP","rank_in_archive_order":1,"of":2,"metrics":{"Image-to-sound R@100":"0.434","Median Rank":"159","Sound-to-image R@100":"0.434"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.10667","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.10667"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mvrl/geoclap","reach":{"status":"ok"}}],"summary":{"ran":4,"ran_fixture":1,"unverified":5},"by_repo_kind":{"official":{"samples":10,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"38f07e336b29c0a5","entry":"build_transform","repo":"mvrl/geoclap","repo_kind":"official","path":"geoclap/utilities/SATMAE_transform.py","file_url":"https://github.com/mvrl/geoclap/blob/HEAD/geoclap/utilities/SATMAE_transform.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"38f07e336b29c0a5"}},{"code_sha256_prefix":"4bdd36eab04c3f62","entry":"get_1d_sincos_pos_embed_from_grid","repo":"mvrl/geoclap","repo_kind":"official","path":"geoclap/models/SATMAE.py","file_url":"https://github.com/mvrl/geoclap/blob/HEAD/geoclap/models/SATMAE.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4bdd36eab04c3f62"}},{"code_sha256_prefix":"3a569587facdd418","entry":"get_retrevial_metrics","repo":"mvrl/geoclap","repo_kind":"official","path":"geoclap/metrics.py","file_url":"https://github.com/mvrl/geoclap/blob/HEAD/geoclap/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3a569587facdd418"}},{"code_sha256_prefix":"7d232c101df226df","entry":"read_csv","repo":"mvrl/geoclap","repo_kind":"official","path":"geoclap/dataloader.py","file_url":"https://github.com/mvrl/geoclap/blob/HEAD/geoclap/dataloader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7d232c101df226df"}},{"code_sha256_prefix":"abed70a10c6a4d1d","entry":"stratified_sample_df","repo":"mvrl/geoclap","repo_kind":"official","path":"geoclap/miscs/compute_similarity.py","file_url":"https://github.com/mvrl/geoclap/blob/HEAD/geoclap/miscs/compute_similarity.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"abed70a10c6a4d1d"}},{"code_sha256_prefix":"9417ae492629cf06","entry":"get_2d_sincos_pos_embed","repo":"mvrl/geoclap","repo_kind":"official","path":"geoclap/models/SATMAE.py","file_url":"https://github.com/mvrl/geoclap/blob/HEAD/geoclap/models/SATMAE.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9417ae492629cf06"}},{"code_sha256_prefix":"4fd80de79832745d","entry":"get_2d_sincos_pos_embed_from_grid","repo":"mvrl/geoclap","repo_kind":"official","path":"geoclap/models/SATMAE.py","file_url":"https://github.com/mvrl/geoclap/blob/HEAD/geoclap/models/SATMAE.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4fd80de79832745d"}},{"code_sha256_prefix":"46a8aabfa19195d4","entry":"get_processed_audio","repo":"mvrl/geoclap","repo_kind":"official","path":"geoclap/dataloader.py","file_url":"https://github.com/mvrl/geoclap/blob/HEAD/geoclap/dataloader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"46a8aabfa19195d4"}},{"code_sha256_prefix":"cfde410a49fd6d24","entry":"get_processed_text","repo":"mvrl/geoclap","repo_kind":"official","path":"geoclap/dataloader.py","file_url":"https://github.com/mvrl/geoclap/blob/HEAD/geoclap/dataloader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cfde410a49fd6d24"}},{"code_sha256_prefix":"d32a4eff3b49ec7b","entry":"get_retrevial","repo":"mvrl/geoclap","repo_kind":"official","path":"geoclap/metrics.py","file_url":"https://github.com/mvrl/geoclap/blob/HEAD/geoclap/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d32a4eff3b49ec7b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}