{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/visually-grounded-reasoning-across-languages","title":"Visually Grounded Reasoning across Languages and Cultures","arxiv_id":"2109.13238","date":"2021-09-28","proceeding":"EMNLP 2021 11","authors":["Fangyu Liu","Emanuele Bugliarello","Edoardo Maria Ponti","Siva Reddy","Nigel Collier","Desmond Elliott"],"abstract":"The design of widespread vision-and-language datasets and pre-trained encoders directly adopts, or draws inspiration from, the concepts and images of ImageNet. While one can hardly overestimate how much this benchmark contributed to progress in computer vision, it is mostly derived from lexical databases and image queries in English, resulting in source material with a North American or Western European bias. Therefore, we devise a new protocol to construct an ImageNet-style hierarchy representative of more languages and cultures. In particular, we let the selection of both concepts and images be entirely driven by native speakers, rather than scraping them automatically. Specifically, we focus on a typologically diverse set of languages, namely, Indonesian, Mandarin Chinese, Swahili, Tamil, and Turkish. On top of the concepts and images obtained through this new protocol, we create a multilingual dataset for {M}ulticultur{a}l {R}easoning over {V}ision and {L}anguage (MaRVL) by eliciting statements from native speaker annotators about pairs of images. The task consists of discriminating whether each grounded statement is true or false. We establish a series of baselines using state-of-the-art models and find that their cross-lingual transfer performance lags dramatically behind supervised performance in English. These results invite us to reassess the robustness and accuracy of current state-of-the-art models beyond a narrow domain, but also open up new exciting challenges for the development of truly multilingual and multicultural systems.","url_abs":"https://arxiv.org/abs/2109.13238v2","url_pdf":"https://arxiv.org/pdf/2109.13238v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"visually-grounded-reasoning-across-languages","repo_url":"https://github.com/e-bug/volta","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"visually-grounded-reasoning-across-languages","repo_url":"https://github.com/marvl-challenge/marvl-code","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"visually-grounded-reasoning-across-languages","repo_url":"https://github.com/shin-ee-chen/bla","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"cross-lingual-transfer","task_name":"Cross-Lingual Transfer"},{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"}],"methods":[{"method_slug":null,"method_name":"American"}],"datasets_introduced":[{"slug":"marvl","name":"MaRVL","full_name":"Multicultural Reasoning over Vision and Language"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/zero-shot-cross-lingual-transfer-on-marvl","task":"Zero-Shot Cross-Lingual Transfer","dataset":"MaRVL","model":"xUNITER","rank_in_archive_order":1,"of":2,"metrics":{"Accuracy (%)":"56.1"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-cross-lingual-transfer-on-marvl","task":"Zero-Shot Cross-Lingual Transfer","dataset":"MaRVL","model":"mUNITER","rank_in_archive_order":2,"of":2,"metrics":{"Accuracy (%)":"54.0"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2109.13238","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2109.13238"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shin-ee-chen/bla","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/marvl-challenge/marvl-code","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/e-bug/volta","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4c2f89511d326a22","entry":"collate_fn","repo":"marvl-challenge/marvl-code","repo_kind":"official","path":"feature_extraction/detectron2_proposal_maxnms.py","file_url":"https://github.com/marvl-challenge/marvl-code/blob/HEAD/feature_extraction/detectron2_proposal_maxnms.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4c2f89511d326a22"}},{"code_sha256_prefix":"a2faafba33358747","entry":"kl_1601","repo":"marvl-challenge/marvl-code","repo_kind":"official","path":"volta/volta/losses.py","file_url":"https://github.com/marvl-challenge/marvl-code/blob/HEAD/volta/volta/losses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a2faafba33358747"}},{"code_sha256_prefix":"1adf127958517c64","entry":"mse_2048","repo":"marvl-challenge/marvl-code","repo_kind":"official","path":"volta/volta/losses.py","file_url":"https://github.com/marvl-challenge/marvl-code/blob/HEAD/volta/volta/losses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1adf127958517c64"}},{"code_sha256_prefix":"60cab169db54e986","entry":"nce_2048","repo":"marvl-challenge/marvl-code","repo_kind":"official","path":"volta/volta/losses.py","file_url":"https://github.com/marvl-challenge/marvl-code/blob/HEAD/volta/volta/losses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"60cab169db54e986"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}