{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/worldcuisines-a-massive-scale-benchmark-for","title":"WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisines","arxiv_id":"2410.12705","date":"2024-10-16","proceeding":null,"authors":["Genta Indra Winata","Frederikus Hudi","Patrick Amadeus Irawan","David Anugraha","Rifki Afina Putri","Yutong Wang","Adam Nohejl","Ubaidillah Ariq Prathama","Nedjma Ousidhoum","Afifa Amriani","Anar Rzayev","Anirban Das","Ashmari Pramodya","Aulia Adila","Bryan Wilie","Candy Olivia Mawalim","Ching Lam Cheng","Daud Abolade","Emmanuele Chersoni","Enrico Santus","Fariz Ikhwantri","Garry Kuwanto","Hanyang Zhao","Haryo Akbarianto Wibowo","Holy Lovenia","Jan Christian Blaise Cruz","Jan Wira Gotama Putra","Junho Myung","Lucky Susanto","Maria Angelica Riera Machin","Marina Zhukova","Michael Anugraha","Muhammad Farid Adilazuarda","Natasha Santosa","Peerat Limkonchotiwat","Raj Dabre","Rio Alexander Audino","Samuel Cahyawijaya","Shi-Xiong Zhang","Stephanie Yulia Salim","Yi Zhou","Yinxuan Gui","David Ifeoluwa Adelani","En-Shiun Annie Lee","Shogo Okada","Ayu Purwarianti","Alham Fikri Aji","Taro Watanabe","Derry Tanti Wijaya","Alice Oh","Chong-Wah Ngo"],"abstract":"Vision Language Models (VLMs) often struggle with culture-specific knowledge, particularly in languages other than English and in underrepresented cultural contexts. To evaluate their understanding of such knowledge, we introduce WorldCuisines, a massive-scale benchmark for multilingual and multicultural, visually grounded language understanding. This benchmark includes a visual question answering (VQA) dataset with text-image pairs across 30 languages and dialects, spanning 9 language families and featuring over 1 million data points, making it the largest multicultural VQA benchmark to date. It includes tasks for identifying dish names and their origins. We provide evaluation datasets in two sizes (12k and 60k instances) alongside a training dataset (1 million instances). Our findings show that while VLMs perform better with correct location context, they struggle with adversarial contexts and predicting specific regional cuisines and languages. To support future research, we release a knowledge base with annotated food entries and images along with the VQA data.","url_abs":"https://arxiv.org/abs/2410.12705v5","url_pdf":"https://arxiv.org/pdf/2410.12705v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"worldcuisines-a-massive-scale-benchmark-for","repo_url":"https://github.com/worldcuisines/worldcuisines","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[],"datasets_introduced":[{"slug":"worldcuisines","name":"WorldCuisines","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2410.12705","atlas_url":"https://app.syntology.ai/?focus=2410.12705","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.12705"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/worldcuisines/worldcuisines","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":9,"unverified":8},"by_repo_kind":{"official":{"samples":17,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c4227ba3d818e9d5","entry":"eval_instance","repo":"worldcuisines/worldcuisines","repo_kind":"official","path":"evaluation/src/aria.py","file_url":"https://github.com/worldcuisines/worldcuisines/blob/HEAD/evaluation/src/aria.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c4227ba3d818e9d5"}},{"code_sha256_prefix":"884a4e29f634f7da","entry":"eval_instance","repo":"worldcuisines/worldcuisines","repo_kind":"official","path":"evaluation/src/llama.py","file_url":"https://github.com/worldcuisines/worldcuisines/blob/HEAD/evaluation/src/llama.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"884a4e29f634f7da"}},{"code_sha256_prefix":"ef2279719389627b","entry":"eval_instance","repo":"worldcuisines/worldcuisines","repo_kind":"official","path":"evaluation/src/llava.py","file_url":"https://github.com/worldcuisines/worldcuisines/blob/HEAD/evaluation/src/llava.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ef2279719389627b"}},{"code_sha256_prefix":"8fb75acb8c7ec039","entry":"final_cat_mapping","repo":"worldcuisines/worldcuisines","repo_kind":"official","path":"evaluation/score/score.py","file_url":"https://github.com/worldcuisines/worldcuisines/blob/HEAD/evaluation/score/score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8fb75acb8c7ec039"}},{"code_sha256_prefix":"92244c7286cce0f9","entry":"get_unique_filename","repo":"worldcuisines/worldcuisines","repo_kind":"official","path":"evaluation/src/base_model.py","file_url":"https://github.com/worldcuisines/worldcuisines/blob/HEAD/evaluation/src/base_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"92244c7286cce0f9"}},{"code_sha256_prefix":"c5c29786001f4200","entry":"get_url_jpg_map","repo":"worldcuisines/worldcuisines","repo_kind":"official","path":"evaluation/src/base_model.py","file_url":"https://github.com/worldcuisines/worldcuisines/blob/HEAD/evaluation/src/base_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c5c29786001f4200"}},{"code_sha256_prefix":"bf44cfc1771f959a","entry":"load_jsonl","repo":"worldcuisines/worldcuisines","repo_kind":"official","path":"evaluation/score/score.py","file_url":"https://github.com/worldcuisines/worldcuisines/blob/HEAD/evaluation/score/score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bf44cfc1771f959a"}},{"code_sha256_prefix":"1af4a289b725ad56","entry":"load_model_processor","repo":"worldcuisines/worldcuisines","repo_kind":"official","path":"evaluation/src/molmo.py","file_url":"https://github.com/worldcuisines/worldcuisines/blob/HEAD/evaluation/src/molmo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1af4a289b725ad56"}},{"code_sha256_prefix":"3acf8494948af533","entry":"load_uploaded_files","repo":"worldcuisines/worldcuisines","repo_kind":"official","path":"evaluation/gemini.py","file_url":"https://github.com/worldcuisines/worldcuisines/blob/HEAD/evaluation/gemini.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3acf8494948af533"}},{"code_sha256_prefix":"4423a1422aa933d1","entry":"eval_instance","repo":"worldcuisines/worldcuisines","repo_kind":"official","path":"evaluation/src/molmo.py","file_url":"https://github.com/worldcuisines/worldcuisines/blob/HEAD/evaluation/src/molmo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4423a1422aa933d1"}},{"code_sha256_prefix":"59daa7bc4cd39ffa","entry":"get_image","repo":"worldcuisines/worldcuisines","repo_kind":"official","path":"evaluation/gemini.py","file_url":"https://github.com/worldcuisines/worldcuisines/blob/HEAD/evaluation/gemini.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"59daa7bc4cd39ffa"}},{"code_sha256_prefix":"ad4fdc995ecb91f4","entry":"get_vqa_from_hf","repo":"worldcuisines/worldcuisines","repo_kind":"official","path":"evaluation/generate_openai_batch_api.py","file_url":"https://github.com/worldcuisines/worldcuisines/blob/HEAD/evaluation/generate_openai_batch_api.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ad4fdc995ecb91f4"}},{"code_sha256_prefix":"e57f7366380e2537","entry":"get_vqa_from_hf","repo":"worldcuisines/worldcuisines","repo_kind":"official","path":"evaluation/src/base_model.py","file_url":"https://github.com/worldcuisines/worldcuisines/blob/HEAD/evaluation/src/base_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e57f7366380e2537"}},{"code_sha256_prefix":"cb78c8e64d7b0e3a","entry":"load_model_processor","repo":"worldcuisines/worldcuisines","repo_kind":"official","path":"evaluation/src/aria.py","file_url":"https://github.com/worldcuisines/worldcuisines/blob/HEAD/evaluation/src/aria.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cb78c8e64d7b0e3a"}},{"code_sha256_prefix":"d3cf62ea670a86f0","entry":"load_model_processor","repo":"worldcuisines/worldcuisines","repo_kind":"official","path":"evaluation/src/llama.py","file_url":"https://github.com/worldcuisines/worldcuisines/blob/HEAD/evaluation/src/llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d3cf62ea670a86f0"}},{"code_sha256_prefix":"b981a004699d96c8","entry":"load_model_processor","repo":"worldcuisines/worldcuisines","repo_kind":"official","path":"evaluation/src/llava.py","file_url":"https://github.com/worldcuisines/worldcuisines/blob/HEAD/evaluation/src/llava.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b981a004699d96c8"}},{"code_sha256_prefix":"d2fb3f56c8a5c3b4","entry":"load_result","repo":"worldcuisines/worldcuisines","repo_kind":"official","path":"evaluation/score/score.py","file_url":"https://github.com/worldcuisines/worldcuisines/blob/HEAD/evaluation/score/score.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d2fb3f56c8a5c3b4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}