{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/shopping-mmlu-a-massive-multi-task-online","title":"Shopping MMLU: A Massive Multi-Task Online Shopping Benchmark for Large Language Models","arxiv_id":"2410.20745","date":"2024-10-28","proceeding":null,"authors":["Yilun Jin","Zheng Li","Chenwei Zhang","Tianyu Cao","Yifan Gao","Pratik Jayarao","Mao Li","Xin Liu","Ritesh Sarkhel","Xianfeng Tang","Haodong Wang","Zhengyang Wang","Wenju Xu","Jingfeng Yang","Qingyu Yin","Xian Li","Priyanka Nigam","Yi Xu","Kai Chen","Qiang Yang","Meng Jiang","Bing Yin"],"abstract":"Online shopping is a complex multi-task, few-shot learning problem with a wide and evolving range of entities, relations, and tasks. However, existing models and benchmarks are commonly tailored to specific tasks, falling short of capturing the full complexity of online shopping. Large Language Models (LLMs), with their multi-task and few-shot learning abilities, have the potential to profoundly transform online shopping by alleviating task-specific engineering efforts and by providing users with interactive conversations. Despite the potential, LLMs face unique challenges in online shopping, such as domain-specific concepts, implicit knowledge, and heterogeneous user behaviors. Motivated by the potential and challenges, we propose Shopping MMLU, a diverse multi-task online shopping benchmark derived from real-world Amazon data. Shopping MMLU consists of 57 tasks covering 4 major shopping skills: concept understanding, knowledge reasoning, user behavior alignment, and multi-linguality, and can thus comprehensively evaluate the abilities of LLMs as general shop assistants. With Shopping MMLU, we benchmark over 20 existing LLMs and uncover valuable insights about practices and prospects of building versatile LLM-based shop assistants. Shopping MMLU can be publicly accessed at https://github.com/KL4805/ShoppingMMLU. In addition, with Shopping MMLU, we host a competition in KDD Cup 2024 with over 500 participating teams. The winning solutions and the associated workshop can be accessed at our website https://amazon-kddcup24.github.io/.","url_abs":"https://arxiv.org/abs/2410.20745v2","url_pdf":"https://arxiv.org/pdf/2410.20745v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"shopping-mmlu-a-massive-multi-task-online","repo_url":"https://github.com/kl4805/shoppingmmlu","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"few-shot-learning","task_name":"Few-Shot Learning"},{"task_slug":"mmlu","task_name":"MMLU"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2410.20745","atlas_url":"https://app.syntology.ai/?focus=2410.20745","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.20745"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/KL4805/ShoppingMMLU","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kl4805/shoppingmmlu","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":9},"by_repo_kind":{"official":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"28028603b5a28958","entry":"accuracy","repo":"KL4805/ShoppingMMLU","repo_kind":"official","path":"skill_wise_eval/metrics.py","file_url":"https://github.com/KL4805/ShoppingMMLU/blob/HEAD/skill_wise_eval/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"28028603b5a28958"}},{"code_sha256_prefix":"c78d363cf62ef64c","entry":"format_example","repo":"KL4805/ShoppingMMLU","repo_kind":"official","path":"task_wise_eval/utils.py","file_url":"https://github.com/KL4805/ShoppingMMLU/blob/HEAD/task_wise_eval/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c78d363cf62ef64c"}},{"code_sha256_prefix":"83fbf53277c007f3","entry":"format_subject","repo":"KL4805/ShoppingMMLU","repo_kind":"official","path":"task_wise_eval/utils.py","file_url":"https://github.com/KL4805/ShoppingMMLU/blob/HEAD/task_wise_eval/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"83fbf53277c007f3"}},{"code_sha256_prefix":"bf4dd50c2d214d4e","entry":"hit_rate","repo":"KL4805/ShoppingMMLU","repo_kind":"official","path":"skill_wise_eval/metrics.py","file_url":"https://github.com/KL4805/ShoppingMMLU/blob/HEAD/skill_wise_eval/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bf4dd50c2d214d4e"}},{"code_sha256_prefix":"8cac98717a6cf50a","entry":"is_permutation","repo":"KL4805/ShoppingMMLU","repo_kind":"official","path":"task_wise_eval/hf_ranking.py","file_url":"https://github.com/KL4805/ShoppingMMLU/blob/HEAD/task_wise_eval/hf_ranking.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8cac98717a6cf50a"}},{"code_sha256_prefix":"6cad10d0763d137f","entry":"load_tokenizer_and_model","repo":"KL4805/ShoppingMMLU","repo_kind":"official","path":"skill_wise_eval/utils.py","file_url":"https://github.com/KL4805/ShoppingMMLU/blob/HEAD/skill_wise_eval/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6cad10d0763d137f"}},{"code_sha256_prefix":"f2b76d01892e989d","entry":"load_tokenizer_and_model","repo":"KL4805/ShoppingMMLU","repo_kind":"official","path":"task_wise_eval/utils.py","file_url":"https://github.com/KL4805/ShoppingMMLU/blob/HEAD/task_wise_eval/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f2b76d01892e989d"}},{"code_sha256_prefix":"8d9104b9e41ddb8b","entry":"ndcg","repo":"KL4805/ShoppingMMLU","repo_kind":"official","path":"task_wise_eval/hf_ranking.py","file_url":"https://github.com/KL4805/ShoppingMMLU/blob/HEAD/task_wise_eval/hf_ranking.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8d9104b9e41ddb8b"}},{"code_sha256_prefix":"101fc21a6b0cd8ff","entry":"rougel","repo":"KL4805/ShoppingMMLU","repo_kind":"official","path":"skill_wise_eval/metrics.py","file_url":"https://github.com/KL4805/ShoppingMMLU/blob/HEAD/skill_wise_eval/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"101fc21a6b0cd8ff"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}