{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mm-soc-benchmarking-multimodal-large-language","title":"MM-Soc: Benchmarking Multimodal Large Language Models in Social Media Platforms","arxiv_id":"2402.14154","date":"2024-02-21","proceeding":null,"authors":["Yiqiao Jin","MinJe Choi","Gaurav Verma","Jindong Wang","Srijan Kumar"],"abstract":"Social media platforms are hubs for multimodal information exchange, encompassing text, images, and videos, making it challenging for machines to comprehend the information or emotions associated with interactions in online spaces. Multimodal Large Language Models (MLLMs) have emerged as a promising solution to these challenges, yet they struggle to accurately interpret human emotions and complex content such as misinformation. This paper introduces MM-Soc, a comprehensive benchmark designed to evaluate MLLMs' understanding of multimodal social media content. MM-Soc compiles prominent multimodal datasets and incorporates a novel large-scale YouTube tagging dataset, targeting a range of tasks from misinformation detection, hate speech detection, and social context generation. Through our exhaustive evaluation on ten size-variants of four open-source MLLMs, we have identified significant performance disparities, highlighting the need for advancements in models' social understanding capabilities. Our analysis reveals that, in a zero-shot setting, various types of MLLMs generally exhibit difficulties in handling social media tasks. However, MLLMs demonstrate performance improvements post fine-tuning, suggesting potential pathways for improvement. Our code and data are available at https://github.com/claws-lab/MMSoc.git.","url_abs":"https://arxiv.org/abs/2402.14154v3","url_pdf":"https://arxiv.org/pdf/2402.14154v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mm-soc-benchmarking-multimodal-large-language","repo_url":"https://github.com/claws-lab/mmsoc","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"hate-speech-detection","task_name":"Hate Speech Detection"},{"task_slug":"misinformation","task_name":"Misinformation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.14154","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.14154"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/claws-lab/mmsoc","reach":{"status":"ok"}}],"summary":{"ran":5},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"1f67e155cfe5cae9","entry":"pad_tensor","repo":"claws-lab/mmsoc","repo_kind":"official","path":"mmsoc/utils/data_utils.py","file_url":"https://github.com/claws-lab/mmsoc/blob/HEAD/mmsoc/utils/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1f67e155cfe5cae9"}},{"code_sha256_prefix":"7ed141d2cbb448e4","entry":"pad_tensors","repo":"claws-lab/mmsoc","repo_kind":"official","path":"mmsoc/utils/data_utils.py","file_url":"https://github.com/claws-lab/mmsoc/blob/HEAD/mmsoc/utils/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7ed141d2cbb448e4"}},{"code_sha256_prefix":"e832b5b3e8908881","entry":"parse_args","repo":"claws-lab/mmsoc","repo_kind":"official","path":"mmsoc/arguments.py","file_url":"https://github.com/claws-lab/mmsoc/blob/HEAD/mmsoc/arguments.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e832b5b3e8908881"}},{"code_sha256_prefix":"d146939d9fdf8a96","entry":"truncate_input","repo":"claws-lab/mmsoc","repo_kind":"official","path":"mmsoc/utils/model_utils.py","file_url":"https://github.com/claws-lab/mmsoc/blob/HEAD/mmsoc/utils/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d146939d9fdf8a96"}},{"code_sha256_prefix":"1374fce5cd4d2302","entry":"truncate_input","repo":"claws-lab/mmsoc","repo_kind":"official","path":"mmsoc/utils/data_utils.py","file_url":"https://github.com/claws-lab/mmsoc/blob/HEAD/mmsoc/utils/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1374fce5cd4d2302"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}