{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-explainable-harmful-meme-detection","title":"Towards Explainable Harmful Meme Detection through Multimodal Debate between Large Language Models","arxiv_id":"2401.13298","date":"2024-01-24","proceeding":null,"authors":["Hongzhan Lin","Ziyang Luo","Wei Gao","Jing Ma","Bo wang","Ruichao Yang"],"abstract":"The age of social media is flooded with Internet memes, necessitating a clear grasp and effective identification of harmful ones. This task presents a significant challenge due to the implicit meaning embedded in memes, which is not explicitly conveyed through the surface text and image. However, existing harmful meme detection methods do not present readable explanations that unveil such implicit meaning to support their detection decisions. In this paper, we propose an explainable approach to detect harmful memes, achieved through reasoning over conflicting rationales from both harmless and harmful positions. Specifically, inspired by the powerful capacity of Large Language Models (LLMs) on text generation and reasoning, we first elicit multimodal debate between LLMs to generate the explanations derived from the contradictory arguments. Then we propose to fine-tune a small language model as the debate judge for harmfulness inference, to facilitate multimodal fusion between the harmfulness rationales and the intrinsic multimodal information within memes. In this way, our model is empowered to perform dialectical reasoning over intricate and implicit harm-indicative patterns, utilizing multimodal explanations originating from both harmless and harmful arguments. Extensive experiments on three public meme datasets demonstrate that our harmful meme detection approach achieves much better performance than state-of-the-art methods and exhibits a superior capacity for explaining the meme harmfulness of the model predictions.","url_abs":"https://arxiv.org/abs/2401.13298v1","url_pdf":"https://arxiv.org/pdf/2401.13298v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-explainable-harmful-meme-detection","repo_url":"https://github.com/hkbunlp/explainhm-www2024","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"hateful-meme-classification","task_name":"Hateful Meme Classification"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"small-language-model","task_name":"Small Language Model"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/hateful-meme-classification-on-harm-p","task":"Hateful Meme Classification","dataset":"Harm-P","model":"ExplainHM","rank_in_archive_order":2,"of":5,"metrics":{"Accuracy":"90.7","F1":"90.7"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.13298","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.13298"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hkbunlp/explainhm-www2024","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":4,"unverified":2},"by_repo_kind":{"official":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"839a33f049497ac2","entry":"all_gather","repo":"hkbunlp/explainhm-www2024","repo_kind":"official","path":"src/modules/dist_utils.py","file_url":"https://github.com/hkbunlp/explainhm-www2024/blob/HEAD/src/modules/dist_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"839a33f049497ac2"}},{"code_sha256_prefix":"92a41a954c7ecb1d","entry":"check_non_acc_grad","repo":"hkbunlp/explainhm-www2024","repo_kind":"official","path":"src/modules/mm_utils.py","file_url":"https://github.com/hkbunlp/explainhm-www2024/blob/HEAD/src/modules/mm_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"92a41a954c7ecb1d"}},{"code_sha256_prefix":"cc8ab2b168b5373d","entry":"gather","repo":"hkbunlp/explainhm-www2024","repo_kind":"official","path":"src/modules/dist_utils.py","file_url":"https://github.com/hkbunlp/explainhm-www2024/blob/HEAD/src/modules/dist_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cc8ab2b168b5373d"}},{"code_sha256_prefix":"53838c15d969eb79","entry":"reduce_dict","repo":"hkbunlp/explainhm-www2024","repo_kind":"official","path":"src/modules/dist_utils.py","file_url":"https://github.com/hkbunlp/explainhm-www2024/blob/HEAD/src/modules/dist_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"53838c15d969eb79"}},{"code_sha256_prefix":"9472ae65ddd756a4","entry":"compute_clf","repo":"hkbunlp/explainhm-www2024","repo_kind":"official","path":"src/modules/objectives.py","file_url":"https://github.com/hkbunlp/explainhm-www2024/blob/HEAD/src/modules/objectives.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9472ae65ddd756a4"}},{"code_sha256_prefix":"fb1f56a05ab8f4eb","entry":"get_pretrained_tokenizer","repo":"hkbunlp/explainhm-www2024","repo_kind":"official","path":"src/datamodules/datamodule_base.py","file_url":"https://github.com/hkbunlp/explainhm-www2024/blob/HEAD/src/datamodules/datamodule_base.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fb1f56a05ab8f4eb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}