{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/vdc-versatile-data-cleanser-for-detecting","title":"VDC: Versatile Data Cleanser based on Visual-Linguistic Inconsistency by Multimodal Large Language Models","arxiv_id":"2309.16211","date":"2023-09-28","proceeding":null,"authors":["Zihao Zhu","Mingda Zhang","Shaokui Wei","Bingzhe Wu","Baoyuan Wu"],"abstract":"The role of data in building AI systems has recently been emphasized by the emerging concept of data-centric AI. Unfortunately, in the real-world, datasets may contain dirty samples, such as poisoned samples from backdoor attack, noisy labels in crowdsourcing, and even hybrids of them. The presence of such dirty samples makes the DNNs vunerable and unreliable.Hence, it is critical to detect dirty samples to improve the quality and realiability of dataset. Existing detectors only focus on detecting poisoned samples or noisy labels, that are often prone to weak generalization when dealing with dirty samples from other domains.In this paper, we find a commonality of various dirty samples is visual-linguistic inconsistency between images and associated labels. To capture the semantic inconsistency between modalities, we propose versatile data cleanser (VDC) leveraging the surpassing capabilities of multimodal large language models (MLLM) in cross-modal alignment and reasoning.It consists of three consecutive modules: the visual question generation module to generate insightful questions about the image; the visual question answering module to acquire the semantics of the visual content by answering the questions with MLLM; followed by the visual answer evaluation module to evaluate the inconsistency.Extensive experiments demonstrate its superior performance and generalization to various categories and types of dirty samples. The code is available at \\url{https://github.com/zihao-ai/vdc}.","url_abs":"https://arxiv.org/abs/2309.16211v2","url_pdf":"https://arxiv.org/pdf/2309.16211v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"vdc-versatile-data-cleanser-for-detecting","repo_url":"https://github.com/zihao-ai/vdc","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"backdoor-attack","task_name":"Backdoor Attack"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"question-generation","task_name":"Question Generation"},{"task_slug":"question-generation","task_name":"Question-Generation"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"cross-modal-alignment","task_name":"cross-modal alignment"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.16211","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.16211"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/zihao-ai/vdc","reach":null}],"summary":{"ran":5,"ran_draft_wrong":3,"unverified":9},"by_repo_kind":{"official":{"samples":17,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"db733d746340ae5a","entry":"ImageTextPair","repo":"zihao-ai/vdc","repo_kind":"official","path":"vdc/cleanser.py","file_url":"https://github.com/zihao-ai/vdc/blob/HEAD/vdc/cleanser.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"db733d746340ae5a"}},{"code_sha256_prefix":"74776e27896ffb46","entry":"QAPair","repo":"zihao-ai/vdc","repo_kind":"official","path":"vdc/cleanser.py","file_url":"https://github.com/zihao-ai/vdc/blob/HEAD/vdc/cleanser.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"74776e27896ffb46"}},{"code_sha256_prefix":"30a82bb00f9fb340","entry":"VDCConfig","repo":"zihao-ai/vdc","repo_kind":"official","path":"vdc/cleanser.py","file_url":"https://github.com/zihao-ai/vdc/blob/HEAD/vdc/cleanser.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30a82bb00f9fb340"}},{"code_sha256_prefix":"703d30e5d1885cc9","entry":"VideoBatchQAPrompt","repo":"zihao-ai/vdc","repo_kind":"official","path":"vdc/cleanser.py","file_url":"https://github.com/zihao-ai/vdc/blob/HEAD/vdc/cleanser.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"703d30e5d1885cc9"}},{"code_sha256_prefix":"8dfe3ce2161c4e7e","entry":"VideoTextPair","repo":"zihao-ai/vdc","repo_kind":"official","path":"vdc/cleanser.py","file_url":"https://github.com/zihao-ai/vdc/blob/HEAD/vdc/cleanser.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8dfe3ce2161c4e7e"}},{"code_sha256_prefix":"b5fa7736e0dfb6df","entry":"encode_image","repo":"zihao-ai/vdc","repo_kind":"official","path":"vdc/cleanser.py","file_url":"https://github.com/zihao-ai/vdc/blob/HEAD/vdc/cleanser.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b5fa7736e0dfb6df"}},{"code_sha256_prefix":"b3a3983654d23cc7","entry":"generate_text","repo":"zihao-ai/vdc","repo_kind":"official","path":"vdc/cleanser.py","file_url":"https://github.com/zihao-ai/vdc/blob/HEAD/vdc/cleanser.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b3a3983654d23cc7"}},{"code_sha256_prefix":"4dca5b7c93dadb0a","entry":"get_mime_type","repo":"zihao-ai/vdc","repo_kind":"official","path":"vdc/cleanser.py","file_url":"https://github.com/zihao-ai/vdc/blob/HEAD/vdc/cleanser.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4dca5b7c93dadb0a"}},{"code_sha256_prefix":"f1f52fedd5594d5b","entry":"BaseDataPair","repo":"zihao-ai/vdc","repo_kind":"official","path":"vdc/cleanser.py","file_url":"https://github.com/zihao-ai/vdc/blob/HEAD/vdc/cleanser.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f1f52fedd5594d5b"}},{"code_sha256_prefix":"40604a123d40ea11","entry":"DataCleanser","repo":"zihao-ai/vdc","repo_kind":"official","path":"vdc/cleanser.py","file_url":"https://github.com/zihao-ai/vdc/blob/HEAD/vdc/cleanser.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"40604a123d40ea11"}},{"code_sha256_prefix":"8fe89a9b67ae22f6","entry":"ImageBatchQAPrompt","repo":"zihao-ai/vdc","repo_kind":"official","path":"vdc/cleanser.py","file_url":"https://github.com/zihao-ai/vdc/blob/HEAD/vdc/cleanser.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8fe89a9b67ae22f6"}},{"code_sha256_prefix":"97a96bf65e202f18","entry":"ImageQAPrompt","repo":"zihao-ai/vdc","repo_kind":"official","path":"vdc/cleanser.py","file_url":"https://github.com/zihao-ai/vdc/blob/HEAD/vdc/cleanser.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"97a96bf65e202f18"}},{"code_sha256_prefix":"a8cf3b67773ddbdb","entry":"ImageQGPrompt","repo":"zihao-ai/vdc","repo_kind":"official","path":"vdc/cleanser.py","file_url":"https://github.com/zihao-ai/vdc/blob/HEAD/vdc/cleanser.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a8cf3b67773ddbdb"}},{"code_sha256_prefix":"3ad0967f34a83055","entry":"VideoQAPrompt","repo":"zihao-ai/vdc","repo_kind":"official","path":"vdc/cleanser.py","file_url":"https://github.com/zihao-ai/vdc/blob/HEAD/vdc/cleanser.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3ad0967f34a83055"}},{"code_sha256_prefix":"87e17a9fd46db8ad","entry":"VideoQGPrompt","repo":"zihao-ai/vdc","repo_kind":"official","path":"vdc/cleanser.py","file_url":"https://github.com/zihao-ai/vdc/blob/HEAD/vdc/cleanser.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"87e17a9fd46db8ad"}},{"code_sha256_prefix":"25ce289971b5fced","entry":"generate_text_with_images","repo":"zihao-ai/vdc","repo_kind":"official","path":"vdc/cleanser.py","file_url":"https://github.com/zihao-ai/vdc/blob/HEAD/vdc/cleanser.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"25ce289971b5fced"}},{"code_sha256_prefix":"b87180285f8c51b8","entry":"generate_text_with_video","repo":"zihao-ai/vdc","repo_kind":"official","path":"vdc/cleanser.py","file_url":"https://github.com/zihao-ai/vdc/blob/HEAD/vdc/cleanser.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b87180285f8c51b8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}