{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/membership-inference-attacks-against-large","title":"Membership Inference Attacks against Large Vision-Language Models","arxiv_id":"2411.02902","date":"2024-11-05","proceeding":null,"authors":["Zhan Li","Yongtao Wu","Yihang Chen","Francesco Tonin","Elias Abad Rocamora","Volkan Cevher"],"abstract":"Large vision-language models (VLLMs) exhibit promising capabilities for processing multi-modal tasks across various application scenarios. However, their emergence also raises significant data security concerns, given the potential inclusion of sensitive information, such as private photos and medical records, in their training datasets. Detecting inappropriately used data in VLLMs remains a critical and unresolved issue, mainly due to the lack of standardized datasets and suitable methodologies. In this study, we introduce the first membership inference attack (MIA) benchmark tailored for various VLLMs to facilitate training data detection. Then, we propose a novel MIA pipeline specifically designed for token-level image detection. Lastly, we present a new metric called MaxR\\'enyi-K%, which is based on the confidence of the model output and applies to both text and image data. We believe that our work can deepen the understanding and methodology of MIAs in the context of VLLMs. Our code and datasets are available at https://github.com/LIONS-EPFL/VL-MIA.","url_abs":"https://arxiv.org/abs/2411.02902v1","url_pdf":"https://arxiv.org/pdf/2411.02902v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"membership-inference-attacks-against-large","repo_url":"https://github.com/lions-epfl/vl-mia","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"inference-attack","task_name":"Inference Attack"},{"task_slug":"membership-inference-attack","task_name":"Membership Inference Attack"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.02902","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.02902"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/LIONS-EPFL/VL-MIA","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lions-epfl/vl-mia","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":1,"unverified":9},"by_repo_kind":{"official":{"samples":10,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ddc4f87a08fc5e64","entry":"get_meta_metrics","repo":"lions-epfl/vl-mia","repo_kind":"official","path":"metric_util.py","file_url":"https://github.com/lions-epfl/vl-mia/blob/HEAD/metric_util.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ddc4f87a08fc5e64"}},{"code_sha256_prefix":"35ad6ca5945fd1a4","entry":"get_img_metric","repo":"LIONS-EPFL/VL-MIA","repo_kind":"official","path":"metric_util.py","file_url":"https://github.com/LIONS-EPFL/VL-MIA/blob/HEAD/metric_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"35ad6ca5945fd1a4"}},{"code_sha256_prefix":"53113dc385f93c57","entry":"get_text_metric","repo":"LIONS-EPFL/VL-MIA","repo_kind":"official","path":"metric_util.py","file_url":"https://github.com/LIONS-EPFL/VL-MIA/blob/HEAD/metric_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"53113dc385f93c57"}},{"code_sha256_prefix":"af3c773aaee1147e","entry":"kl_divergence","repo":"LIONS-EPFL/VL-MIA","repo_kind":"official","path":"metric_util.py","file_url":"https://github.com/LIONS-EPFL/VL-MIA/blob/HEAD/metric_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"af3c773aaee1147e"}},{"code_sha256_prefix":"4552e857b0cf7ef7","entry":"load_image","repo":"LIONS-EPFL/VL-MIA","repo_kind":"official","path":"run_with_text.py","file_url":"https://github.com/LIONS-EPFL/VL-MIA/blob/HEAD/run_with_text.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4552e857b0cf7ef7"}},{"code_sha256_prefix":"016ce1ddc61eea2c","entry":"load_image","repo":"LIONS-EPFL/VL-MIA","repo_kind":"official","path":"run_with_img.py","file_url":"https://github.com/LIONS-EPFL/VL-MIA/blob/HEAD/run_with_img.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"016ce1ddc61eea2c"}},{"code_sha256_prefix":"99d66c86efac69ff","entry":"load_images","repo":"LIONS-EPFL/VL-MIA","repo_kind":"official","path":"run_with_text.py","file_url":"https://github.com/LIONS-EPFL/VL-MIA/blob/HEAD/run_with_text.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"99d66c86efac69ff"}},{"code_sha256_prefix":"a37042e5de9c8a99","entry":"load_images","repo":"LIONS-EPFL/VL-MIA","repo_kind":"official","path":"run_with_img.py","file_url":"https://github.com/LIONS-EPFL/VL-MIA/blob/HEAD/run_with_img.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a37042e5de9c8a99"}},{"code_sha256_prefix":"833f0b5d618afaf0","entry":"logits_forward","repo":"LIONS-EPFL/VL-MIA","repo_kind":"official","path":"llama_adapter_v21/run_with_text.py","file_url":"https://github.com/LIONS-EPFL/VL-MIA/blob/HEAD/llama_adapter_v21/run_with_text.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"833f0b5d618afaf0"}},{"code_sha256_prefix":"208e33dc4d741d08","entry":"shuffle_sentence","repo":"lions-epfl/vl-mia","repo_kind":"official","path":"run_with_text.py","file_url":"https://github.com/lions-epfl/vl-mia/blob/HEAD/run_with_text.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"208e33dc4d741d08"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}