{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/private-attribute-inference-from-images-with","title":"Private Attribute Inference from Images with Vision-Language Models","arxiv_id":"2404.10618","date":"2024-04-16","proceeding":null,"authors":["Batuhan Tömekçe","Mark Vero","Robin Staab","Martin Vechev"],"abstract":"As large language models (LLMs) become ubiquitous in our daily tasks and digital interactions, associated privacy risks are increasingly in focus. While LLM privacy research has primarily focused on the leakage of model training data, it has recently been shown that LLMs can make accurate privacy-infringing inferences from previously unseen texts. With the rise of vision-language models (VLMs), capable of understanding both images and text, a key question is whether this concern transfers to the previously unexplored domain of benign images posted online. To answer this question, we compile an image dataset with human-annotated labels of the image owner's personal attributes. In order to understand the privacy risks posed by VLMs beyond traditional human attribute recognition, our dataset consists of images where the inferable private attributes do not stem from direct depictions of humans. On this dataset, we evaluate 7 state-of-the-art VLMs, finding that they can infer various personal attributes at up to 77.6% accuracy. Concerningly, we observe that accuracy scales with the general capabilities of the models, implying that future models can be misused as stronger inferential adversaries, establishing an imperative for the development of adequate defenses.","url_abs":"https://arxiv.org/abs/2404.10618v2","url_pdf":"https://arxiv.org/pdf/2404.10618v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"private-attribute-inference-from-images-with","repo_url":"https://github.com/eth-sri/privacy-inference-multimodal","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"attribute","task_name":"Attribute"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.10618","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.10618"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/eth-sri/privacy-inference-multimodal","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"ran":1,"ran_honours":2},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a7bee88c1c7fd6a3","entry":"image_parser","repo":"eth-sri/privacy-inference-multimodal","repo_kind":"official","path":"Experimentation/LLaVa-NeXT/run_llava.py","file_url":"https://github.com/eth-sri/privacy-inference-multimodal/blob/HEAD/Experimentation/LLaVa-NeXT/run_llava.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a7bee88c1c7fd6a3"}},{"code_sha256_prefix":"f33e27dbfdb1bcb5","entry":"load_config","repo":"eth-sri/privacy-inference-multimodal","repo_kind":"official","path":"Experimentation/src/configs/config.py","file_url":"https://github.com/eth-sri/privacy-inference-multimodal/blob/HEAD/Experimentation/src/configs/config.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f33e27dbfdb1bcb5"}},{"code_sha256_prefix":"9b3c1cb391672ccb","entry":"load_image","repo":"eth-sri/privacy-inference-multimodal","repo_kind":"official","path":"Experimentation/LLaVa-NeXT/run_llava.py","file_url":"https://github.com/eth-sri/privacy-inference-multimodal/blob/HEAD/Experimentation/LLaVa-NeXT/run_llava.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9b3c1cb391672ccb"}},{"code_sha256_prefix":"d5044bdde33c54c2","entry":"load_images","repo":"eth-sri/privacy-inference-multimodal","repo_kind":"official","path":"Experimentation/LLaVa-NeXT/run_llava.py","file_url":"https://github.com/eth-sri/privacy-inference-multimodal/blob/HEAD/Experimentation/LLaVa-NeXT/run_llava.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d5044bdde33c54c2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}