{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/docmia-document-level-membership-inference","title":"DocMIA: Document-Level Membership Inference Attacks against DocVQA Models","arxiv_id":"2502.03692","date":"2025-02-06","proceeding":null,"authors":["Khanh Nguyen","Raouf Kerkouche","Mario Fritz","Dimosthenis Karatzas"],"abstract":"Document Visual Question Answering (DocVQA) has introduced a new paradigm for end-to-end document understanding, and quickly became one of the standard benchmarks for multimodal LLMs. Automating document processing workflows, driven by DocVQA models, presents significant potential for many business sectors. However, documents tend to contain highly sensitive information, raising concerns about privacy risks associated with training such DocVQA models. One significant privacy vulnerability, exploited by the membership inference attack, is the possibility for an adversary to determine if a particular record was part of the model's training data. In this paper, we introduce two novel membership inference attacks tailored specifically to DocVQA models. These attacks are designed for two different adversarial scenarios: a white-box setting, where the attacker has full access to the model architecture and parameters, and a black-box setting, where only the model's outputs are available. Notably, our attacks assume the adversary lacks access to auxiliary datasets, which is more realistic in practice but also more challenging. Our unsupervised methods outperform existing state-of-the-art membership inference attacks across a variety of DocVQA models and datasets, demonstrating their effectiveness and highlighting the privacy risks in this domain.","url_abs":"https://arxiv.org/abs/2502.03692v1","url_pdf":"https://arxiv.org/pdf/2502.03692v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"docmia-document-level-membership-inference","repo_url":"https://github.com/khanhnguyen21006/mia_docvqa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"inference-attack","task_name":"Inference Attack"},{"task_slug":"membership-inference-attack","task_name":"Membership Inference Attack"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"document-understanding","task_name":"document understanding"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2502.03692","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.03692"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/khanhnguyen21006/mia_docvqa","reach":null}],"summary":{"ran_honours":1,"ran_draft_wrong":1,"ran_fixture":1,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"90d76cb2e8fbdca2","entry":"get_generative_confidence","repo":"khanhnguyen21006/mia_docvqa","repo_kind":"official","path":"modules/pfl/models/_model_utils.py","file_url":"https://github.com/khanhnguyen21006/mia_docvqa/blob/HEAD/modules/pfl/models/_model_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"90d76cb2e8fbdca2"}},{"code_sha256_prefix":"25b5c26383d545b4","entry":"get_state_dict","repo":"khanhnguyen21006/mia_docvqa","repo_kind":"official","path":"run_white_box.py","file_url":"https://github.com/khanhnguyen21006/mia_docvqa/blob/HEAD/run_white_box.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"25b5c26383d545b4"}},{"code_sha256_prefix":"5416a88b7a293960","entry":"sample_indices","repo":"khanhnguyen21006/mia_docvqa","repo_kind":"official","path":"run_white_box.py","file_url":"https://github.com/khanhnguyen21006/mia_docvqa/blob/HEAD/run_white_box.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5416a88b7a293960"}},{"code_sha256_prefix":"eca27ace7e9d1f45","entry":"query_blackbox","repo":"khanhnguyen21006/mia_docvqa","repo_kind":"official","path":"run_black_box.py","file_url":"https://github.com/khanhnguyen21006/mia_docvqa/blob/HEAD/run_black_box.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eca27ace7e9d1f45"}},{"code_sha256_prefix":"f2ca77256f5b5f3c","entry":"select_finetune_layer","repo":"khanhnguyen21006/mia_docvqa","repo_kind":"official","path":"run_white_box.py","file_url":"https://github.com/khanhnguyen21006/mia_docvqa/blob/HEAD/run_white_box.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f2ca77256f5b5f3c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}