{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/canary-in-a-coalmine-better-membership","title":"Canary in a Coalmine: Better Membership Inference with Ensembled Adversarial Queries","arxiv_id":"2210.10750","date":"2022-10-19","proceeding":null,"authors":["Yuxin Wen","Arpit Bansal","Hamid Kazemi","Eitan Borgnia","Micah Goldblum","Jonas Geiping","Tom Goldstein"],"abstract":"As industrial applications are increasingly automated by machine learning models, enforcing personal data ownership and intellectual property rights requires tracing training data back to their rightful owners. Membership inference algorithms approach this problem by using statistical techniques to discern whether a target sample was included in a model's training set. However, existing methods only utilize the unaltered target sample or simple augmentations of the target to compute statistics. Such a sparse sampling of the model's behavior carries little information, leading to poor inference capabilities. In this work, we use adversarial tools to directly optimize for queries that are discriminative and diverse. Our improvements achieve significantly more accurate membership inference than existing methods, especially in offline scenarios and in the low false-positive regime which is critical in legal settings. Code is available at https://github.com/YuxinWenRick/canary-in-a-coalmine.","url_abs":"https://arxiv.org/abs/2210.10750v2","url_pdf":"https://arxiv.org/pdf/2210.10750v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"canary-in-a-coalmine-better-membership","repo_url":"https://github.com/yuxinwenrick/canary-in-a-coalmine","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.10750","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.10750"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/YuxinWenRick/canary-in-a-coalmine","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yuxinwenrick/canary-in-a-coalmine","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"ran_honours":1,"unverified":3},"by_repo_kind":{"official":{"samples":5,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"504964ff2fd0941b","entry":"generate_class_dict","repo":"yuxinwenrick/canary-in-a-coalmine","repo_kind":"official","path":"gen_canary.py","file_url":"https://github.com/yuxinwenrick/canary-in-a-coalmine/blob/HEAD/gen_canary.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"504964ff2fd0941b"}},{"code_sha256_prefix":"2efe5bad2dfe6de8","entry":"generate_close_imgs","repo":"yuxinwenrick/canary-in-a-coalmine","repo_kind":"official","path":"gen_canary.py","file_url":"https://github.com/yuxinwenrick/canary-in-a-coalmine/blob/HEAD/gen_canary.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2efe5bad2dfe6de8"}},{"code_sha256_prefix":"067a3278347d245c","entry":"generate_canary","repo":"yuxinwenrick/canary-in-a-coalmine","repo_kind":"official","path":"gen_canary.py","file_url":"https://github.com/yuxinwenrick/canary-in-a-coalmine/blob/HEAD/gen_canary.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"067a3278347d245c"}},{"code_sha256_prefix":"eee39fe29fb52301","entry":"generate_canary_one_shot","repo":"yuxinwenrick/canary-in-a-coalmine","repo_kind":"official","path":"gen_canary.py","file_url":"https://github.com/yuxinwenrick/canary-in-a-coalmine/blob/HEAD/gen_canary.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eee39fe29fb52301"}},{"code_sha256_prefix":"cb4df7cc368ea339","entry":"initialize_poison","repo":"yuxinwenrick/canary-in-a-coalmine","repo_kind":"official","path":"gen_canary.py","file_url":"https://github.com/yuxinwenrick/canary-in-a-coalmine/blob/HEAD/gen_canary.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cb4df7cc368ea339"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}