{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sampling-based-pseudo-likelihood-for","title":"Sampling-based Pseudo-Likelihood for Membership Inference Attacks","arxiv_id":"2404.11262","date":"2024-04-17","proceeding":null,"authors":["Masahiro Kaneko","Youmi Ma","Yuki Wata","Naoaki Okazaki"],"abstract":"Large Language Models (LLMs) are trained on large-scale web data, which makes it difficult to grasp the contribution of each text. This poses the risk of leaking inappropriate data such as benchmarks, personal information, and copyrighted texts in the training data. Membership Inference Attacks (MIA), which determine whether a given text is included in the model's training data, have been attracting attention. Previous studies of MIAs revealed that likelihood-based classification is effective for detecting leaks in LLMs. However, the existing methods cannot be applied to some proprietary models like ChatGPT or Claude 3 because the likelihood is unavailable to the user. In this study, we propose a Sampling-based Pseudo-Likelihood (\\textbf{SPL}) method for MIA (\\textbf{SaMIA}) that calculates SPL using only the text generated by an LLM to detect leaks. The SaMIA treats the target text as the reference text and multiple outputs from the LLM as text samples, calculates the degree of $n$-gram match as SPL, and determines the membership of the text in the training data. Even without likelihoods, SaMIA performed on par with existing likelihood-based methods.","url_abs":"https://arxiv.org/abs/2404.11262v1","url_pdf":"https://arxiv.org/pdf/2404.11262v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sampling-based-pseudo-likelihood-for","repo_url":"https://github.com/nlp-titech/samia","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"sps","method_name":"SPL"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2404.11262","atlas_url":"https://app.syntology.ai/?focus=2404.11262","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.11262"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nlp-titech/samia","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0ff086b8ebd54874","entry":"generate_text","repo":"nlp-titech/samia","repo_kind":"official","path":"src/sampling.py","file_url":"https://github.com/nlp-titech/samia/blob/HEAD/src/sampling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0ff086b8ebd54874"}},{"code_sha256_prefix":"506146b248d41ae9","entry":"get_suffix","repo":"nlp-titech/samia","repo_kind":"official","path":"src/eval_samia.py","file_url":"https://github.com/nlp-titech/samia/blob/HEAD/src/eval_samia.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"506146b248d41ae9"}},{"code_sha256_prefix":"b28fb51299e949da","entry":"load_jsonl","repo":"nlp-titech/samia","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/nlp-titech/samia/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b28fb51299e949da"}},{"code_sha256_prefix":"5572f146d01337d4","entry":"load_model","repo":"nlp-titech/samia","repo_kind":"official","path":"src/model_loader.py","file_url":"https://github.com/nlp-titech/samia/blob/HEAD/src/model_loader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5572f146d01337d4"}},{"code_sha256_prefix":"6b2373f2ab62192c","entry":"ngrams","repo":"nlp-titech/samia","repo_kind":"official","path":"src/eval_samia.py","file_url":"https://github.com/nlp-titech/samia/blob/HEAD/src/eval_samia.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6b2373f2ab62192c"}},{"code_sha256_prefix":"1c31724ab6c583c2","entry":"rouge_n","repo":"nlp-titech/samia","repo_kind":"official","path":"src/eval_samia.py","file_url":"https://github.com/nlp-titech/samia/blob/HEAD/src/eval_samia.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1c31724ab6c583c2"}},{"code_sha256_prefix":"85d6909ba1d0cb83","entry":"get_prefix","repo":"nlp-titech/samia","repo_kind":"official","path":"src/sampling.py","file_url":"https://github.com/nlp-titech/samia/blob/HEAD/src/sampling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"85d6909ba1d0cb83"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}