{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/privaci-bench-evaluating-privacy-with","title":"PrivaCI-Bench: Evaluating Privacy with Contextual Integrity and Legal Compliance","arxiv_id":"2502.17041","date":"2025-02-24","proceeding":null,"authors":["Haoran Li","Wenbin Hu","Huihao Jing","Yulin Chen","Qi Hu","Sirui Han","Tianshu Chu","Peizhao Hu","Yangqiu Song"],"abstract":"Recent advancements in generative large language models (LLMs) have enabled wider applicability, accessibility, and flexibility. However, their reliability and trustworthiness are still in doubt, especially for concerns regarding individuals' data privacy. Great efforts have been made on privacy by building various evaluation benchmarks to study LLMs' privacy awareness and robustness from their generated outputs to their hidden representations. Unfortunately, most of these works adopt a narrow formulation of privacy and only investigate personally identifiable information (PII). In this paper, we follow the merit of the Contextual Integrity (CI) theory, which posits that privacy evaluation should not only cover the transmitted attributes but also encompass the whole relevant social context through private information flows. We present PrivaCI-Bench, a comprehensive contextual privacy evaluation benchmark targeted at legal compliance to cover well-annotated privacy and safety regulations, real court cases, privacy policies, and synthetic data built from the official toolkit to study LLMs' privacy and safety compliance. We evaluate the latest LLMs, including the recent reasoner models QwQ-32B and Deepseek R1. Our experimental results suggest that though LLMs can effectively capture key CI parameters inside a given context, they still require further advancements for privacy compliance.","url_abs":"https://arxiv.org/abs/2502.17041v1","url_pdf":"https://arxiv.org/pdf/2502.17041v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"privaci-bench-evaluating-privacy-with","repo_url":"https://github.com/HKUST-KnowComp/PrivaCI-Bench","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"adopt","method_name":"ADOPT"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2502.17041","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.17041"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/HKUST-KnowComp/PrivaCI-Bench","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":5},"by_repo_kind":{"official":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"93ac6f96a03989e5","entry":"KB_to_dict","repo":"HKUST-KnowComp/PrivaCI-Bench","repo_kind":"official","path":"search_content_for_answer.py","file_url":"https://github.com/HKUST-KnowComp/PrivaCI-Bench/blob/HEAD/search_content_for_answer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"93ac6f96a03989e5"}},{"code_sha256_prefix":"ce050e92179f30cc","entry":"list_intersection","repo":"HKUST-KnowComp/PrivaCI-Bench","repo_kind":"official","path":"utils.py","file_url":"https://github.com/HKUST-KnowComp/PrivaCI-Bench/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ce050e92179f30cc"}},{"code_sha256_prefix":"e5293b13768b7fbd","entry":"load_local_HF_dataset","repo":"HKUST-KnowComp/PrivaCI-Bench","repo_kind":"official","path":"dataset_loader.py","file_url":"https://github.com/HKUST-KnowComp/PrivaCI-Bench/blob/HEAD/dataset_loader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e5293b13768b7fbd"}},{"code_sha256_prefix":"58431336c8d2ad48","entry":"parse_log","repo":"HKUST-KnowComp/PrivaCI-Bench","repo_kind":"official","path":"search_content_for_answer.py","file_url":"https://github.com/HKUST-KnowComp/PrivaCI-Bench/blob/HEAD/search_content_for_answer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"58431336c8d2ad48"}},{"code_sha256_prefix":"b72457ef27876481","entry":"read_events","repo":"HKUST-KnowComp/PrivaCI-Bench","repo_kind":"official","path":"utils.py","file_url":"https://github.com/HKUST-KnowComp/PrivaCI-Bench/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b72457ef27876481"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}