{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/knowledgesg-privacy-preserving-synthetic-text","title":"KnowledgeSG: Privacy-Preserving Synthetic Text Generation with Knowledge Distillation from Server","arxiv_id":"2410.05725","date":"2024-10-08","proceeding":null,"authors":["Wenhao Wang","Xiaoyu Liang","Rui Ye","Jingyi Chai","Siheng Chen","Yanfeng Wang"],"abstract":"The success of large language models (LLMs) facilitate many parties to fine-tune LLMs on their own private data. However, this practice raises privacy concerns due to the memorization of LLMs. Existing solutions, such as utilizing synthetic data for substitution, struggle to simultaneously improve performance and preserve privacy. They either rely on a local model for generation, resulting in a performance decline, or take advantage of APIs, directly exposing the data to API servers. To address this issue, we propose KnowledgeSG, a novel client-server framework which enhances synthetic data quality and improves model performance while ensuring privacy. We achieve this by learning local knowledge from the private data with differential privacy (DP) and distilling professional knowledge from the server. Additionally, inspired by federated learning, we transmit models rather than data between the client and server to prevent privacy leakage. Extensive experiments in medical and financial domains demonstrate the effectiveness of KnowledgeSG. Our code is now publicly available at https://github.com/wwh0411/KnowledgeSG.","url_abs":"https://arxiv.org/abs/2410.05725v2","url_pdf":"https://arxiv.org/pdf/2410.05725v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"knowledgesg-privacy-preserving-synthetic-text","repo_url":"https://github.com/wwh0411/knowledgesg","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"federated-learning","task_name":"Federated Learning"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"memorization","task_name":"Memorization"},{"task_slug":"privacy-preserving","task_name":"Privacy Preserving"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.05725","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.05725"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/wwh0411/KnowledgeSG","reach":{"status":"ok"}}],"summary":{"ran":5,"unverified":3},"by_repo_kind":{"official":{"samples":8,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"7fedf1b6457b251e","entry":"calculate_fid","repo":"wwh0411/KnowledgeSG","repo_kind":"official","path":"evaluation/similarity.py","file_url":"https://github.com/wwh0411/KnowledgeSG/blob/HEAD/evaluation/similarity.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7fedf1b6457b251e"}},{"code_sha256_prefix":"dd4d470e76fb6338","entry":"map_alpaca","repo":"wwh0411/KnowledgeSG","repo_kind":"official","path":"process_dataset.py","file_url":"https://github.com/wwh0411/KnowledgeSG/blob/HEAD/process_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dd4d470e76fb6338"}},{"code_sha256_prefix":"4cc8e4f7d37fd893","entry":"map_chatbot","repo":"wwh0411/KnowledgeSG","repo_kind":"official","path":"process_dataset.py","file_url":"https://github.com/wwh0411/KnowledgeSG/blob/HEAD/process_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4cc8e4f7d37fd893"}},{"code_sha256_prefix":"fdb6c78b0700aabc","entry":"map_medical_chatbot","repo":"wwh0411/KnowledgeSG","repo_kind":"official","path":"process_dataset.py","file_url":"https://github.com/wwh0411/KnowledgeSG/blob/HEAD/process_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fdb6c78b0700aabc"}},{"code_sha256_prefix":"f3096dabb60bc3e1","entry":"partition_dataset","repo":"wwh0411/KnowledgeSG","repo_kind":"official","path":"dataset/partition.py","file_url":"https://github.com/wwh0411/KnowledgeSG/blob/HEAD/dataset/partition.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f3096dabb60bc3e1"}},{"code_sha256_prefix":"ee221f574196b9ba","entry":"get_dataset_embed","repo":"wwh0411/KnowledgeSG","repo_kind":"official","path":"evaluation/similarity.py","file_url":"https://github.com/wwh0411/KnowledgeSG/blob/HEAD/evaluation/similarity.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ee221f574196b9ba"}},{"code_sha256_prefix":"48b1bd5978f1385e","entry":"get_perplexity_and_embedding_part_text","repo":"wwh0411/KnowledgeSG","repo_kind":"official","path":"evaluation/ifd_gene.py","file_url":"https://github.com/wwh0411/KnowledgeSG/blob/HEAD/evaluation/ifd_gene.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"48b1bd5978f1385e"}},{"code_sha256_prefix":"bafe0f8c61e7e6a9","entry":"get_perplexity_and_embedding_whole_text","repo":"wwh0411/KnowledgeSG","repo_kind":"official","path":"evaluation/ifd_gene.py","file_url":"https://github.com/wwh0411/KnowledgeSG/blob/HEAD/evaluation/ifd_gene.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bafe0f8c61e7e6a9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}