{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/privacy-preserving-data-deduplication-for","title":"Privacy-Preserving Data Deduplication for Enhancing Federated Learning of Language Models (Extended Version)","arxiv_id":"2407.08152","date":"2024-07-11","proceeding":null,"authors":["Aydin Abadi","Vishnu Asutosh Dasu","Sumanta Sarkar"],"abstract":"Deduplication is a vital preprocessing step that enhances machine learning model performance and saves training time and energy. However, enhancing federated learning through deduplication poses challenges, especially regarding scalability and potential privacy violations if deduplication involves sharing all clients' data. In this paper, we address the problem of deduplication in a federated setup by introducing a pioneering protocol, Efficient Privacy-Preserving Multi-Party Deduplication (EP-MPD). It efficiently removes duplicates from multiple clients' datasets without compromising data privacy. EP-MPD is constructed in a modular fashion, utilizing two novel variants of the Private Set Intersection protocol. Our extensive experiments demonstrate the significant benefits of deduplication in federated learning of large language models. For instance, we observe up to 19.62\\% improvement in perplexity and up to 27.95\\% reduction in running time while varying the duplication level between 10\\% and 30\\%. EP-MPD effectively balances privacy and performance in federated learning, making it a valuable solution for large-scale applications.","url_abs":"https://arxiv.org/abs/2407.08152v2","url_pdf":"https://arxiv.org/pdf/2407.08152v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"privacy-preserving-data-deduplication-for","repo_url":"https://github.com/vdasu/deduplication","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"federated-learning","task_name":"Federated Learning"},{"task_slug":"privacy-preserving","task_name":"Privacy Preserving"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.08152","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.08152"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vdasu/deduplication","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4c58842a2ab4e7ab","entry":"get_client_times","repo":"vdasu/deduplication","repo_kind":"official","path":"EP_MPD/run_paper/plot_client.py","file_url":"https://github.com/vdasu/deduplication/blob/HEAD/EP_MPD/run_paper/plot_client.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4c58842a2ab4e7ab"}},{"code_sha256_prefix":"bdb82802ca761ebd","entry":"get_tp_times","repo":"vdasu/deduplication","repo_kind":"official","path":"EP_MPD/run_paper/plot_tee.py","file_url":"https://github.com/vdasu/deduplication/blob/HEAD/EP_MPD/run_paper/plot_tee.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bdb82802ca761ebd"}},{"code_sha256_prefix":"1b1b901833a96775","entry":"imdb_to_list","repo":"vdasu/deduplication","repo_kind":"official","path":"FL/utils.py","file_url":"https://github.com/vdasu/deduplication/blob/HEAD/FL/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1b1b901833a96775"}},{"code_sha256_prefix":"77d3828734c33c11","entry":"jokes_to_list","repo":"vdasu/deduplication","repo_kind":"official","path":"FL/utils.py","file_url":"https://github.com/vdasu/deduplication/blob/HEAD/FL/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"77d3828734c33c11"}},{"code_sha256_prefix":"c4ab682c928cf63a","entry":"rotten_to_list","repo":"vdasu/deduplication","repo_kind":"official","path":"FL/utils.py","file_url":"https://github.com/vdasu/deduplication/blob/HEAD/FL/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c4ab682c928cf63a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}