{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/federated-learning-with-differential-privacy","title":"Enabling Differentially Private Federated Learning for Speech Recognition: Benchmarks, Adaptive Optimizers and Gradient Clipping","arxiv_id":"2310.00098","date":"2023-09-29","proceeding":null,"authors":["Martin Pelikan","Sheikh Shams Azam","Vitaly Feldman","Jan \"Honza\" Silovsky","Kunal Talwar","Christopher G. Brinton","Tatiana Likhomanenko"],"abstract":"While federated learning (FL) and differential privacy (DP) have been extensively studied, their application to automatic speech recognition (ASR) remains largely unexplored due to the challenges in training large transformer models. Specifically, large models further exacerbate issues in FL as they are particularly susceptible to gradient heterogeneity across layers, unlike the relatively uniform gradient behavior observed in shallow models. As a result, prior works struggle to converge with standard optimization techniques, even in the absence of DP mechanisms. To the best of our knowledge, no existing work establishes a competitive, practical recipe for FL with DP in the context of ASR. To address this gap, we establish \\textbf{the first benchmark for FL with DP in end-to-end ASR}. Our approach centers on per-layer clipping and layer-wise gradient normalization: theoretical analysis reveals that these techniques together mitigate clipping bias and gradient heterogeneity across layers in deeper models. Consistent with these theoretical insights, our empirical results show that FL with DP is viable under strong privacy guarantees, provided a population of at least several million users. Specifically, we achieve user-level (7.2, $10^{-9}$)-DP (resp. (4.5, $10^{-9}$)-DP) with only a 1.3% (resp. 4.6%) absolute drop in word error rate when extrapolating to high (resp. low) population scales for FL with DP in ASR. Although our experiments focus on ASR, the underlying principles we uncover - particularly those concerning gradient heterogeneity and layer-wise gradient normalization - offer broader guidance for designing scalable, privacy-preserving FL algorithms for large models across domains.","url_abs":"https://arxiv.org/abs/2310.00098v2","url_pdf":"https://arxiv.org/pdf/2310.00098v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"automatic-speech-recognition-2","task_name":"Automatic Speech Recognition"},{"task_slug":"automatic-speech-recognition","task_name":"Automatic Speech Recognition (ASR)"},{"task_slug":"federated-learning","task_name":"Federated Learning"},{"task_slug":"privacy-preserving","task_name":"Privacy Preserving"},{"task_slug":"speech-recognition","task_name":"Speech Recognition"},{"task_slug":"speech-recognition-1","task_name":"speech-recognition"}],"methods":[{"method_slug":"aware","method_name":"AWARE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.00098","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.00098"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/apple/ml-pfl4asr","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":7,"unverified":4},"by_repo_kind":{"found_in_text":{"samples":11,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"b9e91f6edc56434d","entry":"check_clients_in_batch","repo":"apple/ml-pfl4asr","repo_kind":"found_in_text","path":"pfl4asr/pfl4asr/pfl/data.py","file_url":"https://github.com/apple/ml-pfl4asr/blob/HEAD/pfl4asr/pfl4asr/pfl/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b9e91f6edc56434d"}},{"code_sha256_prefix":"cb75b689ebb90229","entry":"init_model","repo":"apple/ml-pfl4asr","repo_kind":"found_in_text","path":"pfl4asr/pfl4asr/utils/model.py","file_url":"https://github.com/apple/ml-pfl4asr/blob/HEAD/pfl4asr/pfl4asr/utils/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"cb75b689ebb90229"}},{"code_sha256_prefix":"f138f32036ff8abf","entry":"length_masked_normalize2d","repo":"apple/ml-pfl4asr","repo_kind":"found_in_text","path":"pfl4asr/pfl4asr/modules/functions.py","file_url":"https://github.com/apple/ml-pfl4asr/blob/HEAD/pfl4asr/pfl4asr/modules/functions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f138f32036ff8abf"}},{"code_sha256_prefix":"cf403abf60cc696f","entry":"length_to_mask","repo":"apple/ml-pfl4asr","repo_kind":"found_in_text","path":"pfl4asr/pfl4asr/modules/functions.py","file_url":"https://github.com/apple/ml-pfl4asr/blob/HEAD/pfl4asr/pfl4asr/modules/functions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"cf403abf60cc696f"}},{"code_sha256_prefix":"44193dd0473ba550","entry":"load_model","repo":"apple/ml-pfl4asr","repo_kind":"found_in_text","path":"pfl4asr/pfl4asr/utils/model.py","file_url":"https://github.com/apple/ml-pfl4asr/blob/HEAD/pfl4asr/pfl4asr/utils/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"44193dd0473ba550"}},{"code_sha256_prefix":"2a08d8fe48e4efbf","entry":"masked_mean2d","repo":"apple/ml-pfl4asr","repo_kind":"found_in_text","path":"pfl4asr/pfl4asr/modules/functions.py","file_url":"https://github.com/apple/ml-pfl4asr/blob/HEAD/pfl4asr/pfl4asr/modules/functions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2a08d8fe48e4efbf"}},{"code_sha256_prefix":"6f75d8570b3be853","entry":"scale_by_rss_my","repo":"apple/ml-pfl4asr","repo_kind":"found_in_text","path":"pfl4asr/pfl4asr/utils/adagrad.py","file_url":"https://github.com/apple/ml-pfl4asr/blob/HEAD/pfl4asr/pfl4asr/utils/adagrad.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6f75d8570b3be853"}},{"code_sha256_prefix":"210c580dba95921c","entry":"adagrad","repo":"apple/ml-pfl4asr","repo_kind":"found_in_text","path":"pfl4asr/pfl4asr/utils/adagrad.py","file_url":"https://github.com/apple/ml-pfl4asr/blob/HEAD/pfl4asr/pfl4asr/utils/adagrad.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"210c580dba95921c"}},{"code_sha256_prefix":"0bac200d943c752b","entry":"all_reduce","repo":"apple/ml-pfl4asr","repo_kind":"found_in_text","path":"pfl4asr/pfl4asr/pfl/server.py","file_url":"https://github.com/apple/ml-pfl4asr/blob/HEAD/pfl4asr/pfl4asr/pfl/server.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0bac200d943c752b"}},{"code_sha256_prefix":"74b905b1821cbbac","entry":"generate_optim_step","repo":"apple/ml-pfl4asr","repo_kind":"found_in_text","path":"pfl4asr/pfl4asr/pfl/server.py","file_url":"https://github.com/apple/ml-pfl4asr/blob/HEAD/pfl4asr/pfl4asr/pfl/server.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"74b905b1821cbbac"}},{"code_sha256_prefix":"7e9b49d4823ed0aa","entry":"get_next_sample_on_device","repo":"apple/ml-pfl4asr","repo_kind":"found_in_text","path":"pfl4asr/pfl4asr/utils/data.py","file_url":"https://github.com/apple/ml-pfl4asr/blob/HEAD/pfl4asr/pfl4asr/utils/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7e9b49d4823ed0aa"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}