{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-private-neural-language-modeling","title":"Learning Private Neural Language Modeling with Attentive Aggregation","arxiv_id":"1812.07108","date":"2018-12-17","proceeding":null,"authors":["Shaoxiong Ji","Shirui Pan","Guodong Long","Xue Li","Jing Jiang","Zi Huang"],"abstract":"Mobile keyboard suggestion is typically regarded as a word-level language\nmodeling problem. Centralized machine learning technique requires massive user\ndata collected to train on, which may impose privacy concerns for sensitive\npersonal typing data of users. Federated learning (FL) provides a promising\napproach to learning private language modeling for intelligent personalized\nkeyboard suggestion by training models in distributed clients rather than\ntraining in a central server. To obtain a global model for prediction, existing\nFL algorithms simply average the client models and ignore the importance of\neach client during model aggregation. Furthermore, there is no optimization for\nlearning a well-generalized global model on the central server. To solve these\nproblems, we propose a novel model aggregation with the attention mechanism\nconsidering the contribution of clients models to the global model, together\nwith an optimization technique during server aggregation. Our proposed\nattentive aggregation method minimizes the weighted distance between the server\nmodel and client models through iterative parameters updating while attends the\ndistance between the server model and client models. Through experiments on two\npopular language modeling datasets and a social media dataset, our proposed\nmethod outperforms its counterparts in terms of perplexity and communication\ncost in most settings of comparison.","url_abs":"http://arxiv.org/abs/1812.07108v2","url_pdf":"http://arxiv.org/pdf/1812.07108v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-private-neural-language-modeling","repo_url":"https://github.com/shaoxiongji/fed-att","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"learning-private-neural-language-modeling","repo_url":"https://github.com/LeoSerena/MSThesis","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"learning-private-neural-language-modeling","repo_url":"https://github.com/shaoxiongji/federated-learning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"learning-private-neural-language-modeling","repo_url":"https://github.com/wuyenlin/federated-learning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"GPL-3.0"}}],"tasks":[{"task_slug":"federated-learning","task_name":"Federated Learning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1812.07108","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1812.07108"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shaoxiongji/federated-learning","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wuyenlin/federated-learning","reach":{"status":"ok","spdx":"GPL-3.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/LeoSerena/MSThesis","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shaoxiongji/fed-att","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"ran":1,"unverified":5},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1},"listed":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f9964d719c5cb8f4","entry":"FedAvg","repo":"shaoxiongji/federated-learning","repo_kind":"listed","path":"models/Fed.py","file_url":"https://github.com/shaoxiongji/federated-learning/blob/HEAD/models/Fed.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f9964d719c5cb8f4"}},{"code_sha256_prefix":"c29db2c2b5697607","entry":"cifar_iid","repo":"shaoxiongji/federated-learning","repo_kind":"listed","path":"utils/sampling.py","file_url":"https://github.com/shaoxiongji/federated-learning/blob/HEAD/utils/sampling.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c29db2c2b5697607"}},{"code_sha256_prefix":"d9107114749c3e49","entry":"mnist_iid","repo":"shaoxiongji/federated-learning","repo_kind":"listed","path":"utils/sampling.py","file_url":"https://github.com/shaoxiongji/federated-learning/blob/HEAD/utils/sampling.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d9107114749c3e49"}},{"code_sha256_prefix":"dad97f90c5c53b3c","entry":"aggregate_att","repo":"shaoxiongji/fed-att","repo_kind":"official","path":"src/agg/aggregate.py","file_url":"https://github.com/shaoxiongji/fed-att/blob/HEAD/src/agg/aggregate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dad97f90c5c53b3c"}},{"code_sha256_prefix":"7a711a11ae243b32","entry":"average_weights","repo":"shaoxiongji/fed-att","repo_kind":"official","path":"src/agg/avg.py","file_url":"https://github.com/shaoxiongji/fed-att/blob/HEAD/src/agg/avg.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7a711a11ae243b32"}},{"code_sha256_prefix":"37a8837b66a64988","entry":"mnist_noniid","repo":"shaoxiongji/federated-learning","repo_kind":"listed","path":"utils/sampling.py","file_url":"https://github.com/shaoxiongji/federated-learning/blob/HEAD/utils/sampling.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"37a8837b66a64988"}},{"code_sha256_prefix":"bc9519946c0aea20","entry":"partition","repo":"shaoxiongji/fed-att","repo_kind":"official","path":"src/utils/sampling.py","file_url":"https://github.com/shaoxiongji/fed-att/blob/HEAD/src/utils/sampling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc9519946c0aea20"}},{"code_sha256_prefix":"a15078a8d47f36f1","entry":"test","repo":"shaoxiongji/federated-learning","repo_kind":"listed","path":"main_nn.py","file_url":"https://github.com/shaoxiongji/federated-learning/blob/HEAD/main_nn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a15078a8d47f36f1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}