{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/preservation-of-the-global-knowledge-by-not","title":"Preservation of the Global Knowledge by Not-True Distillation in Federated Learning","arxiv_id":"2106.03097","date":"2021-06-06","proceeding":null,"authors":["Gihun Lee","Minchan Jeong","Yongjin Shin","Sangmin Bae","Se-Young Yun"],"abstract":"In federated learning, a strong global model is collaboratively learned by aggregating clients' locally trained models. Although this precludes the need to access clients' data directly, the global model's convergence often suffers from data heterogeneity. This study starts from an analogy to continual learning and suggests that forgetting could be the bottleneck of federated learning. We observe that the global model forgets the knowledge from previous rounds, and the local training induces forgetting the knowledge outside of the local distribution. Based on our findings, we hypothesize that tackling down forgetting will relieve the data heterogeneity problem. To this end, we propose a novel and effective algorithm, Federated Not-True Distillation (FedNTD), which preserves the global perspective on locally available data only for the not-true classes. In the experiments, FedNTD shows state-of-the-art performance on various setups without compromising data privacy or incurring additional communication costs.","url_abs":"https://arxiv.org/abs/2106.03097v5","url_pdf":"https://arxiv.org/pdf/2106.03097v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"preservation-of-the-global-knowledge-by-not","repo_url":"https://github.com/Lee-Gihun/FedNTD","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"preservation-of-the-global-knowledge-by-not","repo_url":"https://github.com/thejungwon/gc-fed","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"preservation-of-the-global-knowledge-by-not","repo_url":"https://github.com/TsingZ0/PFL-Non-IID","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"continual-learning","task_name":"Continual Learning"},{"task_slug":"federated-learning","task_name":"Federated Learning"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"self-knowledge-distillation","task_name":"Self-Knowledge Distillation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2106.03097","atlas_url":"https://app.syntology.ai/?focus=2106.03097","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.03097"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Lee-Gihun/FedNTD","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thejungwon/gc-fed","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/TsingZ0/PFL-Non-IID","reach":null}],"summary":{"ran":3,"ran_draft_wrong":1,"unverified":5},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":8,"ran":3,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"b2630c53b756f58c","entry":"Client","repo":"TsingZ0/PFL-Non-IID","repo_kind":"listed","path":"system/flcore/clients/clientntd.py","file_url":"https://github.com/TsingZ0/PFL-Non-IID/blob/HEAD/system/flcore/clients/clientntd.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b2630c53b756f58c"}},{"code_sha256_prefix":"0f492f116222f383","entry":"NTD_Loss","repo":"Lee-Gihun/FedNTD","repo_kind":"official","path":"algorithms/fedntd/criterion.py","file_url":"https://github.com/Lee-Gihun/FedNTD/blob/HEAD/algorithms/fedntd/criterion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f492f116222f383"}},{"code_sha256_prefix":"95d5cdba6e3175e5","entry":"NTD_Loss","repo":"thejungwon/gc-fed","repo_kind":"listed","path":"algorithms/fedntd.py","file_url":"https://github.com/thejungwon/gc-fed/blob/HEAD/algorithms/fedntd.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"95d5cdba6e3175e5"}},{"code_sha256_prefix":"b1c10f9b9003aa57","entry":"refine_as_not_true","repo":"TsingZ0/PFL-Non-IID","repo_kind":"listed","path":"system/flcore/clients/clientntd.py","file_url":"https://github.com/TsingZ0/PFL-Non-IID/blob/HEAD/system/flcore/clients/clientntd.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b1c10f9b9003aa57"}},{"code_sha256_prefix":"d862b7d6ee3577ef","entry":"FedNTD","repo":"thejungwon/gc-fed","repo_kind":"listed","path":"algorithms/fedntd.py","file_url":"https://github.com/thejungwon/gc-fed/blob/HEAD/algorithms/fedntd.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d862b7d6ee3577ef"}},{"code_sha256_prefix":"04996cc03e29c346","entry":"FedNTDClient","repo":"thejungwon/gc-fed","repo_kind":"listed","path":"algorithms/fedntd.py","file_url":"https://github.com/thejungwon/gc-fed/blob/HEAD/algorithms/fedntd.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"04996cc03e29c346"}},{"code_sha256_prefix":"722bf9f4417d11df","entry":"clientNTD","repo":"TsingZ0/PFL-Non-IID","repo_kind":"listed","path":"system/flcore/clients/clientntd.py","file_url":"https://github.com/TsingZ0/PFL-Non-IID/blob/HEAD/system/flcore/clients/clientntd.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"722bf9f4417d11df"}},{"code_sha256_prefix":"1a0a0b3bfb388c6e","entry":"evaluate","repo":"thejungwon/gc-fed","repo_kind":"listed","path":"algorithms/fedntd.py","file_url":"https://github.com/thejungwon/gc-fed/blob/HEAD/algorithms/fedntd.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1a0a0b3bfb388c6e"}},{"code_sha256_prefix":"55dd9a2df5092e25","entry":"set_random_seed","repo":"thejungwon/gc-fed","repo_kind":"listed","path":"algorithms/fedntd.py","file_url":"https://github.com/thejungwon/gc-fed/blob/HEAD/algorithms/fedntd.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"55dd9a2df5092e25"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}