{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/leaf-a-benchmark-for-federated-settings","title":"LEAF: A Benchmark for Federated Settings","arxiv_id":"1812.01097","date":"2018-12-03","proceeding":null,"authors":["Sebastian Caldas","Sai Meher Karthik Duddu","Peter Wu","Tian Li","Jakub Konečný","H. Brendan McMahan","Virginia Smith","Ameet Talwalkar"],"abstract":"Modern federated networks, such as those comprised of wearable devices, mobile phones, or autonomous vehicles, generate massive amounts of data each day. This wealth of data can help to learn models that can improve the user experience on each device. However, the scale and heterogeneity of federated data presents new challenges in research areas such as federated learning, meta-learning, and multi-task learning. As the machine learning community begins to tackle these challenges, we are at a critical time to ensure that developments made in these areas are grounded with realistic benchmarks. To this end, we propose LEAF, a modular benchmarking framework for learning in federated settings. LEAF includes a suite of open-source federated datasets, a rigorous evaluation framework, and a set of reference implementations, all geared towards capturing the obstacles and intricacies of practical federated environments.","url_abs":"https://arxiv.org/abs/1812.01097v3","url_pdf":"https://arxiv.org/pdf/1812.01097v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"leaf-a-benchmark-for-federated-settings","repo_url":"https://github.com/TalwalkarLab/leaf","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"leaf-a-benchmark-for-federated-settings","repo_url":"https://github.com/PaddlePaddle/PaddleFL","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"paddle","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"leaf-a-benchmark-for-federated-settings","repo_url":"https://github.com/Superminionsfy/Personalized-Data-Generation-using-GAN","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"leaf-a-benchmark-for-federated-settings","repo_url":"https://github.com/aioz-ai/multigraphfl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"leaf-a-benchmark-for-federated-settings","repo_url":"https://github.com/lishenghui/ARFL","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"leaf-a-benchmark-for-federated-settings","repo_url":"https://github.com/lizonghang/leaf-mx","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"mxnet","reach":{"status":"gone","observed_at":"2026-09-18","how":"tree_404+repo_404"}},{"paper_slug":"leaf-a-benchmark-for-federated-settings","repo_url":"https://github.com/vaseline555/Federated-Learning-in-PyTorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"autonomous-vehicles","task_name":"Autonomous Vehicles"},{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"federated-learning","task_name":"Federated Learning"},{"task_slug":"meta-learning","task_name":"Meta-Learning"},{"task_slug":"multi-task-learning","task_name":"Multi-Task Learning"}],"methods":[],"datasets_introduced":[{"slug":"leaf-benchmark","name":"LEAF Benchmark","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1812.01097","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1812.01097"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/TalwalkarLab/leaf","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aioz-ai/multigraphfl","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lishenghui/ARFL","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Superminionsfy/Personalized-Data-Generation-using-GAN","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lizonghang/leaf-mx","reach":{"status":"gone","observed_at":"2026-09-18","how":"tree_404+repo_404"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vaseline555/Federated-Learning-in-PyTorch","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/PaddlePaddle/PaddleFL","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"unverified":2},"by_repo_kind":{"listed":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7690c4d309642d3f","entry":"get_metrics_names","repo":"lishenghui/ARFL","repo_kind":"listed","path":"models/metrics/writer.py","file_url":"https://github.com/lishenghui/ARFL/blob/HEAD/models/metrics/writer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7690c4d309642d3f"}},{"code_sha256_prefix":"414700a37378e111","entry":"bi_partitioning","repo":"lishenghui/ARFL","repo_kind":"listed","path":"models/utils/algorithms.py","file_url":"https://github.com/lishenghui/ARFL/blob/HEAD/models/utils/algorithms.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"414700a37378e111"}},{"code_sha256_prefix":"42f9eec2cac36fb9","entry":"cosine_similarity_lists","repo":"lishenghui/ARFL","repo_kind":"listed","path":"models/utils/algorithms.py","file_url":"https://github.com/lishenghui/ARFL/blob/HEAD/models/utils/algorithms.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"42f9eec2cac36fb9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}