{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dynamic-mini-batch-sgd-for-elastic","title":"Dynamic Mini-batch SGD for Elastic Distributed Training: Learning in the Limbo of Resources","arxiv_id":"1904.12043","date":"2019-04-26","proceeding":null,"authors":["Haibin Lin","Hang Zhang","Yifei Ma","Tong He","Zhi Zhang","Sheng Zha","Mu Li"],"abstract":"With an increasing demand for training powers for deep learning algorithms\nand the rapid growth of computation resources in data centers, it is desirable\nto dynamically schedule different distributed deep learning tasks to maximize\nresource utilization and reduce cost. In this process, different tasks may\nreceive varying numbers of machines at different time, a setting we call\nelastic distributed training. Despite the recent successes in large mini-batch\ndistributed training, these methods are rarely tested in elastic distributed\ntraining environments and suffer degraded performance in our experiments, when\nwe adjust the learning rate linearly immediately with respect to the batch\nsize. One difficulty we observe is that the noise in the stochastic momentum\nestimation is accumulated over time and will have delayed effects when the\nbatch size changes. We therefore propose to smoothly adjust the learning rate\nover time to alleviate the influence of the noisy momentum estimation. Our\nexperiments on image classification, object detection and semantic segmentation\nhave demonstrated that our proposed Dynamic SGD method achieves stabilized\nperformance when varying the number of GPUs from 8 to 128. We also provide\ntheoretical understanding on the optimality of linear learning rate scheduling\nand the effects of stochastic momentum.","url_abs":"http://arxiv.org/abs/1904.12043v2","url_pdf":"http://arxiv.org/pdf/1904.12043v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dynamic-mini-batch-sgd-for-elastic","repo_url":"https://github.com/awslabs/dynamic-training-with-apache-mxnet-on-aws","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"mxnet","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"dynamic-mini-batch-sgd-for-elastic","repo_url":"https://github.com/AlanFokCo/compensation-tools","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"scheduling","task_name":"Scheduling"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"image-classification","task_name":"image-classification"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1904.12043","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1904.12043"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/AlanFokCo/compensation-tools","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/awslabs/dynamic-training-with-apache-mxnet-on-aws","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":20},"by_repo_kind":{"listed":{"samples":20,"ran":0,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"f235e1530ce3d53d","entry":"bulk","repo":"awslabs/dynamic-training-with-apache-mxnet-on-aws","repo_kind":"listed","path":"python/mxnet/engine.py","file_url":"https://github.com/awslabs/dynamic-training-with-apache-mxnet-on-aws/blob/HEAD/python/mxnet/engine.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"f235e1530ce3d53d"}},{"code_sha256_prefix":"3bc3ed8fa916a182","entry":"c_array","repo":"awslabs/dynamic-training-with-apache-mxnet-on-aws","repo_kind":"listed","path":"amalgamation/python/mxnet_predict.py","file_url":"https://github.com/awslabs/dynamic-training-with-apache-mxnet-on-aws/blob/HEAD/amalgamation/python/mxnet_predict.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"3bc3ed8fa916a182"}},{"code_sha256_prefix":"c037ab026fd36b3a","entry":"c_array","repo":"awslabs/dynamic-training-with-apache-mxnet-on-aws","repo_kind":"listed","path":"python/mxnet/base.py","file_url":"https://github.com/awslabs/dynamic-training-with-apache-mxnet-on-aws/blob/HEAD/python/mxnet/base.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"c037ab026fd36b3a"}},{"code_sha256_prefix":"313d72408f5f6086","entry":"c_str","repo":"awslabs/dynamic-training-with-apache-mxnet-on-aws","repo_kind":"listed","path":"amalgamation/python/mxnet_predict.py","file_url":"https://github.com/awslabs/dynamic-training-with-apache-mxnet-on-aws/blob/HEAD/amalgamation/python/mxnet_predict.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"313d72408f5f6086"}},{"code_sha256_prefix":"47ebd1622534322e","entry":"classproperty","repo":"awslabs/dynamic-training-with-apache-mxnet-on-aws","repo_kind":"listed","path":"python/mxnet/base.py","file_url":"https://github.com/awslabs/dynamic-training-with-apache-mxnet-on-aws/blob/HEAD/python/mxnet/base.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"47ebd1622534322e"}},{"code_sha256_prefix":"9ff2673d6a779a1a","entry":"create","repo":"awslabs/dynamic-training-with-apache-mxnet-on-aws","repo_kind":"listed","path":"python/mxnet/kvstore.py","file_url":"https://github.com/awslabs/dynamic-training-with-apache-mxnet-on-aws/blob/HEAD/python/mxnet/kvstore.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"9ff2673d6a779a1a"}},{"code_sha256_prefix":"df77b3eca6bd676b","entry":"do_checkpoint","repo":"awslabs/dynamic-training-with-apache-mxnet-on-aws","repo_kind":"listed","path":"python/mxnet/callback.py","file_url":"https://github.com/awslabs/dynamic-training-with-apache-mxnet-on-aws/blob/HEAD/python/mxnet/callback.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"df77b3eca6bd676b"}},{"code_sha256_prefix":"3c9ed3332c1f6ca8","entry":"load_data","repo":"AlanFokCo/compensation-tools","repo_kind":"listed","path":"elastic_demos/data_process.py","file_url":"https://github.com/AlanFokCo/compensation-tools/blob/HEAD/elastic_demos/data_process.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3c9ed3332c1f6ca8"}},{"code_sha256_prefix":"2d6b2877daa0effe","entry":"load_data_single","repo":"AlanFokCo/compensation-tools","repo_kind":"listed","path":"elastic_demos/add_data.py","file_url":"https://github.com/AlanFokCo/compensation-tools/blob/HEAD/elastic_demos/add_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2d6b2877daa0effe"}},{"code_sha256_prefix":"e987ba7540887ac6","entry":"load_val_data","repo":"AlanFokCo/compensation-tools","repo_kind":"listed","path":"elastic_demos/data_process.py","file_url":"https://github.com/AlanFokCo/compensation-tools/blob/HEAD/elastic_demos/data_process.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e987ba7540887ac6"}},{"code_sha256_prefix":"5bc9dd9627f0eb1c","entry":"log_train_metric","repo":"awslabs/dynamic-training-with-apache-mxnet-on-aws","repo_kind":"listed","path":"python/mxnet/callback.py","file_url":"https://github.com/awslabs/dynamic-training-with-apache-mxnet-on-aws/blob/HEAD/python/mxnet/callback.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"5bc9dd9627f0eb1c"}},{"code_sha256_prefix":"d46ae594e2adad67","entry":"map_func","repo":"AlanFokCo/compensation-tools","repo_kind":"listed","path":"elastic_demos/tensorflow_v2_cifar_resnet.py","file_url":"https://github.com/AlanFokCo/compensation-tools/blob/HEAD/elastic_demos/tensorflow_v2_cifar_resnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d46ae594e2adad67"}},{"code_sha256_prefix":"4950353b99f3413d","entry":"module_checkpoint","repo":"awslabs/dynamic-training-with-apache-mxnet-on-aws","repo_kind":"listed","path":"python/mxnet/callback.py","file_url":"https://github.com/awslabs/dynamic-training-with-apache-mxnet-on-aws/blob/HEAD/python/mxnet/callback.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"4950353b99f3413d"}},{"code_sha256_prefix":"b98592423270ab35","entry":"normalization","repo":"AlanFokCo/compensation-tools","repo_kind":"listed","path":"elastic_demos/data_process.py","file_url":"https://github.com/AlanFokCo/compensation-tools/blob/HEAD/elastic_demos/data_process.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b98592423270ab35"}},{"code_sha256_prefix":"b86308bc3bc53b49","entry":"record","repo":"awslabs/dynamic-training-with-apache-mxnet-on-aws","repo_kind":"listed","path":"python/mxnet/autograd.py","file_url":"https://github.com/awslabs/dynamic-training-with-apache-mxnet-on-aws/blob/HEAD/python/mxnet/autograd.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"b86308bc3bc53b49"}},{"code_sha256_prefix":"8bcbec3a30a50379","entry":"set_bulk_size","repo":"awslabs/dynamic-training-with-apache-mxnet-on-aws","repo_kind":"listed","path":"python/mxnet/engine.py","file_url":"https://github.com/awslabs/dynamic-training-with-apache-mxnet-on-aws/blob/HEAD/python/mxnet/engine.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"8bcbec3a30a50379"}},{"code_sha256_prefix":"84a2b5addc51f455","entry":"set_recording","repo":"awslabs/dynamic-training-with-apache-mxnet-on-aws","repo_kind":"listed","path":"python/mxnet/autograd.py","file_url":"https://github.com/awslabs/dynamic-training-with-apache-mxnet-on-aws/blob/HEAD/python/mxnet/autograd.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"84a2b5addc51f455"}},{"code_sha256_prefix":"565479a708f0d1c6","entry":"set_training","repo":"awslabs/dynamic-training-with-apache-mxnet-on-aws","repo_kind":"listed","path":"python/mxnet/autograd.py","file_url":"https://github.com/awslabs/dynamic-training-with-apache-mxnet-on-aws/blob/HEAD/python/mxnet/autograd.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"565479a708f0d1c6"}},{"code_sha256_prefix":"91b57896044b9479","entry":"unpickle","repo":"AlanFokCo/compensation-tools","repo_kind":"listed","path":"elastic_demos/add_data.py","file_url":"https://github.com/AlanFokCo/compensation-tools/blob/HEAD/elastic_demos/add_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"91b57896044b9479"}},{"code_sha256_prefix":"05550ef10b65470b","entry":"with_metaclass","repo":"awslabs/dynamic-training-with-apache-mxnet-on-aws","repo_kind":"listed","path":"python/mxnet/base.py","file_url":"https://github.com/awslabs/dynamic-training-with-apache-mxnet-on-aws/blob/HEAD/python/mxnet/base.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"05550ef10b65470b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}