{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-block-random-algorithm-for-learning-on","title":"A block-random algorithm for learning on distributed, heterogeneous data","arxiv_id":"1903.00091","date":"2019-02-28","proceeding":null,"authors":["Prakash Mohan","Marc T. Henry de Frahan","Ryan King","Ray W. Grout"],"abstract":"Most deep learning models are based on deep neural networks with multiple\nlayers between input and output. The parameters defining these layers are\ninitialized using random values and are \"learned\" from data, typically using\nstochastic gradient descent based algorithms. These algorithms rely on data\nbeing randomly shuffled before optimization. The randomization of the data\nprior to processing in batches that is formally required for stochastic\ngradient descent algorithm to effectively derive a useful deep learning model\nis expected to be prohibitively expensive for in situ model training because of\nthe resulting data communications across the processor nodes. We show that the\nstochastic gradient descent (SGD) algorithm can still make useful progress if\nthe batches are defined on a per-processor basis and processed in random order\neven though (i) the batches are constructed from data samples from a single\nclass or specific flow region, and (ii) the overall data samples are\nheterogeneous. We present block-random gradient descent, a new algorithm that\nworks on distributed, heterogeneous data without having to pre-shuffle. This\nalgorithm enables in situ learning for exascale simulations. The performance of\nthis algorithm is demonstrated on a set of benchmark classification models and\nthe construction of a subgrid scale large eddy simulations (LES) model for\nturbulent channel flow using a data model similar to that which will be\nencountered in exascale simulation.","url_abs":"http://arxiv.org/abs/1903.00091v1","url_pdf":"http://arxiv.org/pdf/1903.00091v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-block-random-algorithm-for-learning-on","repo_url":"https://github.com/NREL/block-random","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}