{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hydra-a-peer-to-peer-distributed-training","title":"Hydra: A Peer to Peer Distributed Training & Data Collection Framework","arxiv_id":"1811.09878","date":"2018-11-24","proceeding":null,"authors":["Vaibhav Mathur","Karanbir Chahal"],"abstract":"The world needs diverse and unbiased data to train deep learning models.\nCurrently data comes from a variety of sources that are unmoderated to a large\nextent. The outcomes of training neural networks with unverified data yields\nbiased models with various strains of homophobia, sexism and racism. Another\ntrend observed in the world of deep learning is the rise of distributed\ntraining. Although cloud companies provide high performance compute for\ntraining models in the form of GPU's connected with a low latency network,\nusing these services comes at a high cost. We propose Hydra, a system that\nseeks to solve both of these problems in a novel manner by proposing a\ndecentralized distributed framework which utilizes the substantial amount of\nidle compute of everyday electronic devices like smartphones and desktop\ncomputers for training and data collection purposes. Hydra couples a\nspecialized distributed training framework on a network of these low powered\ndevices with a reward scheme that incentivizes users to provide high quality\ndata to unleash the compute capability on this training framework. Such a\nsystem has the ability to capture data from a wide variety of diverse sources\nwhich has been an issue in the current scenario of deep learning. Hydra brings\nin several new innovations in training on low powered devices including a fault\ntolerant version of the All Reduce algorithm. Furthermore we introduce a\nreinforcement learning policy to decide the size of training jobs on different\nmachines on a heterogeneous cluster of devices with varying network latencies\nfor Synchronous SGD. The novel thing about such a network is the ability of\neach machine to shut down and resume training capabilities at any point of time\nwithout restarting the overall training. To enable such an asynchronous\nbehaviour we propose a communication framework inspired by the Bittorrent\nprotocol and the Kademlia DHT.","url_abs":"http://arxiv.org/abs/1811.09878v1","url_pdf":"http://arxiv.org/pdf/1811.09878v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hydra-a-peer-to-peer-distributed-training","repo_url":"https://github.com/hydra-hoard/hydra","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"}],"methods":[{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}