{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sparknet-training-deep-networks-in-spark","title":"SparkNet: Training Deep Networks in Spark","arxiv_id":"1511.06051","date":"2015-11-19","proceeding":null,"authors":["Philipp Moritz","Robert Nishihara","Ion Stoica","Michael. I. Jordan"],"abstract":"Training deep networks is a time-consuming process, with networks for object\nrecognition often requiring multiple days to train. For this reason, leveraging\nthe resources of a cluster to speed up training is an important area of work.\nHowever, widely-popular batch-processing computational frameworks like\nMapReduce and Spark were not designed to support the asynchronous and\ncommunication-intensive workloads of existing distributed deep learning\nsystems. We introduce SparkNet, a framework for training deep networks in\nSpark. Our implementation includes a convenient interface for reading data from\nSpark RDDs, a Scala interface to the Caffe deep learning framework, and a\nlightweight multi-dimensional tensor library. Using a simple parallelization\nscheme for stochastic gradient descent, SparkNet scales well with the cluster\nsize and tolerates very high-latency communication. Furthermore, it is easy to\ndeploy and use with no parameter tuning, and it is compatible with existing\nCaffe models. We quantify the dependence of the speedup obtained by SparkNet on\nthe number of machines, the communication frequency, and the cluster's\ncommunication overhead, and we benchmark our system's performance on the\nImageNet dataset.","url_abs":"http://arxiv.org/abs/1511.06051v4","url_pdf":"http://arxiv.org/pdf/1511.06051v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sparknet-training-deep-networks-in-spark","repo_url":"https://github.com/amplab/SparkNet","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[],"methods":[{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1511.06051","atlas_url":"https://app.syntology.ai/?focus=1511.06051","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}