{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tictac-accelerating-distributed-deep-learning","title":"TicTac: Accelerating Distributed Deep Learning with Communication Scheduling","arxiv_id":"1803.03288","date":"2018-03-08","proceeding":null,"authors":["Sayed Hadi Hashemi","Sangeetha Abdu Jyothi","Roy H. Campbell"],"abstract":"State-of-the-art deep learning systems rely on iterative distributed training\nto tackle the increasing complexity of models and input data. The iteration\ntime in these communication-heavy systems depends on the computation time,\ncommunication time and the extent of overlap of computation and communication.\n  In this work, we identify a shortcoming in systems with graph representation\nfor computation, such as TensorFlow and PyTorch, that result in high variance\nin iteration time --- random order of received parameters across workers. We\ndevelop a system, TicTac, to improve the iteration time by fixing this issue in\ndistributed deep learning with Parameter Servers while guaranteeing\nnear-optimal overlap of communication and computation. TicTac identifies and\nenforces an order of network transfers which improves the iteration time using\nprioritization. Our system is implemented over TensorFlow and requires no\nchanges to the model or developer inputs. TicTac improves the throughput by up\nto $37.7\\%$ in inference and $19.2\\%$ in training, while also reducing\nstraggler effect by up to $2.3\\times$. Our code is publicly available.","url_abs":"http://arxiv.org/abs/1803.03288v2","url_pdf":"http://arxiv.org/pdf/1803.03288v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tictac-accelerating-distributed-deep-learning","repo_url":"https://github.com/xldrx/tictac","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"deep-learning","task_name":"Deep Learning"},{"task_slug":"scheduling","task_name":"Scheduling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1803.03288","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}