{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/parallelizing-word2vec-in-multi-core-and-many","title":"Parallelizing Word2Vec in Multi-Core and Many-Core Architectures","arxiv_id":"1611.06172","date":"2016-11-18","proceeding":null,"authors":["Shihao Ji","Nadathur Satish","Sheng Li","Pradeep Dubey"],"abstract":"Word2vec is a widely used algorithm for extracting low-dimensional vector\nrepresentations of words. State-of-the-art algorithms including those by\nMikolov et al. have been parallelized for multi-core CPU architectures, but are\nbased on vector-vector operations with \"Hogwild\" updates that are\nmemory-bandwidth intensive and do not efficiently use computational resources.\nIn this paper, we propose \"HogBatch\" by improving reuse of various data\nstructures in the algorithm through the use of minibatching and negative sample\nsharing, hence allowing us to express the problem using matrix multiply\noperations. We also explore different techniques to distribute word2vec\ncomputation across nodes in a compute cluster, and demonstrate good strong\nscalability up to 32 nodes. The new algorithm is particularly suitable for\nmodern multi-core/many-core architectures, especially Intel's latest Knights\nLanding processors, and allows us to scale up the computation near linearly\nacross cores and nodes, and process hundreds of millions of words per second,\nwhich is the fastest word2vec implementation to the best of our knowledge.","url_abs":"http://arxiv.org/abs/1611.06172v2","url_pdf":"http://arxiv.org/pdf/1611.06172v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"parallelizing-word2vec-in-multi-core-and-many","repo_url":"https://github.com/IntelLabs/pWord2Vec","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":null,"task_name":"CPU"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}