{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-batch-noise-contrastive-estimation-approach","title":"A Batch Noise Contrastive Estimation Approach for Training Large Vocabulary Language Models","arxiv_id":"1708.05997","date":"2017-08-20","proceeding":null,"authors":["Youssef Oualil","Dietrich Klakow"],"abstract":"Training large vocabulary Neural Network Language Models (NNLMs) is a\ndifficult task due to the explicit requirement of the output layer\nnormalization, which typically involves the evaluation of the full softmax\nfunction over the complete vocabulary. This paper proposes a Batch Noise\nContrastive Estimation (B-NCE) approach to alleviate this problem. This is\nachieved by reducing the vocabulary, at each time step, to the target words in\nthe batch and then replacing the softmax by the noise contrastive estimation\napproach, where these words play the role of targets and noise samples at the\nsame time. In doing so, the proposed approach can be fully formulated and\nimplemented using optimal dense matrix operations. Applying B-NCE to train\ndifferent NNLMs on the Large Text Compression Benchmark (LTCB) and the One\nBillion Word Benchmark (OBWB) shows a significant reduction of the training\ntime with no noticeable degradation of the models performance. This paper also\npresents a new baseline comparative study of different standard NNLMs on the\nlarge OBWB on a single Titan-X GPU.","url_abs":"http://arxiv.org/abs/1708.05997v2","url_pdf":"http://arxiv.org/pdf/1708.05997v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-batch-noise-contrastive-estimation-approach","repo_url":"https://github.com/Stonesjtu/Pytorch-NCE","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"text-compression","task_name":"Text Compression"}],"methods":[{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}