{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/efficient-exact-gradient-update-for-training","title":"Efficient Exact Gradient Update for training Deep Networks with Very Large Sparse Targets","arxiv_id":"1412.7091","date":"2014-12-22","proceeding":"NeurIPS 2015 12","authors":["Pascal Vincent","Alexandre de Brébisson","Xavier Bouthillier"],"abstract":"An important class of problems involves training deep neural networks with\nsparse prediction targets of very high dimension D. These occur naturally in\ne.g. neural language models or the learning of word-embeddings, often posed as\npredicting the probability of next words among a vocabulary of size D (e.g. 200\n000). Computing the equally large, but typically non-sparse D-dimensional\noutput vector from a last hidden layer of reasonable dimension d (e.g. 500)\nincurs a prohibitive O(Dd) computational cost for each example, as does\nupdating the D x d output weight matrix and computing the gradient needed for\nbackpropagation to previous layers. While efficient handling of large sparse\nnetwork inputs is trivial, the case of large sparse targets is not, and has\nthus so far been sidestepped with approximate alternatives such as hierarchical\nsoftmax or sampling-based approximations during training. In this work we\ndevelop an original algorithmic approach which, for a family of loss functions\nthat includes squared error and spherical softmax, can compute the exact loss,\ngradient update for the output weights, and gradient for backpropagation, all\nin O(d^2) per example instead of O(Dd), remarkably without ever computing the\nD-dimensional output. The proposed algorithm yields a speedup of D/4d , i.e.\ntwo orders of magnitude for typical sizes, for that critical part of the\ncomputations that often dominates the training time in this kind of network\narchitecture.","url_abs":"http://arxiv.org/abs/1412.7091v3","url_pdf":"http://arxiv.org/pdf/1412.7091v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"efficient-exact-gradient-update-for-training","repo_url":"https://github.com/pascal20100/factored_output_layer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"word-embeddings","task_name":"Word Embeddings"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1412.7091","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}