{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/predefined-sparseness-in-recurrent-sequence","title":"Predefined Sparseness in Recurrent Sequence Models","arxiv_id":"1808.08720","date":"2018-08-27","proceeding":"CONLL 2018 10","authors":["Thomas Demeester","Johannes Deleu","Fréderic Godin","Chris Develder"],"abstract":"Inducing sparseness while training neural networks has been shown to yield\nmodels with a lower memory footprint but similar effectiveness to dense models.\nHowever, sparseness is typically induced starting from a dense model, and thus\nthis advantage does not hold during training. We propose techniques to enforce\nsparseness upfront in recurrent sequence models for NLP applications, to also\nbenefit training. First, in language modeling, we show how to increase hidden\nstate sizes in recurrent layers without increasing the number of parameters,\nleading to more expressive models. Second, for sequence labeling, we show that\nword embeddings with predefined sparseness lead to similar performance as dense\nembeddings, at a fraction of the number of trainable parameters.","url_abs":"http://arxiv.org/abs/1808.08720v1","url_pdf":"http://arxiv.org/pdf/1808.08720v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"predefined-sparseness-in-recurrent-sequence","repo_url":"https://github.com/tdmeeste/SparseSeqModels","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"word-embeddings","task_name":"Word Embeddings"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}