{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/character-level-language-modeling-with-deeper","title":"Character-Level Language Modeling with Deeper Self-Attention","arxiv_id":"1808.04444","date":"2018-08-09","proceeding":null,"authors":["Rami Al-Rfou","Dokook Choe","Noah Constant","Mandy Guo","Llion Jones"],"abstract":"LSTMs and other RNN variants have shown strong performance on character-level\nlanguage modeling. These models are typically trained using truncated\nbackpropagation through time, and it is common to assume that their success\nstems from their ability to remember long-term contexts. In this paper, we show\nthat a deep (64-layer) transformer model with fixed context outperforms RNN\nvariants by a large margin, achieving state of the art on two popular\nbenchmarks: 1.13 bits per character on text8 and 1.06 on enwik8. To get good\nresults at this depth, we show that it is important to add auxiliary losses,\nboth at intermediate network layers and intermediate sequence positions.","url_abs":"http://arxiv.org/abs/1808.04444v2","url_pdf":"http://arxiv.org/pdf/1808.04444v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"character-level-language-modeling-with-deeper","repo_url":"https://github.com/facebookresearch/code-prediction-transformer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/language-modelling-on-hutter-prize","task":"Language Modelling","dataset":"Hutter Prize","model":"64-layer Character Transformer Model","rank_in_archive_order":8,"of":18,"metrics":{"Bit per Character (BPC)":"1.06","Number of params":"235M"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-hutter-prize","task":"Language Modelling","dataset":"Hutter Prize","model":"12-layer Character Transformer Model","rank_in_archive_order":11,"of":18,"metrics":{"Bit per Character (BPC)":"1.11","Number of params":"44M"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-text8","task":"Language Modelling","dataset":"Text8","model":"64-layer Character Transformer Model","rank_in_archive_order":11,"of":24,"metrics":{"Bit per Character (BPC)":"1.13","Number of params":"235M"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-text8","task":"Language Modelling","dataset":"Text8","model":"12-layer Character Transformer Model","rank_in_archive_order":13,"of":24,"metrics":{"Bit per Character (BPC)":"1.18","Number of params":"44M"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-enwiki8","task":"Language Modelling","dataset":"enwik8","model":"Transformer (64 layers)","rank_in_archive_order":25,"of":42,"metrics":{"Bit per Character (BPC)":"1.06","Number of params":"235M"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-enwiki8","task":"Language Modelling","dataset":"enwik8","model":"64-layer Character Transformer Model","rank_in_archive_order":29,"of":42,"metrics":{"Bit per Character (BPC)":"1.11","Number of params":"44M"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1808.04444","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}