{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-the-state-of-the-art-of-evaluation-in","title":"On the State of the Art of Evaluation in Neural Language Models","arxiv_id":"1707.05589","date":"2017-07-18","proceeding":"ICLR 2018 1","authors":["Gábor Melis","Chris Dyer","Phil Blunsom"],"abstract":"Ongoing innovations in recurrent neural network architectures have provided a\nsteady influx of apparently state-of-the-art results on language modelling\nbenchmarks. However, these have been evaluated using differing code bases and\nlimited computational resources, which represent uncontrolled sources of\nexperimental variation. We reevaluate several popular architectures and\nregularisation methods with large-scale automatic black-box hyperparameter\ntuning and arrive at the somewhat surprising conclusion that standard LSTM\narchitectures, when properly regularised, outperform more recent models. We\nestablish a new state of the art on the Penn Treebank and Wikitext-2 corpora,\nas well as strong baselines on the Hutter Prize dataset.","url_abs":"http://arxiv.org/abs/1707.05589v2","url_pdf":"http://arxiv.org/pdf/1707.05589v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-the-state-of-the-art-of-evaluation-in","repo_url":"https://github.com/deepmind/lamb","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/language-modelling-on-wikitext-2","task":"Language Modelling","dataset":"WikiText-2","model":"Melis et al. (2017) - 1-layer LSTM (tied)","rank_in_archive_order":32,"of":38,"metrics":{"Number of params":"24M","Test perplexity":"65.9","Validation perplexity":"69.3"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1707.05589","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}