{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/maybe-deep-neural-networks-are-the-best","title":"Maybe Deep Neural Networks are the Best Choice for Modeling Source Code","arxiv_id":"1903.05734","date":"2019-03-13","proceeding":null,"authors":["Rafael-Michael Karampatsis","Charles Sutton"],"abstract":"Statistical language modeling techniques have successfully been applied to\nsource code, yielding a variety of new software development tools, such as\ntools for code suggestion and improving readability. A major issue with these\ntechniques is that code introduces new vocabulary at a far higher rate than\nnatural language, as new identifier names proliferate. But traditional language\nmodels limit the vocabulary to a fixed set of common words. For code, this\nstrong assumption has been shown to have a significant negative effect on\npredictive performance. But the open vocabulary version of the neural network\nlanguage models for code have not been introduced in the literature. We present\na new open-vocabulary neural language model for code that is not limited to a\nfixed vocabulary of identifier names. We employ a segmentation into subword\nunits, subsequences of tokens chosen based on a compression criterion,\nfollowing previous work in machine translation. Our network achieves best in\nclass performance, outperforming even the state-of-the-art methods of\nHellendoorn and Devanbu that are designed specifically to model code.\nFurthermore, we present a simple method for dynamically adapting the model to a\nnew test project, resulting in increased performance. We showcase our\nmethodology on code corpora in three different languages of over a billion\ntokens each, hundreds of times larger than in previous work. To our knowledge,\nthis is the largest neural language model for code that has been reported.","url_abs":"http://arxiv.org/abs/1903.05734v1","url_pdf":"http://arxiv.org/pdf/1903.05734v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"maybe-deep-neural-networks-are-the-best","repo_url":"https://github.com/mast-group/OpenVocabCodeNLM","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"machine-translation","task_name":"Machine Translation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}