{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/how-large-a-vocabulary-does-text","title":"How Large a Vocabulary Does Text Classification Need? A Variational Approach to Vocabulary Selection","arxiv_id":"1902.10339","date":"2019-02-27","proceeding":"NAACL 2019 6","authors":["Wenhu Chen","Yu Su","Yilin Shen","Zhiyu Chen","Xifeng Yan","William Wang"],"abstract":"With the rapid development in deep learning, deep neural networks have been\nwidely adopted in many real-life natural language applications. Under deep\nneural networks, a pre-defined vocabulary is required to vectorize text inputs.\nThe canonical approach to select pre-defined vocabulary is based on the word\nfrequency, where a threshold is selected to cut off the long tail distribution.\nHowever, we observed that such simple approach could easily lead to under-sized\nvocabulary or over-sized vocabulary issues. Therefore, we are interested in\nunderstanding how the end-task classification accuracy is related to the\nvocabulary size and what is the minimum required vocabulary size to achieve a\nspecific performance. In this paper, we provide a more sophisticated\nvariational vocabulary dropout (VVD) based on variational dropout to perform\nvocabulary selection, which can intelligently select the subset of the\nvocabulary to achieve the required performance. To evaluate different\nalgorithms on the newly proposed vocabulary selection problem, we propose two\nnew metrics: Area Under Accuracy-Vocab Curve and Vocab Size under X\\% Accuracy\nDrop. Through extensive experiments on various NLP classification tasks, our\nvariational framework is shown to significantly outperform the frequency-based\nand other selection baselines on these metrics.","url_abs":"http://arxiv.org/abs/1902.10339v4","url_pdf":"http://arxiv.org/pdf/1902.10339v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"how-large-a-vocabulary-does-text","repo_url":"https://github.com/wenhuchen/Variational-Vocabulary-Selection","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"classification","task_name":"General Classification"},{"task_slug":"text-classification","task_name":"Text Classification"},{"task_slug":"text-classification-1","task_name":"text-classification"}],"methods":[{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"variational-dropout","method_name":"Variational Dropout"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1902.10339","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1902.10339"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wenhuchen/Variational-Vocabulary-Selection","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2513d2d212c88f1f","entry":"ROC","repo":"wenhuchen/Variational-Vocabulary-Selection","repo_kind":"official","path":"data_utils.py","file_url":"https://github.com/wenhuchen/Variational-Vocabulary-Selection/blob/HEAD/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2513d2d212c88f1f"}},{"code_sha256_prefix":"74d1f3431662a7cf","entry":"enhanced","repo":"wenhuchen/Variational-Vocabulary-Selection","repo_kind":"official","path":"data_utils.py","file_url":"https://github.com/wenhuchen/Variational-Vocabulary-Selection/blob/HEAD/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"74d1f3431662a7cf"}},{"code_sha256_prefix":"575ab852ece5d750","entry":"obtain_interval","repo":"wenhuchen/Variational-Vocabulary-Selection","repo_kind":"official","path":"data_utils.py","file_url":"https://github.com/wenhuchen/Variational-Vocabulary-Selection/blob/HEAD/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"575ab852ece5d750"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}