{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/subword-encoding-in-lattice-lstm-for-chinese","title":"Subword Encoding in Lattice LSTM for Chinese Word Segmentation","arxiv_id":"1810.12594","date":"2018-10-30","proceeding":"NAACL 2019 6","authors":["Jie Yang","Yue Zhang","Shuailong Liang"],"abstract":"We investigate a lattice LSTM network for Chinese word segmentation (CWS) to\nutilize words or subwords. It integrates the character sequence features with\nall subsequences information matched from a lexicon. The matched subsequences\nserve as information shortcut tunnels which link their start and end characters\ndirectly. Gated units are used to control the contribution of multiple input\nlinks. Through formula derivation and comparison, we show that the lattice LSTM\nis an extension of the standard LSTM with the ability to take multiple inputs.\nPrevious lattice LSTM model takes word embeddings as the lexicon input, we\nprove that subword encoding can give the comparable performance and has the\nbenefit of not relying on any external segmentor. The contribution of lattice\nLSTM comes from both lexicon and pretrained embeddings information, we find\nthat the lexicon information contributes more than the pretrained embeddings\ninformation through controlled experiments. Our experiments show that the\nlattice structure with subword encoding gives competitive or better results\nwith previous state-of-the-art methods on four segmentation benchmarks.\nDetailed analyses are conducted to compare the performance of word encoding and\nsubword encoding in lattice LSTM. We also investigate the performance of\nlattice LSTM structure under different circumstances and when this model works\nor fails.","url_abs":"http://arxiv.org/abs/1810.12594v1","url_pdf":"http://arxiv.org/pdf/1810.12594v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"subword-encoding-in-lattice-lstm-for-chinese","repo_url":"https://github.com/jiesutd/SubwordEncoding-CWS","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"chinese-word-segmentation","task_name":"Chinese Word Segmentation"},{"task_slug":"word-embeddings","task_name":"Word Embeddings"}],"methods":[{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1810.12594","atlas_url":"https://app.syntology.ai/?focus=1810.12594","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}