{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ctran-cnn-transformer-based-network-for","title":"CTRAN: CNN-Transformer-based Network for Natural Language Understanding","arxiv_id":"2303.10606","date":"2023-03-19","proceeding":null,"authors":["Mehrdad Rafiepour","Javad Salimi Sartakhti"],"abstract":"Intent-detection and slot-filling are the two main tasks in natural language understanding. In this study, we propose CTRAN, a novel encoder-decoder CNN-Transformer-based architecture for intent-detection and slot-filling. In the encoder, we use BERT, followed by several convolutional layers, and rearrange the output using window feature sequence. We use stacked Transformer encoders after the window feature sequence. For the intent-detection decoder, we utilize self-attention followed by a linear layer. In the slot-filling decoder, we introduce the aligned Transformer decoder, which utilizes a zero diagonal mask, aligning output tags with input tokens. We apply our network on ATIS and SNIPS, and surpass the current state-of-the-art in slot-filling on both datasets. Furthermore, we incorporate the language model as word embeddings, and show that this strategy yields a better result when compared to the language model as an encoder.","url_abs":"https://arxiv.org/abs/2303.10606v1","url_pdf":"https://arxiv.org/pdf/2303.10606v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ctran-cnn-transformer-based-network-for","repo_url":"https://github.com/rafiepour/CTran","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"intent-detection","task_name":"Intent Detection"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"natural-language-understanding","task_name":"Natural Language Understanding"},{"task_slug":"slot-filling","task_name":"Slot Filling"},{"task_slug":"word-embeddings","task_name":"Word Embeddings"},{"task_slug":"slot-filling-1","task_name":"slot-filling"}],"methods":[{"method_slug":"1d-cnn","method_name":"1D CNN"},{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/intent-detection-on-atis","task":"Intent Detection","dataset":"ATIS","model":"CTRAN","rank_in_archive_order":5,"of":16,"metrics":{"Accuracy":"98.07"},"uses_additional_data":false},{"leaderboard":"/sota/intent-detection-on-snips","task":"Intent Detection","dataset":"SNIPS","model":"CTRAN","rank_in_archive_order":1,"of":10,"metrics":{"Accuracy":"99.42"},"uses_additional_data":false},{"leaderboard":"/sota/slot-filling-on-atis","task":"Slot Filling","dataset":"ATIS","model":"CTRAN","rank_in_archive_order":1,"of":14,"metrics":{"F1":"0.9846"},"uses_additional_data":false},{"leaderboard":"/sota/slot-filling-on-snips","task":"Slot Filling","dataset":"SNIPS","model":"CTRAN","rank_in_archive_order":1,"of":10,"metrics":{"F1":"98.3"},"uses_additional_data":false}],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}