Papers › Hierarchical Transformer for Task Oriented Dialog Systems

Hierarchical Transformer for Task Oriented Dialog Systems

24 Oct 2020NAACL 2021 4arXiv:2011.08067archive 2025-07-28

Bishal Santra, Potnuru Anusha, Pawan Goyal

Generative models for dialog systems have gained much interest because of the recent success of RNN and Transformer based models in tasks like question answering and summarization. Although the task of dialog response generation is generally seen as a sequence-to-sequence (Seq2Seq) problem, researchers in the past have found it challenging to train dialog systems using the standard Seq2Seq models. Therefore, to help the model learn meaningful utterance and conversation level features, Sordoni et al. (2015b); Serban et al. (2016) proposed Hierarchical RNN architecture, which was later adopted by several other RNN based dialog systems. With the transformer-based models dominating the seq2seq problems lately, the natural question to ask is the applicability of the notion of hierarchy in transformer based dialog systems. In this paper, we propose a generalized framework for Hierarchical Transformer Encoders and show how a standard transformer can be morphed into any hierarchical encoder, including HRED and HIBERT like models, by using specially designed attention masks and positional encodings. We demonstrate that Hierarchical Encoding helps achieve better natural language understanding of the contexts in transformer-based models for task-oriented dialog systems through a wide range of experiments.

Paper PDF Conference PDF Code

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

bsantraigi/HIER officialmentioned in papermentioned on GitHubpytorch report

bsantraigi/hier-transformer-pytorch officialmentioned in papermentioned on GitHubpytorch report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Natural Language Understanding Question Answering Response Generation

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Methods

Absolute Position Encodings Adam Attention BPE Dense Connections Dropout LSTM Label Smoothing Layer Normalization Linear Layer Multi-Head Attention Position-Wise Feed-Forward Layer Residual Connection Seq2Seq Sigmoid Activation Softmax Tanh Activation Transformer

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections