Papers › Distance-based Self-Attention Network for Natural Language Inference

Distance-based Self-Attention Network for Natural Language Inference

6 Dec 2017arXiv:1712.02047archive 2025-07-28

Jinbae Im, Sungzoon Cho

Attention mechanism has been used as an ancillary means to help RNN or CNN. However, the Transformer (Vaswani et al., 2017) recently recorded the state-of-the-art performance in machine translation with a dramatic reduction in training time by solely using attention. Motivated by the Transformer, Directional Self Attention Network (Shen et al., 2017), a fully attention-based sentence encoder, was proposed. It showed good performance with various data by using forward and backward directional information in a sentence. But in their study, not considered at all was the distance between words, an important feature when learning the local dependency to help understand the context of input text. We propose Distance-based Self-Attention Network, which considers the word distance by using a simple distance mask in order to model the local dependency without losing the ability of modeling global dependency which attention has inherent. Our model shows good performance with NLI data, and it records the new state-of-the-art result with SNLI data. Additionally, we show that our model has a strength in long sentences or documents.

Paper PDF

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

No code repository is listed for this paper in the archive or in Syntology's graph.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Machine Translation Natural Language Inference Sentence Translation

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Natural Language Inference	SNLI	Distance-based Self-Attention Network	% Test Accuracy	86.3	#57 of 98	Archive leaderboard	report
Natural Language Inference	SNLI	Distance-based Self-Attention Network	% Train Accuracy	89.6	#57 of 98	Archive leaderboard	report
Natural Language Inference	SNLI	Distance-based Self-Attention Network	Parameters	4.7m	#57 of 98	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Absolute Position Encodings Adam Attention BPE Dense Connections Dropout Label Smoothing Layer Normalization Linear Layer Multi-Head Attention Position-Wise Feed-Forward Layer ReLU Residual Connection Softmax Transformer

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections