Papers › FarsTail: A Persian Natural Language Inference Dataset
FarsTail: A Persian Natural Language Inference Dataset
Hossein Amirkhani, Mohammad AzariJafari, Zohreh Pourjafari, Soroush Faridan-Jahromi, Zeinab Kouhkan, Azadeh Amirak
Natural language inference (NLI) is known as one of the central tasks in natural language processing (NLP) which encapsulates many fundamental aspects of language understanding. With the considerable achievements of data-hungry deep learning methods in NLP tasks, a great amount of effort has been devoted to develop more diverse datasets for different languages. In this paper, we present a new dataset for the NLI task in the Persian language, also known as Farsi, which is one of the dominant languages in the Middle East. This dataset, named FarsTail, includes 10,367 samples which are provided in both the Persian language as well as the indexed format to be useful for non-Persian researchers. The samples are generated from 3,539 multiple-choice questions with the least amount of annotator interventions in a way similar to the SciTail dataset. A carefully designed multi-step process is adopted to ensure the quality of the dataset. We also present the results of traditional and state-of-the-art methods on FarsTail including different embedding methods such as word2vec, fastText, ELMo, BERT, and LASER, as well as different modeling approaches such as DecompAtt, ESIM, HBMP, and ULMFiT to provide a solid baseline for the future research. The best obtained test accuracy is 83.38% which shows that there is a big room for improving the current methods to be useful for real-world NLP applications in different languages. We also investigate the extent to which the models exploit superficial clues, also known as dataset biases, in FarsTail, and partition the test set into easy and hard subsets according to the success of biased models. The dataset is available at https://github.com/dml-qom/FarsTail.
In Syntology View this paper on Syntology: its repositories, every harvested function with whether it ran, its licence and the call to fetch it.
Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Datasets
Introduced by this paper, per the archive.
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Natural Language Inference | FarsTail | mBERT | % Test Accuracy | 83.38 | #1 of 10 | Archive leaderboard | report |
| Natural Language Inference | FarsTail | ParsBERT | % Test Accuracy | 82.99 | #2 of 10 | Archive leaderboard | report |
| Natural Language Inference | FarsTail | Translate-Source + fastText | % Test Accuracy | 78.13 | #3 of 10 | Archive leaderboard | report |
| Natural Language Inference | FarsTail | LSTM + BERT (concat) | % Test Accuracy | 75.83 | #4 of 10 | Archive leaderboard | report |
| Natural Language Inference | FarsTail | ESIM + BERT (FarsTail, MultiNLI) | % Test Accuracy | 74.62 | #5 of 10 | Archive leaderboard | report |
| Natural Language Inference | FarsTail | ULMFiT | % Test Accuracy | 72.44 | #6 of 10 | Archive leaderboard | report |
| Natural Language Inference | FarsTail | ESIM + fastText | % Test Accuracy | 71.16 | #7 of 10 | Archive leaderboard | report |
| Natural Language Inference | FarsTail | Translate-Target + fastText | % Test Accuracy | 70.46 | #8 of 10 | Archive leaderboard | report |
| Natural Language Inference | FarsTail | Decomposable Attention Model + word2vec | % Test Accuracy | 66.62 | #9 of 10 | Archive leaderboard | report |
| Natural Language Inference | FarsTail | HBMP + word2vec | % Test Accuracy | 66.04 | #10 of 10 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections