Papers › GhazalBench: Canonical Verse Access in LLMs across Persian Ghazals and Shakespearean Sonnets
GhazalBench: Canonical Verse Access in LLMs across Persian Ghazals and Shakespearean Sonnets
Ghazal Kalhor, Yadollah Yaghoobzadeh
Title, abstract, authors and date from arXiv's metadata (CC0); this paper is not in the Papers with Code archive (frozen 2025-07-28).
Persian poetry plays an active role in Iranian cultural practice, where verses by canonical poets such as Hafez and Saadi are frequently quoted, paraphrased, or completed from incomplete cues. Supporting such interactions requires language models to reliably access canonical verses from semantic and lexical information. We introduce GhazalBench, a benchmark for evaluating how large language models (LLMs) access the canonical surface forms of Persian ghazal under usage-grounded conditions. Unlike prior work that primarily treats memorization as a liability, GhazalBench studies settings in which access to exact wording is functionally important. The benchmark evaluates completion and recognition under varied semantic and lexical cues. Across proprietary and open-weight multilingual LLMs, exact completion remains challenging, while recognition is substantially stronger. Performance also varies across poets and model families. Parallel experiments on Shakespearean sonnets yield markedly higher completion for several models, consistent with differences in exposure to canonical texts. {Additional experiments suggest that post-training may reduce direct continuation-based access to canonical text.} Our findings motivate evaluation frameworks that distinguish production from recognition and assess access to culturally significant canonical texts. GhazalBench is available at https://github.com/kalhorghazal/GhazalBench.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Syntology holds the repository link but has not harvested or run code from it.
Results from the paper
The Papers with Code archive ends with its 2025-07-28 snapshot. This paper's arXiv identifier, 2603.09979, was issued in March 2026, after that date, so the archive has no leaderboard rows for it.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections