Papers › Jam-ALT: A Formatting-Aware Lyrics Transcription Benchmark

Jam-ALT: A Formatting-Aware Lyrics Transcription Benchmark

23 Nov 2023arXiv:2311.13987archive 2025-07-28

Ondřej Cífka, Constantinos Dimitriou, Cheng-i Wang, Hendrik Schreiber, Luke Miner, Fabian-Robert Stöter

Current automatic lyrics transcription (ALT) benchmarks focus exclusively on word content and ignore the finer nuances of written lyrics including formatting and punctuation, which leads to a potential misalignment with the creative products of musicians and songwriters as well as listeners' experiences. For example, line breaks are important in conveying information about rhythm, emotional emphasis, rhyme, and high-level structure. To address this issue, we introduce Jam-ALT, a new lyrics transcription benchmark based on the JamendoLyrics dataset. Our contribution is twofold. Firstly, a complete revision of the transcripts, geared specifically towards ALT evaluation by following a newly created annotation guide that unifies the music industry's guidelines, covering aspects such as punctuation, line breaks, spelling, background vocals, and non-word sounds. Secondly, a suite of evaluation metrics designed, unlike the traditional word error rate, to capture such phenomena. We hope that the proposed benchmark contributes to the ALT task, enabling more precise and reliable assessments of transcription systems and enhancing the user experience in lyrics applications such as subtitle renderings for live captioning or karaoke.

PaperPDFCode

Syntology This paper's arXiv id is not in Syntology's Atlas id list (549,449 ids, as this build read them; the read time is in the build record), so this page links to no Syntology page for it. The list is not the whole graph; Syntology's feed shows papers as they enter the graph.

Code

audioshake/alt-eval officialmentioned in paperMITSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Robust Speech Recognition via Large-Scale Weak Supervision (arXiv:2212.04356; that paper's own run record: 49 ran (of which 0 constructed an object rather than computing a result; 36 with no instrument failure: 4 honoured, 1 violated, 31 with no contract checked; 13 where Syntology's instrument failed) · 10 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Automatic Lyrics TranscriptionRhythm

Datasets

Introduced by this paper, per the archive.

Jam-ALT

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Automatic Lyrics Transcription Jam-ALT AudioShake v1 Case Error Rate 3.4 #2 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT AudioShake v1 Line break F1 82.3 #2 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT AudioShake v1 Parenthesis F-1 29.4 #2 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT AudioShake v1 Punctuation F1 50.5 #2 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT AudioShake v1 Section break F1 72.1 #2 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT AudioShake v1 Word Error Rate (WER) 26.0 #2 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Whisper v3 Case Error Rate 4.3 #7 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Whisper v3 Line break F1 73.5 #7 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Whisper v3 Punctuation F1 41.6 #7 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Whisper v3 Section break F1 1.0 #7 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Whisper v3 Word Error Rate (WER) 35.5 #7 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Whisper v2 Case Error Rate 4.5 #8 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Whisper v2 Line break F1 69.3 #8 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Whisper v2 Punctuation F1 41.7 #8 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Whisper v2 Section break F1 3.3 #8 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Whisper v2 Word Error Rate (WER) 35.7 #8 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Whisper v2 +demucs Case Error Rate 5.3 #10 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Whisper v2 +demucs Line break F1 61.2 #10 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Whisper v2 +demucs Punctuation F1 28.0 #10 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Whisper v2 +demucs Word Error Rate (WER) 44.0 #10 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Whisper v3 +demucs Case Error Rate 3.8 #13 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Whisper v3 +demucs Line break F1 65.7 #13 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Whisper v3 +demucs Punctuation F1 29.0 #13 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Whisper v3 +demucs Word Error Rate (WER) 47.9 #13 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT English AudioShake v1 Case Error Rate 3.4 #2 of 18 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT English AudioShake v1 Line break F-1 80.7 #2 of 18 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT English AudioShake v1 Parenthesis F-1 32.4 #2 of 18 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT English AudioShake v1 Punctuation F-1 59.0 #2 of 18 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT English AudioShake v1 Section break F-1 77.4 #2 of 18 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT English AudioShake v1 Word Error Rate (WER) 22.1 #2 of 18 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT English LyricWhiz Case Error Rate 3.5 #4 of 18 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT English LyricWhiz Line break F-1 74.0 #4 of 18 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT English LyricWhiz Punctuation F-1 34.0 #4 of 18 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT English LyricWhiz Section break F-1 1.4 #4 of 18 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT English LyricWhiz Word Error Rate (WER) 24.6 #4 of 18 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT English Whisper v2 +demucs Case Error Rate 5.3 #5 of 18 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT English Whisper v2 +demucs Line break F-1 53.8 #5 of 18 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT English Whisper v2 +demucs Punctuation F-1 39.2 #5 of 18 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT English Whisper v2 +demucs Word Error Rate (WER) 32.3 #5 of 18 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT English Whisper v3 Case Error Rate 4.8 #10 of 18 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT English Whisper v3 Line break F-1 71.5 #10 of 18 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT English Whisper v3 Punctuation F-1 40.9 #10 of 18 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT English Whisper v3 Section break F-1 2.6 #10 of 18 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT English Whisper v3 Word Error Rate (WER) 37.7 #10 of 18 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT English Whisper v3 +demucs Case Error Rate 4.1 #14 of 18 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT English Whisper v3 +demucs Line break F-1 66.8 #14 of 18 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT English Whisper v3 +demucs Punctuation F-1 23.3 #14 of 18 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT English Whisper v3 +demucs Word Error Rate (WER) 43.0 #14 of 18 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT English Whisper v2 Case Error Rate 3.5 #16 of 18 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT English Whisper v2 Line break F-1 63.0 #16 of 18 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT English Whisper v2 Punctuation F-1 31.3 #16 of 18 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT English Whisper v2 Section break F-1 11.2 #16 of 18 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT English Whisper v2 Word Error Rate (WER) 43.8 #16 of 18 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT French Whisper v2 Case Error Rate 3.2 #4 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT French Whisper v2 Line break F-1 73.4 #4 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT French Whisper v2 Punctuation F-1 45.8 #4 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT French Whisper v2 Section break F-1 1.4 #4 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT French Whisper v2 Word Error Rate (WER) 27.7 #4 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT French Whisper v3 Case Error Rate 3.3 #7 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT French Whisper v3 Line break F-1 77.8 #7 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT French Whisper v3 Punctuation F-1 42.4 #7 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT French Whisper v3 Word Error Rate (WER) 34.7 #7 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT French AudioShake v1 Case Error Rate 2.0 #8 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT French AudioShake v1 Line break F-1 84.9 #8 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT French AudioShake v1 Parenthesis F-1 41.3 #8 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT French AudioShake v1 Punctuation F-1 45.8 #8 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT French AudioShake v1 Section break F-1 72.5 #8 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT French AudioShake v1 Word Error Rate (WER) 34.9 #8 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT French Whisper v2 +demucs Case Error Rate 3.2 #11 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT French Whisper v2 +demucs Line break F-1 66.1 #11 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT French Whisper v2 +demucs Punctuation F-1 34.9 #11 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT French Whisper v2 +demucs Word Error Rate (WER) 43.3 #11 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT French Whisper v3 +demucs Case Error Rate 3.2 #14 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT French Whisper v3 +demucs Line break F-1 69.4 #14 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT French Whisper v3 +demucs Punctuation F-1 30.9 #14 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT French Whisper v3 +demucs Word Error Rate (WER) 44.9 #14 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT German AudioShake v1 Case Error Rate 4.1 #4 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT German AudioShake v1 Line break F-1 81.2 #4 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT German AudioShake v1 Parenthesis F-1 8.1 #4 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT German AudioShake v1 Punctuation F-1 48.5 #4 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT German AudioShake v1 Section break F-1 69.2 #4 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT German AudioShake v1 Word Error Rate (WER) 24.4 #4 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT German Whisper v3 Case Error Rate 4.0 #7 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT German Whisper v3 Line break F-1 71.2 #7 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT German Whisper v3 Punctuation F-1 41.2 #7 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT German Whisper v3 Section break F-1 1.2 #7 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT German Whisper v3 Word Error Rate (WER) 40.7 #7 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT German Whisper v3 +demucs Case Error Rate 4.4 #10 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT German Whisper v3 +demucs Line break F-1 72.0 #10 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT German Whisper v3 +demucs Punctuation F-1 34.0 #10 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT German Whisper v3 +demucs Word Error Rate (WER) 43.5 #10 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT German Whisper v2 Case Error Rate 5.3 #11 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT German Whisper v2 Line break F-1 69.9 #11 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT German Whisper v2 Punctuation F-1 38.7 #11 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT German Whisper v2 Word Error Rate (WER) 45.4 #11 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT German Whisper v2 +demucs Case Error Rate 5.9 #16 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT German Whisper v2 +demucs Line break F-1 67.5 #16 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT German Whisper v2 +demucs Punctuation F-1 30.2 #16 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT German Whisper v2 +demucs Word Error Rate (WER) 65.2 #16 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Spanish AudioShake v1 Case Error Rate 4.1 #4 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Spanish AudioShake v1 Line break F-1 82.7 #4 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Spanish AudioShake v1 Parenthesis F-1 38.0 #4 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Spanish AudioShake v1 Punctuation F-1 47.8 #4 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Spanish AudioShake v1 Section break F-1 69.6 #4 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Spanish AudioShake v1 Word Error Rate (WER) 22.5 #4 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Spanish Whisper v2 Case Error Rate 6.5 #5 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Spanish Whisper v2 Line break F-1 71.7 #5 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Spanish Whisper v2 Punctuation F-1 50.0 #5 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Spanish Whisper v2 Section break F-1 3.1 #5 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Spanish Whisper v2 Word Error Rate (WER) 25.7 #5 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Spanish Whisper v3 Case Error Rate 5.0 #8 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Spanish Whisper v3 Line break F-1 73.7 #8 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Spanish Whisper v3 Punctuation F-1 41.9 #8 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Spanish Whisper v3 Word Error Rate (WER) 28.6 #8 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Spanish Whisper v2 +demucs Case Error Rate 7.1 #10 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Spanish Whisper v2 +demucs Line break F-1 56.4 #10 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Spanish Whisper v2 +demucs Punctuation F-1 17.2 #10 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Spanish Whisper v2 +demucs Word Error Rate (WER) 38.8 #10 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Spanish Whisper v3 +demucs Case Error Rate 3.6 #14 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Spanish Whisper v3 +demucs Line break F-1 52.4 #14 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Spanish Whisper v3 +demucs Punctuation F-1 28.7 #14 of 16 Archive leaderboard report
Automatic Lyrics Transcription Jam-ALT Spanish Whisper v3 +demucs Word Error Rate (WER) 61.5 #14 of 16 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Focus

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections