{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/self-attentional-acoustic-models","title":"Self-Attentional Acoustic Models","arxiv_id":"1803.09519","date":"2018-03-26","proceeding":null,"authors":["Matthias Sperber","Jan Niehues","Graham Neubig","Sebastian Stüker","Alex Waibel"],"abstract":"Self-attention is a method of encoding sequences of vectors by relating these\nvectors to each-other based on pairwise similarities. These models have\nrecently shown promising results for modeling discrete sequences, but they are\nnon-trivial to apply to acoustic modeling due to computational and modeling\nissues. In this paper, we apply self-attention to acoustic modeling, proposing\nseveral improvements to mitigate these issues: First, self-attention memory\ngrows quadratically in the sequence length, which we address through a\ndownsampling technique. Second, we find that previous approaches to incorporate\nposition information into the model are unsuitable and explore other\nrepresentations and hybrid models to this end. Third, to stress the importance\nof local context in the acoustic signal, we propose a Gaussian biasing approach\nthat allows explicit control over the context range. Experiments find that our\nmodel approaches a strong baseline based on LSTMs with network-in-network\nconnections while being much faster to compute. Besides speed, we find that\ninterpretability is a strength of self-attentional acoustic models, and\ndemonstrate that self-attention heads learn a linguistically plausible division\nof labor.","url_abs":"http://arxiv.org/abs/1803.09519v2","url_pdf":"http://arxiv.org/pdf/1803.09519v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"self-attentional-acoustic-models","repo_url":"https://github.com/bagequan/tencent-transformer-with-disagreement","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1803.09519","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}