{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/work-in-progress-linear-transformers-for","title":"Work in Progress: Linear Transformers for TinyML","arxiv_id":null,"date":"2024-03-25","proceeding":"Design, Automation & Test in Europe Conference & Exhibition (DATE) 2024 3","authors":["Moritz Scherer","Cristian Cioflan","Michele Magno","Luca Benini"],"abstract":"We present the WaveFormer, a neural network architecture based on a linear attention transformer to enable long sequence inference for TinyML devices. Waveformer achieves a new state-of-the-art accuracy of 98.8 % and 99.1 % on the Google Speech V2 keyword spotting (KWS) dataset for the 12 and 35 class problems with only 130 kB of weight storage, compatible with MCU class devices. Top-1 accuracy is improved by 0.1 and 0.9 percentage points while reducing the model size and number of operations by 2.5× and 4.7× compared to the state of the art. We also propose a hardware-friendly 8-bit integer quantization algorithm for the linear attention operator, enabling efficient deployment on low-cost, ultra-low-power microcontrollers without loss of accuracy.","url_abs":"https://ieeexplore.ieee.org/document/10546828","url_pdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10546828","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"keyword-spotting","task_name":"Keyword Spotting"},{"task_slug":null,"task_name":"Keyword Spotting on Google Speech Commands"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/keyword-spotting-on-google-speech-commands","task":"Keyword Spotting","dataset":"Google Speech Commands","model":"WaveFormer","rank_in_archive_order":20,"of":42,"metrics":{"Google Speech Commands V2 12":"98.8","Google Speech Commands V2 35":"99.1"},"uses_additional_data":false}],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}