Papers › BWTA: Accurate and Efficient Binarized Transformer by Algorithm-Hardware Co-design
BWTA: Accurate and Efficient Binarized Transformer by Algorithm-Hardware Co-design
Yifu Ding, Xianglong Liu, Shenghao Jin, Jinyang Guo, Jiwen Lu
Title, abstract, authors and date from arXiv's metadata (CC0); this paper is not in the Papers with Code archive (frozen 2025-07-28).
Ultra low-bit quantization brings substantial efficiency for Transformer-based models, but the accuracy degradation and limited GPU support hinder its wide usage. In this paper, we analyze zero-point distortion in binarization and propose a Binary Weights & Ternary Activations (BWTA) quantization scheme, which projects tiny values to zero and preserves the accuracy of extremely low-bit models. For training, we propose Smooth Multi-Stage Quantization, combining a Levelwise Degradation Strategy and a Magnitude-Alignment Projection Factor to enable stable and fast convergence. For inference, we develop a BWTA MatMul CUDA kernel with instruction-level parallel bit-packing and comprehensive binary/ternary MatMul implementations for both linear and attention operators, allowing seamless integration across Transformer architectures. Experiments show that BWTA approaches full-precision performance for BERT, with an average 3.5% drop on GLUE and less than 2% drop on five tasks, and achieves comparable perplexity and accuracy for LLMs. In efficiency, it delivers 16 to 24 times kernel-level speedup over FP16 on NVIDIA GPUs, and 216 to 330 tokens/s end-to-end prefill speedup with lower memory footprint on LLMs. As an algorithm-hardware co-design, BWTA demonstrates practical, low-latency ultra-low-bit inference without sacrificing model quality.
In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.
For agents, on Syntology's MCP service (how to connect):
get_citation_path(paper_1="2604.03957", paper_2="…")with another paper's arXiv id or titleget_concepts_for_paper(arxiv_id="2604.03957")
Code
Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Syntology holds the repository link but has not harvested or run code from it.
Results from the paper
The Papers with Code archive ends with its 2025-07-28 snapshot. This paper's arXiv identifier, 2604.03957, was issued in April 2026, after that date, so the archive has no leaderboard rows for it.
Placed on leaderboards by Syntology Syntology
Syntology's extractor, the model Claude Sonnet 4.5, judged this paper's tables to report on 8 leaderboards (a model's judgement, not a result) and has not placed the paper on any of them: Natural Language Inference · MultiNLI (refused by a rule: the model gave no value to check against the cell); Natural Language Inference · QNLI (refused by a rule: the table's column header does not name the leaderboard's metric); Natural Language Inference · RTE (refused by a rule: the table's column header does not name the leaderboard's metric); parameter-efficient fine-tuning · BoolQ (rejected by the independent check); Text Classification · GLUE MRPC (refused by a rule: the quoted value appears more than once in the row); Text Classification · GLUE RTE (refused by a rule: the table's column header does not name the leaderboard's metric); Text Classification · GLUE SST2 (refused by a rule: the table's column header does not name the leaderboard's metric); Text Classification · SST-2 (refused by a rule: the table's column header does not name the leaderboard's metric). What is not shown.
Syntology has checked 8,886 of the 18,637 papers on this site that are newer than the archive (for 3,717 of them no archive leaderboard matched the paper's tables, so there was nothing further to check); 733 were read and have nothing to place (arXiv has no HTML version of the paper, or that version has no tables), 41 could not be read (the extractor's reply could not be parsed), and results from the other 8,977 appear after they are checked.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections