{"url":"/method/1-bit-adam","slug":"1-bit-adam","name":"1-bit Adam","full_name":"1-bit Adam","full_name_withheld":false,"description_markdown":"**1-bit Adam** is a [stochastic optimization](https://paperswithcode.com/methods/category/stochastic-optimization) technique that is a variant of [ADAM](https://paperswithcode.com/method/adam) with error-compensated 1-bit compression, based on finding that Adam's variance term becomes stable at an early stage. First vanilla Adam is used for a few epochs as a warm-up. After the warm-up stage, the compression stage starts and we stop updating the variance term $\\mathbf{v}$ and use it as a fixed precondition. At the compression stage, we communicate based on the momentum applied with error-compensated 1-bit compression. The momentums are quantized into 1-bit representation (the sign of each element). Accompanying the vector, a scaling factor is computed as $\\frac{\\text { magnitude of compensated gradient }}{\\text { magnitude of quantized gradient }}$. This scaling factor ensures that the compressed momentum has the same magnitude as the uncompressed momentum. This 1-bit compression could reduce the communication cost by $97 \\%$ and $94 \\%$ compared to the original float 32 and float 16 training, respectively.","description_state":"present","introduced_year":null,"introduced_by":{"title":"1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed","paper":"/paper/1-bit-adam-communication-efficient-large","first_author":"Hanlin Tang","n_authors":9,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/1-bit-adam-communication-efficient-large"},"source":{"url":"https://arxiv.org/abs/2102.02888v2","title":"1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Large Batch Optimization","url":"/methods/category/large-batch-optimization","pwc_aliases":[]},{"area":"General","area_id":"general","collection":"Stochastic Optimization","url":"/methods/category/stochastic-optimization","pwc_aliases":[]}],"n_papers_tagged":40,"archive_num_papers":40,"papers_newest_first":[{"paper":null,"title":"ARWI: Arabic Write and Improve","date":"2025-04-16","arxiv_id":"2504.11814","n_code_links":0,"syntology":null},{"paper":"/paper/deepseek-r1-incentivizing-reasoning","title":"DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning","date":"2025-01-22","arxiv_id":"2501.12948","n_code_links":4,"syntology":null},{"paper":"/paper/showui-one-vision-language-action-model-for","title":"ShowUI: One Vision-Language-Action Model for GUI Visual Agent","date":"2024-11-26","arxiv_id":"2411.17465","n_code_links":1,"syntology":{"ran":1,"of":16,"unverified":15,"pointer_only":0}},{"paper":"/paper/2408-01800","title":"MiniCPM-V: A GPT-4V Level MLLM on Your Phone","date":"2024-08-03","arxiv_id":"2408.01800","n_code_links":2,"syntology":{"ran":9,"of":14,"unverified":5,"pointer_only":0}},{"paper":"/paper/yolov10-real-time-end-to-end-object-detection","title":"YOLOv10: Real-Time End-to-End Object Detection","date":"2024-05-23","arxiv_id":"2405.14458","n_code_links":3,"syntology":{"ran":0,"of":3,"unverified":3,"pointer_only":0}},{"paper":"/paper/fast-fishing-approximating-bait-for-efficient","title":"Fast Fishing: Approximating BAIT for Efficient and Scalable Deep Active Image Classification","date":"2024-04-13","arxiv_id":"2404.08981","n_code_links":1,"syntology":null},{"paper":"/paper/tracking-anything-in-high-quality","title":"Tracking Anything in High Quality","date":"2023-07-26","arxiv_id":"2307.13974","n_code_links":1,"syntology":null},{"paper":"/paper/dinov2-learning-robust-visual-features","title":"DINOv2: Learning Robust Visual Features without Supervision","date":"2023-04-14","arxiv_id":"2304.07193","n_code_links":26,"syntology":{"ran":21,"of":46,"unverified":25,"pointer_only":12}},{"paper":"/paper/chatdoctor-a-medical-chat-model-fine-tuned-on","title":"ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge","date":"2023-03-24","arxiv_id":"2303.14070","n_code_links":1,"syntology":null},{"paper":"/paper/cctv-gun-benchmarking-handgun-detection-in","title":"CCTV-Gun: Benchmarking Handgun Detection in CCTV Images","date":"2023-03-19","arxiv_id":"2303.10703","n_code_links":1,"syntology":null},{"paper":"/paper/musiclm-generating-music-from-text","title":"MusicLM: Generating Music From Text","date":"2023-01-26","arxiv_id":"2301.11325","n_code_links":5,"syntology":{"ran":6,"of":16,"unverified":10,"pointer_only":0}},{"paper":null,"title":"Dynamic Gradient Reactivation for Backward Compatible Person Re-identification","date":"2022-07-12","arxiv_id":"2207.05658","n_code_links":0,"syntology":null},{"paper":"/paper/solving-quantitative-reasoning-problems-with","title":"Solving Quantitative Reasoning Problems with Language Models","date":"2022-06-29","arxiv_id":"2206.14858","n_code_links":1,"syntology":null},{"paper":"/paper/maximizing-communication-efficiency-for-large","title":"Maximizing Communication Efficiency for Large-scale Training via 0/1 Adam","date":"2022-02-12","arxiv_id":"2202.06009","n_code_links":1,"syntology":null},{"paper":"/paper/fer2013-recognition-resnet18-with-tricks","title":"Fer2013 Recognition - ResNet18 With Tricks","date":"2021-12-29","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":"/paper/nuwa-visual-synthesis-pre-training-for-neural","title":"NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion","date":"2021-11-24","arxiv_id":"2111.12417","n_code_links":1,"syntology":{"ran":3,"of":3,"unverified":0,"pointer_only":2}},{"paper":"/paper/1-bit-lamb-communication-efficient-large","title":"1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed","date":"2021-04-13","arxiv_id":"2104.06069","n_code_links":1,"syntology":null},{"paper":"/paper/layoutparser-a-unified-toolkit-for-deep","title":"LayoutParser: A Unified Toolkit for Deep Learning Based Document Image Analysis","date":"2021-03-29","arxiv_id":"2103.15348","n_code_links":6,"syntology":null},{"paper":null,"title":"What is it Like to Be a Bot: Simulated, Situated, Structurally Coherent Qualia (S3Q) Theory of Consciousness","date":"2021-03-13","arxiv_id":"2103.12638","n_code_links":0,"syntology":null},{"paper":"/paper/zero-shot-text-to-image-generation","title":"Zero-Shot Text-to-Image Generation","date":"2021-02-24","arxiv_id":"2102.12092","n_code_links":12,"syntology":{"ran":7,"of":7,"unverified":0,"pointer_only":3}},{"paper":"/paper/1-bit-adam-communication-efficient-large","title":"1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed","date":"2021-02-04","arxiv_id":"2102.02888","n_code_links":2,"syntology":null},{"paper":"/paper/infinite-nature-perpetual-view-generation-of","title":"Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Image","date":"2020-12-17","arxiv_id":"2012.09855","n_code_links":1,"syntology":null},{"paper":null,"title":"In silico comparison of spike protein-ACE2 binding affinities across species; significance for the possible origin of the SARS-CoV-2 virus","date":"2020-05-13","arxiv_id":"2005.06199","n_code_links":0,"syntology":null},{"paper":"/paper/mti-net-multi-scale-task-interaction-networks","title":"MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning","date":"2020-01-19","arxiv_id":"2001.06902","n_code_links":1,"syntology":null},{"paper":"/paper/advances-and-open-problems-in-federated","title":"Advances and Open Problems in Federated Learning","date":"2019-12-10","arxiv_id":"1912.04977","n_code_links":9,"syntology":{"ran":0,"of":9,"unverified":9,"pointer_only":0}},{"paper":"/paper/191111236","title":"RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds","date":"2019-11-25","arxiv_id":"1911.11236","n_code_links":9,"syntology":{"ran":1,"of":4,"unverified":3,"pointer_only":0}},{"paper":"/paper/green-ai","title":"Green AI","date":"2019-07-22","arxiv_id":"1907.10597","n_code_links":2,"syntology":{"ran":0,"of":1,"unverified":1,"pointer_only":0}},{"paper":"/paper/self-supervised-representation-learning-by","title":"Self-Supervised Representation Learning by Rotation Feature Decoupling","date":"2019-06-01","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":"/paper/a-comprehensive-overhaul-of-feature","title":"A Comprehensive Overhaul of Feature Distillation","date":"2019-04-03","arxiv_id":"1904.01866","n_code_links":2,"syntology":{"ran":1,"of":6,"unverified":5,"pointer_only":0}},{"paper":null,"title":"Analysis of the Generalization Error: Empirical Risk Minimization over Deep Artificial Neural Networks Overcomes the Curse of Dimensionality in the Numerical Approximation of Black-Scholes Partial Differential Equations","date":"2018-09-09","arxiv_id":"1809.03062","n_code_links":0,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/semantic-segmentation","name":"Semantic Segmentation","papers":5},{"task":"/task/image-generation","name":"Image Generation","papers":4},{"task":"/task/2d-object-detection","name":"2D Object Detection","papers":3},{"task":"/task/image-classification","name":"Image Classification","papers":3},{"task":"/task/object-detection","name":"Object Detection","papers":3},{"task":"/task/retrieval","name":"Retrieval","papers":3},{"task":"/task/object-detection-1","name":"object-detection","papers":3},{"task":"/task/2d-human-pose-estimation","name":"2D Human Pose Estimation","papers":2},{"task":"/task/deep-learning","name":"Deep Learning","papers":2},{"task":"/task/classification","name":"General Classification","papers":2},{"task":"/task/image-retrieval","name":"Image Retrieval","papers":2},{"task":"/task/language-modeling","name":"Language Modeling","papers":2},{"task":"/task/language-modelling","name":"Language Modelling","papers":2},{"task":"/task/large-language-model","name":"Large Language Model","papers":2},{"task":"/task/multi-task-language-understanding","name":"Multi-task Language Understanding","papers":2},{"task":"/task/object","name":"Object","papers":2},{"task":"/task/text-to-image-generation-1","name":"Text to Image Generation","papers":2},{"task":"/task/text-to-image-generation","name":"Text-to-Image Generation","papers":2},{"task":"/task/video-generation","name":"Video Generation","papers":2},{"task":"/task/image-classification","name":"image-classification","papers":2}],"tasks_shown":20,"n_tasks":96,"usage_by_year":[{"year":"1986","papers":1},{"year":"2015","papers":1},{"year":"2016","papers":1},{"year":"2017","papers":4},{"year":"2018","papers":4},{"year":"2019","papers":5},{"year":"2020","papers":3},{"year":"2021","papers":7},{"year":"2022","papers":3},{"year":"2023","papers":5},{"year":"2024","papers":4},{"year":"2025","papers":2}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/1-bit-adam"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}