{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/stable-and-low-precision-training-for-large","title":"Stable and low-precision training for large-scale vision-language models","arxiv_id":"2304.13013","date":"2023-04-25","proceeding":"NeurIPS 2023 11","authors":["Mitchell Wortsman","Tim Dettmers","Luke Zettlemoyer","Ari Morcos","Ali Farhadi","Ludwig Schmidt"],"abstract":"We introduce new methods for 1) accelerating and 2) stabilizing training for large language-vision models. 1) For acceleration, we introduce SwitchBack, a linear layer for int8 quantized training which provides a speed-up of 13-25% while matching the performance of bfloat16 training within 0.1 percentage points for the 1B parameter CLIP ViT-Huge -- the largest int8 training to date. Our main focus is int8 as GPU support for float8 is rare, though we also analyze float8 training through simulation. While SwitchBack proves effective for float8, we show that standard techniques are also successful if the network is trained and initialized so that large feature magnitudes are discouraged, which we accomplish via layer-scale initialized with zeros. 2) For stability, we analyze loss spikes and find they consistently occur 1-8 iterations after the squared gradients become under-estimated by their AdamW second moment estimator. As a result, we recommend an AdamW-Adafactor hybrid which avoids loss spikes when training a CLIP ViT-Huge model and outperforms gradient clipping at the scales we test.","url_abs":"https://arxiv.org/abs/2304.13013v2","url_pdf":"https://arxiv.org/pdf/2304.13013v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"stable-and-low-precision-training-for-large","repo_url":"https://github.com/mlfoundations/open_clip","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":null,"task_name":"GPU"}],"methods":[{"method_slug":"adamw","method_name":"AdamW"},{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"gradient-clipping","method_name":"Gradient Clipping"},{"method_slug":"linear-layer","method_name":"Linear Layer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2304.13013","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2304.13013"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/TimDettmers/bitsandbytes","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/mlfoundations/open_clip","reach":null}],"summary":{"unverified":3},"by_repo_kind":{"found_in_text":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"419fc400d6ba95b4","entry":"Int8Params","repo":"TimDettmers/bitsandbytes","repo_kind":"found_in_text","path":"bitsandbytes/nn/modules.py","file_url":"https://github.com/TimDettmers/bitsandbytes/blob/HEAD/bitsandbytes/nn/modules.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"419fc400d6ba95b4"}},{"code_sha256_prefix":"610984316a762af9","entry":"Linear8bitLt","repo":"TimDettmers/bitsandbytes","repo_kind":"found_in_text","path":"bitsandbytes/nn/modules.py","file_url":"https://github.com/TimDettmers/bitsandbytes/blob/HEAD/bitsandbytes/nn/modules.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"610984316a762af9"}},{"code_sha256_prefix":"5bd645c0fa9375ce","entry":"maybe_rearrange_weight","repo":"TimDettmers/bitsandbytes","repo_kind":"found_in_text","path":"bitsandbytes/nn/modules.py","file_url":"https://github.com/TimDettmers/bitsandbytes/blob/HEAD/bitsandbytes/nn/modules.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5bd645c0fa9375ce"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}