{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2510-27527","title":"TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Control","arxiv_id":"2510.27527","date":"2025-10-31","proceeding":"ICML","authors":["Yuxiang Chen","Yifan Liu","Xiaoming Xu","Pengle Zhang","Michael Beyer","Martin Rapp","Jun Zhu","Jianfei Chen"],"abstract":"Large Language Models (LLMs) training is prohibitively expensive, driving interest in low-precision fully-quantized training (FQT). While novel 4-bit formats like NVFP4 offer substantial efficiency gains, achieving near-lossless training at such low precision remains challenging. We introduce TetraJet-v2, an end-to-end 4-bit FQT method that leverages NVFP4 for activations, weights, and gradients in all linear layers. We identify two critical issues hindering low-precision LLM training: weight oscillation and outliers. To address these, we propose: 1) an unbiased double-block quantization method for NVFP4 linear layers with practically optimal convergence in LLM training, 2) OsciReset, the first effective algorithm to suppress LLMs' weight oscillation bottleneck, and 3) OutControl, a mix-precision algorithm to retain outlier accuracy. TetraJet-v2 outperforms prior methods on FP4 pre-training for LLMs across models up to 370M parameters trained up to 212B tokens, reducing the performance gap to BF16 by an average of 51.3% while enabling an 1.67x end-to-end speedup over FP8. The code is available at https://github.com/thu-ml/TetraJet-v2-NVFP4Training.","url_abs":"https://arxiv.org/abs/2510.27527","url_pdf":"https://arxiv.org/pdf/2510.27527","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2510.27527","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2510.27527"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/thu-ml/TetraJet-v2-NVFP4Training","reach":null}],"summary":{"ran":6,"unverified":9},"by_repo_kind":{"found_in_text":{"samples":15,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fb15d600c2b76528","entry":"NVFP4_Outlier_Selection_Config","repo":"thu-ml/TetraJet-v2-NVFP4Training","repo_kind":"found_in_text","path":"olmo2-training/olmo/quantization_real/oscillation_reset.py","file_url":"https://github.com/thu-ml/TetraJet-v2-NVFP4Training/blob/HEAD/olmo2-training/olmo/quantization_real/oscillation_reset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fb15d600c2b76528"}},{"code_sha256_prefix":"1f2a77a5e466301a","entry":"NVFP4_QuantizerConfig","repo":"thu-ml/TetraJet-v2-NVFP4Training","repo_kind":"found_in_text","path":"olmo2-training/olmo/quantization_real/oscillation_reset.py","file_url":"https://github.com/thu-ml/TetraJet-v2-NVFP4Training/blob/HEAD/olmo2-training/olmo/quantization_real/oscillation_reset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1f2a77a5e466301a"}},{"code_sha256_prefix":"8eb1e855106805d6","entry":"NVFP4_TetraJetv2_MatMulFunc_BwdAllRHT16SQ_Padding128","repo":"thu-ml/TetraJet-v2-NVFP4Training","repo_kind":"found_in_text","path":"olmo2-training/olmo/quantization_real/oscillation_reset.py","file_url":"https://github.com/thu-ml/TetraJet-v2-NVFP4Training/blob/HEAD/olmo2-training/olmo/quantization_real/oscillation_reset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8eb1e855106805d6"}},{"code_sha256_prefix":"65ce9d1406995870","entry":"OLMoConfigurationError","repo":"thu-ml/TetraJet-v2-NVFP4Training","repo_kind":"found_in_text","path":"olmo2-training/olmo/quantization_real/oscillation_reset.py","file_url":"https://github.com/thu-ml/TetraJet-v2-NVFP4Training/blob/HEAD/olmo2-training/olmo/quantization_real/oscillation_reset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"65ce9d1406995870"}},{"code_sha256_prefix":"cc2604f080112c77","entry":"OLMoError","repo":"thu-ml/TetraJet-v2-NVFP4Training","repo_kind":"found_in_text","path":"olmo2-training/olmo/quantization_real/oscillation_reset.py","file_url":"https://github.com/thu-ml/TetraJet-v2-NVFP4Training/blob/HEAD/olmo2-training/olmo/quantization_real/oscillation_reset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cc2604f080112c77"}},{"code_sha256_prefix":"99ef28b8606875c6","entry":"OscillationPerturb_Config","repo":"thu-ml/TetraJet-v2-NVFP4Training","repo_kind":"found_in_text","path":"olmo2-training/olmo/quantization_real/oscillation_reset.py","file_url":"https://github.com/thu-ml/TetraJet-v2-NVFP4Training/blob/HEAD/olmo2-training/olmo/quantization_real/oscillation_reset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"99ef28b8606875c6"}},{"code_sha256_prefix":"60cc4ff441d3ab2c","entry":"BaseConfig","repo":"thu-ml/TetraJet-v2-NVFP4Training","repo_kind":"found_in_text","path":"olmo2-training/olmo/quantization_real/oscillation_reset.py","file_url":"https://github.com/thu-ml/TetraJet-v2-NVFP4Training/blob/HEAD/olmo2-training/olmo/quantization_real/oscillation_reset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"60cc4ff441d3ab2c"}},{"code_sha256_prefix":"ff027b8d502fb52d","entry":"NVFP4_Linear_RealQuant_WeightQuantOnline","repo":"thu-ml/TetraJet-v2-NVFP4Training","repo_kind":"found_in_text","path":"olmo2-training/olmo/quantization_real/oscillation_reset.py","file_url":"https://github.com/thu-ml/TetraJet-v2-NVFP4Training/blob/HEAD/olmo2-training/olmo/quantization_real/oscillation_reset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ff027b8d502fb52d"}},{"code_sha256_prefix":"c5d24254996b4107","entry":"NVFP4_mix_MXFP8_Linear_RealQuant_WeightQuantOnline","repo":"thu-ml/TetraJet-v2-NVFP4Training","repo_kind":"found_in_text","path":"olmo2-training/olmo/quantization_real/oscillation_reset.py","file_url":"https://github.com/thu-ml/TetraJet-v2-NVFP4Training/blob/HEAD/olmo2-training/olmo/quantization_real/oscillation_reset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c5d24254996b4107"}},{"code_sha256_prefix":"8d426745e6c66d1c","entry":"NVFP4_mix_MXFP8_TetraJetv2_MatMulFunc_BwdAllRHT16SQ_Padding128","repo":"thu-ml/TetraJet-v2-NVFP4Training","repo_kind":"found_in_text","path":"olmo2-training/olmo/quantization_real/oscillation_reset.py","file_url":"https://github.com/thu-ml/TetraJet-v2-NVFP4Training/blob/HEAD/olmo2-training/olmo/quantization_real/oscillation_reset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8d426745e6c66d1c"}},{"code_sha256_prefix":"eacce6861a053fd0","entry":"check_mix_channel_done","repo":"thu-ml/TetraJet-v2-NVFP4Training","repo_kind":"found_in_text","path":"olmo2-training/olmo/quantization_real/oscillation_reset.py","file_url":"https://github.com/thu-ml/TetraJet-v2-NVFP4Training/blob/HEAD/olmo2-training/olmo/quantization_real/oscillation_reset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eacce6861a053fd0"}},{"code_sha256_prefix":"1717205ffec8f0af","entry":"check_quant_linear","repo":"thu-ml/TetraJet-v2-NVFP4Training","repo_kind":"found_in_text","path":"olmo2-training/olmo/quantization_real/oscillation_reset.py","file_url":"https://github.com/thu-ml/TetraJet-v2-NVFP4Training/blob/HEAD/olmo2-training/olmo/quantization_real/oscillation_reset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1717205ffec8f0af"}},{"code_sha256_prefix":"1fa4156825dcda41","entry":"check_quant_mix_linear","repo":"thu-ml/TetraJet-v2-NVFP4Training","repo_kind":"found_in_text","path":"olmo2-training/olmo/quantization_real/oscillation_reset.py","file_url":"https://github.com/thu-ml/TetraJet-v2-NVFP4Training/blob/HEAD/olmo2-training/olmo/quantization_real/oscillation_reset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1fa4156825dcda41"}},{"code_sha256_prefix":"0f1fb3f207b264a2","entry":"fp4_elements_centralize_DistRatio","repo":"thu-ml/TetraJet-v2-NVFP4Training","repo_kind":"found_in_text","path":"olmo2-training/olmo/quantization_real/oscillation_reset.py","file_url":"https://github.com/thu-ml/TetraJet-v2-NVFP4Training/blob/HEAD/olmo2-training/olmo/quantization_real/oscillation_reset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0f1fb3f207b264a2"}},{"code_sha256_prefix":"5f980dcdfadd500b","entry":"query_quant_risk_ratio","repo":"thu-ml/TetraJet-v2-NVFP4Training","repo_kind":"found_in_text","path":"olmo2-training/olmo/quantization_real/oscillation_reset.py","file_url":"https://github.com/thu-ml/TetraJet-v2-NVFP4Training/blob/HEAD/olmo2-training/olmo/quantization_real/oscillation_reset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5f980dcdfadd500b"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_api"},"syntology_extracted_results":null}