{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2510-24061","title":"FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmetic","arxiv_id":"2510.24061","date":"2025-10-28","proceeding":"NeurIPS","authors":["Kanghyun Choi","Hyeyoon Lee","SunJong Park","Dain Kwon","Jinho Lee"],"abstract":"Low-bit floating-point (FP) formats, such as FP8, provide significant acceleration and memory savings in model training thanks to native hardware support on modern GPUs and NPUs. However, we analyze that FP8 quantization offers speedup primarily for large-dimensional matrix multiplications, while inherent quantization overheads diminish speedup when applied to low-rank adaptation (LoRA), which uses small-dimensional matrices for efficient fine-tuning of large language models (LLMs). To address this limitation, we propose FALQON, a novel framework that eliminates the quantization overhead from separate LoRA computational paths by directly merging LoRA adapters into an FP8-quantized backbone during fine-tuning. Furthermore, we reformulate the forward and backward computations for merged adapters to significantly reduce quantization overhead, and introduce a row-wise proxy update mechanism that efficiently integrates substantial updates into the quantized backbone. Experimental evaluations demonstrate that FALQON achieves approximately a 3$\\times$ training speedup over existing quantized LoRA methods with a similar level of accuracy, providing a practical solution for efficient large-scale model fine-tuning. Moreover, FALQON's end-to-end FP8 workflow removes the need for post-training quantization, facilitating efficient deployment. Code is available at https://github.com/iamkanghyunchoi/falqon.","url_abs":"https://arxiv.org/abs/2510.24061","url_pdf":"https://arxiv.org/pdf/2510.24061","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2510.24061","atlas_url":"https://app.syntology.ai/?focus=2510.24061","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2510.24061"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/iamkanghyunchoi/falqon","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7,"unverified":8},"by_repo_kind":{"found_in_text":{"samples":15,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ab306165fa74656c","entry":"build_tip_message","repo":"iamkanghyunchoi/falqon","repo_kind":"found_in_text","path":"transformers/utils/deprecate_models.py","file_url":"https://github.com/iamkanghyunchoi/falqon/blob/HEAD/transformers/utils/deprecate_models.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ab306165fa74656c"}},{"code_sha256_prefix":"3d999b95009c7537","entry":"get_config_class_from_processor_class","repo":"iamkanghyunchoi/falqon","repo_kind":"found_in_text","path":"transformers/utils/create_dummy_models.py","file_url":"https://github.com/iamkanghyunchoi/falqon/blob/HEAD/transformers/utils/create_dummy_models.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3d999b95009c7537"}},{"code_sha256_prefix":"938fc7c2422ef91d","entry":"get_list_of_repo_model_paths","repo":"iamkanghyunchoi/falqon","repo_kind":"found_in_text","path":"transformers/utils/models_to_deprecate.py","file_url":"https://github.com/iamkanghyunchoi/falqon/blob/HEAD/transformers/utils/models_to_deprecate.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"938fc7c2422ef91d"}},{"code_sha256_prefix":"051d790d9b197d9d","entry":"get_model_doc_path","repo":"iamkanghyunchoi/falqon","repo_kind":"found_in_text","path":"transformers/utils/deprecate_models.py","file_url":"https://github.com/iamkanghyunchoi/falqon/blob/HEAD/transformers/utils/deprecate_models.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"051d790d9b197d9d"}},{"code_sha256_prefix":"029be6c1f552852b","entry":"get_module_source_from_name","repo":"iamkanghyunchoi/falqon","repo_kind":"found_in_text","path":"transformers/utils/modular_model_converter.py","file_url":"https://github.com/iamkanghyunchoi/falqon/blob/HEAD/transformers/utils/modular_model_converter.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"029be6c1f552852b"}},{"code_sha256_prefix":"0f78c7037b35a137","entry":"get_new_python_files_between_commits","repo":"iamkanghyunchoi/falqon","repo_kind":"found_in_text","path":"transformers/utils/pr_slow_ci_models.py","file_url":"https://github.com/iamkanghyunchoi/falqon/blob/HEAD/transformers/utils/pr_slow_ci_models.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f78c7037b35a137"}},{"code_sha256_prefix":"0c0b81b081919d8e","entry":"get_processor_types_from_config_class","repo":"iamkanghyunchoi/falqon","repo_kind":"found_in_text","path":"transformers/utils/create_dummy_models.py","file_url":"https://github.com/iamkanghyunchoi/falqon/blob/HEAD/transformers/utils/create_dummy_models.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0c0b81b081919d8e"}},{"code_sha256_prefix":"54b7273383fcfc2d","entry":"extract_model_info","repo":"iamkanghyunchoi/falqon","repo_kind":"found_in_text","path":"transformers/utils/deprecate_models.py","file_url":"https://github.com/iamkanghyunchoi/falqon/blob/HEAD/transformers/utils/deprecate_models.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"54b7273383fcfc2d"}},{"code_sha256_prefix":"b176efb26f72e507","entry":"get_architectures_from_config_class","repo":"iamkanghyunchoi/falqon","repo_kind":"found_in_text","path":"transformers/utils/create_dummy_models.py","file_url":"https://github.com/iamkanghyunchoi/falqon/blob/HEAD/transformers/utils/create_dummy_models.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b176efb26f72e507"}},{"code_sha256_prefix":"8d6745e2692cf03d","entry":"get_cased_name","repo":"iamkanghyunchoi/falqon","repo_kind":"found_in_text","path":"transformers/utils/modular_model_converter.py","file_url":"https://github.com/iamkanghyunchoi/falqon/blob/HEAD/transformers/utils/modular_model_converter.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8d6745e2692cf03d"}},{"code_sha256_prefix":"a82d62103131c8a3","entry":"get_models","repo":"iamkanghyunchoi/falqon","repo_kind":"found_in_text","path":"transformers/utils/pr_slow_ci_models.py","file_url":"https://github.com/iamkanghyunchoi/falqon/blob/HEAD/transformers/utils/pr_slow_ci_models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a82d62103131c8a3"}},{"code_sha256_prefix":"f98b77dd63ecbdf2","entry":"module_filter_fn","repo":"iamkanghyunchoi/falqon","repo_kind":"found_in_text","path":"falqon.py","file_url":"https://github.com/iamkanghyunchoi/falqon/blob/HEAD/falqon.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f98b77dd63ecbdf2"}},{"code_sha256_prefix":"1a262eab8717a7cd","entry":"parse_message","repo":"iamkanghyunchoi/falqon","repo_kind":"found_in_text","path":"transformers/utils/pr_slow_ci_models.py","file_url":"https://github.com/iamkanghyunchoi/falqon/blob/HEAD/transformers/utils/pr_slow_ci_models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1a262eab8717a7cd"}},{"code_sha256_prefix":"d646d0a64800a8fe","entry":"preserve_case_replace","repo":"iamkanghyunchoi/falqon","repo_kind":"found_in_text","path":"transformers/utils/modular_model_converter.py","file_url":"https://github.com/iamkanghyunchoi/falqon/blob/HEAD/transformers/utils/modular_model_converter.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d646d0a64800a8fe"}},{"code_sha256_prefix":"f1f5513508755a29","entry":"recursive_getattr","repo":"iamkanghyunchoi/falqon","repo_kind":"found_in_text","path":"lm-evaluation-harness/lm_eval/models/huggingface_falqon.py","file_url":"https://github.com/iamkanghyunchoi/falqon/blob/HEAD/lm-evaluation-harness/lm_eval/models/huggingface_falqon.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f1f5513508755a29"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_api"},"syntology_extracted_results":null}