{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/training-free-bayesianization-for-low-rank","title":"Training-Free Bayesianization for Low-Rank Adapters of Large Language Models","arxiv_id":"2412.05723","date":"2024-12-07","proceeding":null,"authors":["Haizhou Shi","Yibin Wang","Ligong Han","huan zhang","Hao Wang"],"abstract":"Estimating the uncertainty of responses from Large Language Models (LLMs) remains a critical challenge. While recent Bayesian methods have demonstrated effectiveness in quantifying uncertainty through low-rank weight updates, they typically require complex fine-tuning or post-training procedures. In this paper, we propose Training-Free Bayesianization (TFB), a simple yet theoretically grounded framework that efficiently transforms trained low-rank adapters into Bayesian ones without additional training. TFB systematically searches for the maximally acceptable level of variance in the weight posterior, constrained within a family of low-rank isotropic Gaussian distributions. Our theoretical analysis shows that under mild conditions, this search process is equivalent to KL-regularized variational optimization, a generalized form of variational inference. Through comprehensive experiments, we show that TFB achieves superior uncertainty estimation and generalization compared to existing methods while eliminating the need for complex Bayesianization training procedures. Code will be available at https://github.com/Wang-ML-Lab/bayesian-peft.","url_abs":"https://arxiv.org/abs/2412.05723v2","url_pdf":"https://arxiv.org/pdf/2412.05723v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"training-free-bayesianization-for-low-rank","repo_url":"https://github.com/wang-ml-lab/bayesian-peft","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"variational-inference","task_name":"Variational Inference"}],"methods":[{"method_slug":"variational-inference","method_name":"Variational Inference"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.05723","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.05723"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Wang-ML-Lab/bayesian-peft","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wang-ml-lab/bayesian-peft","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"unverified":3},"by_repo_kind":{"official":{"samples":5,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"faa2b4f4b6f7589d","entry":"accuracy_topk","repo":"Wang-ML-Lab/bayesian-peft","repo_kind":"official","path":"modelwrappers/wrapperbase.py","file_url":"https://github.com/Wang-ML-Lab/bayesian-peft/blob/HEAD/modelwrappers/wrapperbase.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"faa2b4f4b6f7589d"}},{"code_sha256_prefix":"c1f0478f8201f8eb","entry":"get_linear_schedule_with_warmup","repo":"Wang-ML-Lab/bayesian-peft","repo_kind":"official","path":"modelwrappers/wrapperbase.py","file_url":"https://github.com/Wang-ML-Lab/bayesian-peft/blob/HEAD/modelwrappers/wrapperbase.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c1f0478f8201f8eb"}},{"code_sha256_prefix":"c8fd03666b0f564c","entry":"blob_8bitlinear_forward","repo":"Wang-ML-Lab/bayesian-peft","repo_kind":"official","path":"modelwrappers/blob.py","file_url":"https://github.com/Wang-ML-Lab/bayesian-peft/blob/HEAD/modelwrappers/blob.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c8fd03666b0f564c"}},{"code_sha256_prefix":"619e60fc0ee56ac8","entry":"blob_linear_forward","repo":"Wang-ML-Lab/bayesian-peft","repo_kind":"official","path":"modelwrappers/blob.py","file_url":"https://github.com/Wang-ML-Lab/bayesian-peft/blob/HEAD/modelwrappers/blob.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"619e60fc0ee56ac8"}},{"code_sha256_prefix":"d52a587b83014f77","entry":"div_posterior_prior","repo":"Wang-ML-Lab/bayesian-peft","repo_kind":"official","path":"modelwrappers/blob.py","file_url":"https://github.com/Wang-ML-Lab/bayesian-peft/blob/HEAD/modelwrappers/blob.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d52a587b83014f77"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}