{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fbi-llm-scaling-up-fully-binarized-llms-from","title":"FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillation","arxiv_id":"2407.07093","date":"2024-07-09","proceeding":null,"authors":["Liqun Ma","MingJie Sun","Zhiqiang Shen"],"abstract":"This work presents a Fully BInarized Large Language Model (FBI-LLM), demonstrating for the first time how to train a large-scale binary language model from scratch (not the partial binary or ternary LLM like BitNet b1.58) to match the performance of its full-precision counterparts (e.g., FP16 or BF16) in transformer-based LLMs. It achieves this by employing an autoregressive distillation (AD) loss with maintaining equivalent model dimensions (130M, 1.3B, 7B) and training data volume as regular LLM pretraining, while delivering competitive results in terms of perplexity and task-specific effectiveness. Intriguingly, by analyzing the training trajectory, we find that the pretrained weight is not necessary for training binarized LLMs from scratch. This research encourages a new computational framework and may facilitate the future design of specialized hardware tailored for fully 1-bit LLMs. We make all models, code, and training dataset fully accessible and transparent to support further research (Code: https://github.com/LiqunMa/FBI-LLM. Model: https://huggingface.co/LiqunMa/).","url_abs":"https://arxiv.org/abs/2407.07093v1","url_pdf":"https://arxiv.org/pdf/2407.07093v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"fbi-llm-scaling-up-fully-binarized-llms-from","repo_url":"https://github.com/liqunma/fbi-llm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2407.07093","atlas_url":"https://app.syntology.ai/?focus=2407.07093","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.07093"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/liqunma/fbi-llm","reach":{"status":"ok"}}],"summary":{"ran_fixture":1,"ran":8,"ran_draft_wrong":2,"unverified":3},"by_repo_kind":{"official":{"samples":14,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"f725bc2d76076485","entry":"apply_rotary_pos_emb","repo":"liqunma/fbi-llm","repo_kind":"official","path":"model_utils/modeling_llama.py","file_url":"https://github.com/liqunma/fbi-llm/blob/HEAD/model_utils/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f725bc2d76076485"}},{"code_sha256_prefix":"06a11aeead10da89","entry":"cal_flip_rate","repo":"liqunma/fbi-llm","repo_kind":"official","path":"analysis.py","file_url":"https://github.com/liqunma/fbi-llm/blob/HEAD/analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"06a11aeead10da89"}},{"code_sha256_prefix":"7895ceb853a24341","entry":"collate_fn","repo":"liqunma/fbi-llm","repo_kind":"official","path":"analysis.py","file_url":"https://github.com/liqunma/fbi-llm/blob/HEAD/analysis.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7895ceb853a24341"}},{"code_sha256_prefix":"35a07ea1a83582da","entry":"get_cosine_lr_decay_fn","repo":"liqunma/fbi-llm","repo_kind":"official","path":"main_utils.py","file_url":"https://github.com/liqunma/fbi-llm/blob/HEAD/main_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"35a07ea1a83582da"}},{"code_sha256_prefix":"e9353db1655d860d","entry":"get_grad_norm","repo":"liqunma/fbi-llm","repo_kind":"official","path":"main_utils.py","file_url":"https://github.com/liqunma/fbi-llm/blob/HEAD/main_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e9353db1655d860d"}},{"code_sha256_prefix":"f79ef099baf027c2","entry":"get_jsonl","repo":"liqunma/fbi-llm","repo_kind":"official","path":"datautils.py","file_url":"https://github.com/liqunma/fbi-llm/blob/HEAD/datautils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f79ef099baf027c2"}},{"code_sha256_prefix":"c7c652c2a1cbf982","entry":"load_json","repo":"liqunma/fbi-llm","repo_kind":"official","path":"utils.py","file_url":"https://github.com/liqunma/fbi-llm/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c7c652c2a1cbf982"}},{"code_sha256_prefix":"a7387837542ca8a3","entry":"load_jsonl_examples","repo":"liqunma/fbi-llm","repo_kind":"official","path":"main_utils.py","file_url":"https://github.com/liqunma/fbi-llm/blob/HEAD/main_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a7387837542ca8a3"}},{"code_sha256_prefix":"5cc6ee4fff500d54","entry":"prepare_model_for_eval","repo":"liqunma/fbi-llm","repo_kind":"official","path":"utils.py","file_url":"https://github.com/liqunma/fbi-llm/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5cc6ee4fff500d54"}},{"code_sha256_prefix":"dfbbf527543890a1","entry":"prepare_model_for_training","repo":"liqunma/fbi-llm","repo_kind":"official","path":"utils.py","file_url":"https://github.com/liqunma/fbi-llm/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dfbbf527543890a1"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"liqunma/fbi-llm","repo_kind":"official","path":"model_utils/modeling_llama.py","file_url":"https://github.com/liqunma/fbi-llm/blob/HEAD/model_utils/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"93d038e23f97f61b","entry":"binarylinear_to_regularlinear","repo":"liqunma/fbi-llm","repo_kind":"official","path":"qat/replace_module.py","file_url":"https://github.com/liqunma/fbi-llm/blob/HEAD/qat/replace_module.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"93d038e23f97f61b"}},{"code_sha256_prefix":"7282a242e1279742","entry":"get_tokenizer","repo":"liqunma/fbi-llm","repo_kind":"official","path":"datautils.py","file_url":"https://github.com/liqunma/fbi-llm/blob/HEAD/datautils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7282a242e1279742"}},{"code_sha256_prefix":"b035d07aea65196d","entry":"replace_with_learnable_binarylinear","repo":"liqunma/fbi-llm","repo_kind":"official","path":"qat/replace_module.py","file_url":"https://github.com/liqunma/fbi-llm/blob/HEAD/qat/replace_module.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b035d07aea65196d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}