{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hyperbolic-fine-tuning-for-large-language","title":"Hyperbolic Fine-tuning for Large Language Models","arxiv_id":"2410.04010","date":"2024-10-05","proceeding":null,"authors":["Menglin Yang","Aosong Feng","Bo Xiong","Jihong Liu","Irwin King","Rex Ying"],"abstract":"Large language models (LLMs) have demonstrated remarkable performance on various tasks. However, it remains an open question whether the default Euclidean space is the most suitable choice for embedding tokens in LLMs. In this study, we first investigate the non-Euclidean characteristics of LLMs. Our findings reveal that token frequency follows a power-law distribution, with high-frequency tokens clustering near the origin and low-frequency tokens positioned farther away. Additionally, token embeddings exhibit a high degree of hyperbolicity, indicating a latent tree-like structure in the embedding space. Building on the observation, we propose to efficiently fine-tune LLMs in hyperbolic space to better exploit the underlying complex structures. However, we found that this fine-tuning in hyperbolic space cannot be achieved with naive application of exponential and logarithmic maps, when the embedding and weight matrices both reside in Euclidean space. To address this technique issue, we introduce a new method called hyperbolic low-rank efficient fine-tuning, HypLoRA, that performs low-rank adaptation directly on the hyperbolic manifold, avoiding the cancellation effect caused by the exponential and logarithmic maps, thus preserving the hyperbolic modeling capabilities. Through extensive experiments, we demonstrate that HypLoRA significantly enhances the performance of LLMs on reasoning tasks, particularly for complex reasoning problems. In particular, HypLoRA improves the performance in the complex AQuA dataset by up to 13.0%, showcasing its effectiveness in handling complex reasoning challenges","url_abs":"https://arxiv.org/abs/2410.04010v1","url_pdf":"https://arxiv.org/pdf/2410.04010v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hyperbolic-fine-tuning-for-large-language","repo_url":"https://github.com/marlin-codes/HypLLM","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2410.04010","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.04010"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/marlin-codes/HypLLM","reach":{"status":"ok"}}],"summary":{"ran_honours":1,"ran_draft_wrong":3,"ran":10,"unverified":7},"by_repo_kind":{"official":{"samples":18,"ran":11,"repositories":1},"community":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":21,"samples":[{"code_sha256_prefix":"9b3f4eb40a06e33e","entry":"compute_token_frequencies","repo":"marlin-codes/hyplora","repo_kind":"community","path":"utils/token_frequency_distribution.py","file_url":"https://github.com/marlin-codes/hyplora/blob/HEAD/utils/token_frequency_distribution.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9b3f4eb40a06e33e"}},{"code_sha256_prefix":"2ac7050b0ae37227","entry":"create_dir","repo":"marlin-codes/hyplora","repo_kind":"community","path":"utils/token_frequency_distribution.py","file_url":"https://github.com/marlin-codes/hyplora/blob/HEAD/utils/token_frequency_distribution.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2ac7050b0ae37227"}},{"code_sha256_prefix":"efa052b7627685a8","entry":"generate_prompt","repo":"marlin-codes/HypLLM","repo_kind":"official","path":"finetune_gemma.py","file_url":"https://github.com/marlin-codes/HypLLM/blob/HEAD/finetune_gemma.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"efa052b7627685a8"}},{"code_sha256_prefix":"855c98801d59aba9","entry":"generate_prompt","repo":"marlin-codes/HypLLM","repo_kind":"official","path":"finetune_gemma7b.py","file_url":"https://github.com/marlin-codes/HypLLM/blob/HEAD/finetune_gemma7b.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"855c98801d59aba9"}},{"code_sha256_prefix":"129a49595d6cd469","entry":"generate_prompt","repo":"marlin-codes/HypLLM","repo_kind":"official","path":"finetune_llama3.py","file_url":"https://github.com/marlin-codes/HypLLM/blob/HEAD/finetune_llama3.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"129a49595d6cd469"}},{"code_sha256_prefix":"10b11110fd400897","entry":"generate_prompt","repo":"marlin-codes/HypLLM","repo_kind":"official","path":"finetune_qwen.py","file_url":"https://github.com/marlin-codes/HypLLM/blob/HEAD/finetune_qwen.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"10b11110fd400897"}},{"code_sha256_prefix":"db94af6846fc86ac","entry":"generate_prompt_gemma","repo":"marlin-codes/HypLLM","repo_kind":"official","path":"commonsense_evaluate.py","file_url":"https://github.com/marlin-codes/HypLLM/blob/HEAD/commonsense_evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"db94af6846fc86ac"}},{"code_sha256_prefix":"0f6cd6a20020ddc7","entry":"generate_prompt_gemma","repo":"marlin-codes/HypLLM","repo_kind":"official","path":"commonsense_evaluate_gemma.py","file_url":"https://github.com/marlin-codes/HypLLM/blob/HEAD/commonsense_evaluate_gemma.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0f6cd6a20020ddc7"}},{"code_sha256_prefix":"349264d7e81ea7aa","entry":"generate_prompt_legacy","repo":"marlin-codes/HypLLM","repo_kind":"official","path":"commonsense_evaluate.py","file_url":"https://github.com/marlin-codes/HypLLM/blob/HEAD/commonsense_evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"349264d7e81ea7aa"}},{"code_sha256_prefix":"bfb00dffb905f6c9","entry":"generate_prompt_llama","repo":"marlin-codes/HypLLM","repo_kind":"official","path":"commonsense_evaluate_gemma.py","file_url":"https://github.com/marlin-codes/HypLLM/blob/HEAD/commonsense_evaluate_gemma.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bfb00dffb905f6c9"}},{"code_sha256_prefix":"108402de33a9b9f9","entry":"generate_prompt_qwen","repo":"marlin-codes/HypLLM","repo_kind":"official","path":"commonsense_evaluate.py","file_url":"https://github.com/marlin-codes/HypLLM/blob/HEAD/commonsense_evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"108402de33a9b9f9"}},{"code_sha256_prefix":"56a9466ff35b35bb","entry":"load_data","repo":"marlin-codes/HypLLM","repo_kind":"official","path":"commonsense_evaluate_qwen.py","file_url":"https://github.com/marlin-codes/HypLLM/blob/HEAD/commonsense_evaluate_qwen.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"56a9466ff35b35bb"}},{"code_sha256_prefix":"9986725c1f5dc7bd","entry":"load_data","repo":"marlin-codes/HypLLM","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/marlin-codes/HypLLM/blob/HEAD/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9986725c1f5dc7bd"}},{"code_sha256_prefix":"edf7c9e239eadabf","entry":"mkdirs","repo":"marlin-codes/hyplora","repo_kind":"community","path":"utils/token_frequency_distribution.py","file_url":"https://github.com/marlin-codes/hyplora/blob/HEAD/utils/token_frequency_distribution.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"edf7c9e239eadabf"}},{"code_sha256_prefix":"27ce6e249f9b1a82","entry":"main","repo":"marlin-codes/HypLLM","repo_kind":"official","path":"commonsense_evaluate_gemma.py","file_url":"https://github.com/marlin-codes/HypLLM/blob/HEAD/commonsense_evaluate_gemma.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"27ce6e249f9b1a82"}},{"code_sha256_prefix":"e563b8e4282a0c32","entry":"main","repo":"marlin-codes/HypLLM","repo_kind":"official","path":"commonsense_evaluate_qwen.py","file_url":"https://github.com/marlin-codes/HypLLM/blob/HEAD/commonsense_evaluate_qwen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e563b8e4282a0c32"}},{"code_sha256_prefix":"11971b0db65e9e75","entry":"main","repo":"marlin-codes/HypLLM","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/marlin-codes/HypLLM/blob/HEAD/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"11971b0db65e9e75"}},{"code_sha256_prefix":"89be12b59a6331d6","entry":"train","repo":"marlin-codes/HypLLM","repo_kind":"official","path":"finetune_gemma.py","file_url":"https://github.com/marlin-codes/HypLLM/blob/HEAD/finetune_gemma.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"89be12b59a6331d6"}},{"code_sha256_prefix":"265773714d80dabf","entry":"train","repo":"marlin-codes/HypLLM","repo_kind":"official","path":"finetune_gemma7b.py","file_url":"https://github.com/marlin-codes/HypLLM/blob/HEAD/finetune_gemma7b.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"265773714d80dabf"}},{"code_sha256_prefix":"fbba0cdd6457bbf4","entry":"train","repo":"marlin-codes/HypLLM","repo_kind":"official","path":"finetune_llama3.py","file_url":"https://github.com/marlin-codes/HypLLM/blob/HEAD/finetune_llama3.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fbba0cdd6457bbf4"}},{"code_sha256_prefix":"1200fafec7c5e3d1","entry":"train","repo":"marlin-codes/HypLLM","repo_kind":"official","path":"finetune_qwen.py","file_url":"https://github.com/marlin-codes/HypLLM/blob/HEAD/finetune_qwen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1200fafec7c5e3d1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}