{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/breaking-the-ceiling-of-the-llm-community-by","title":"Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensembling","arxiv_id":"2406.12585","date":"2024-06-18","proceeding":null,"authors":["Yao-Ching Yu","Chun-Chih Kuo","Ziqi Ye","Yu-Cheng Chang","Yueh-Se Li"],"abstract":"Ensembling multiple models has always been an effective approach to push the limits of existing performance and is widely used in classification tasks by simply averaging the classification probability vectors from multiple classifiers to achieve better accuracy. However, in the thriving open-source Large Language Model (LLM) community, ensembling methods are rare and typically limited to ensembling the full-text outputs of LLMs, such as selecting the best output using a ranker, which leads to underutilization of token-level probability information. In this paper, we treat the Generation of each token by LLMs as a Classification (GaC) for ensembling. This approach fully exploits the probability information at each generation step and better prevents LLMs from producing early incorrect tokens that lead to snowballing errors. In experiments, we ensemble state-of-the-art LLMs on several benchmarks, including exams, mathematics and reasoning, and observe that our method breaks the existing community performance ceiling. Furthermore, we observed that most of the tokens in the answer are simple and do not affect the correctness of the final answer. Therefore, we also experimented with ensembling only key tokens, and the results showed better performance with lower latency across benchmarks.","url_abs":"https://arxiv.org/abs/2406.12585v2","url_pdf":"https://arxiv.org/pdf/2406.12585v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"breaking-the-ceiling-of-the-llm-community-by","repo_url":"https://github.com/yaoching0/gac","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"arithmetic-reasoning","task_name":"Arithmetic Reasoning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"multi-task-language-understanding","task_name":"Multi-task Language Understanding"},{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/arithmetic-reasoning-on-gsm8k","task":"Arithmetic Reasoning","dataset":"GSM8K","model":"GaC(Qwen2-72B-Instruct + Llama-3-70B-Instruct)","rank_in_archive_order":19,"of":164,"metrics":{"Accuracy":"90.91"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-triviaqa","task":"Question Answering","dataset":"TriviaQA","model":"GaC(Qwen2-72B-Instruct + Llama-3-70B-Instruct)","rank_in_archive_order":14,"of":56,"metrics":{"EM":"79.29"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.12585","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.12585"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yaoching0/gac","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"unverified":3},"by_repo_kind":{"official":{"samples":7,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"250e3d0aab21f5dc","entry":"check_priorities","repo":"yaoching0/gac","repo_kind":"official","path":"utils/gac_gen_call.py","file_url":"https://github.com/yaoching0/gac/blob/HEAD/utils/gac_gen_call.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"250e3d0aab21f5dc"}},{"code_sha256_prefix":"3b4f685120e04a9c","entry":"synchronize_unfinished_sequences","repo":"yaoching0/gac","repo_kind":"official","path":"utils/gac_gen_utils.py","file_url":"https://github.com/yaoching0/gac/blob/HEAD/utils/gac_gen_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3b4f685120e04a9c"}},{"code_sha256_prefix":"a1d6799b4d0d673e","entry":"update_input_ids_and_model_kwargs","repo":"yaoching0/gac","repo_kind":"official","path":"utils/gac_gen_utils.py","file_url":"https://github.com/yaoching0/gac/blob/HEAD/utils/gac_gen_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a1d6799b4d0d673e"}},{"code_sha256_prefix":"4fda25a3a31731c5","entry":"validate_and_update_quantization","repo":"yaoching0/gac","repo_kind":"official","path":"utils/gac_gen_call.py","file_url":"https://github.com/yaoching0/gac/blob/HEAD/utils/gac_gen_call.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4fda25a3a31731c5"}},{"code_sha256_prefix":"5640e58ac0f3f8c3","entry":"get_remote_model_generator_class","repo":"yaoching0/gac","repo_kind":"official","path":"utils/ray_actor.py","file_url":"https://github.com/yaoching0/gac/blob/HEAD/utils/ray_actor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5640e58ac0f3f8c3"}},{"code_sha256_prefix":"e008364719672578","entry":"setup_custom_logger","repo":"yaoching0/gac","repo_kind":"official","path":"utils/logger.py","file_url":"https://github.com/yaoching0/gac/blob/HEAD/utils/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e008364719672578"}},{"code_sha256_prefix":"03e59517a81a3729","entry":"setup_model_actors_and_data","repo":"yaoching0/gac","repo_kind":"official","path":"utils/gac_gen_call.py","file_url":"https://github.com/yaoching0/gac/blob/HEAD/utils/gac_gen_call.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"03e59517a81a3729"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}