{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-the-emergence-of-weak-to-strong","title":"On the Emergence of Weak-to-Strong Generalization: A Bias-Variance Perspective","arxiv_id":"2505.24313","date":"2025-05-30","proceeding":null,"authors":["Gengze Xu","Wei Yao","Ziqiao Wang","Yong liu"],"abstract":"Weak-to-strong generalization (W2SG) refers to the phenomenon where a strong student model, trained on a dataset labeled by a weak teacher, ultimately outperforms the teacher on the target task. Recent studies attribute this performance gain to the prediction misfit between the student and teacher models. In this work, we theoretically investigate the emergence of W2SG through a generalized bias-variance decomposition of Bregman divergence. Specifically, we show that the expected population risk gap between the student and teacher is quantified by the expected misfit between the two models. While this aligns with previous results, our analysis removes several restrictive assumptions, most notably, the convexity of the student's hypothesis class, required in earlier works. Moreover, we show that W2SG is more likely to emerge when the student model approximates its posterior mean teacher, rather than mimicking an individual teacher. Using a concrete example, we demonstrate that if the student model has significantly larger capacity than the teacher, it can indeed converge to this posterior mean. Our analysis also suggests that avoiding overfitting to the teacher's supervision and reducing the entropy of student's prediction further facilitate W2SG. In addition, we show that the reverse cross-entropy loss, unlike the standard forward cross-entropy, is less sensitive to the predictive uncertainty of the teacher. Finally, we empirically verify our theoretical insights and demonstrate that incorporating the reverse cross-entropy loss consistently improves student performance.","url_abs":"https://arxiv.org/abs/2505.24313v1","url_pdf":"https://arxiv.org/pdf/2505.24313v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"attribute","task_name":"Attribute"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2505.24313","atlas_url":"https://app.syntology.ai/?focus=2505.24313","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.24313"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Gengze/W2SG-Bregman","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"unverified":5},"by_repo_kind":{"found_in_text":{"samples":6,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3c8b1c2195e4361b","entry":"get_config_foldername","repo":"Gengze/W2SG-Bregman","repo_kind":"found_in_text","path":"train_w2s.py","file_url":"https://github.com/Gengze/W2SG-Bregman/blob/HEAD/train_w2s.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c8b1c2195e4361b"}},{"code_sha256_prefix":"4392e5b76d837eab","entry":"clear_mem","repo":"Gengze/W2SG-Bregman","repo_kind":"found_in_text","path":"weak_to_strong/common.py","file_url":"https://github.com/Gengze/W2SG-Bregman/blob/HEAD/weak_to_strong/common.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4392e5b76d837eab"}},{"code_sha256_prefix":"8254e3ba7a1ada12","entry":"get_tokenizer","repo":"Gengze/W2SG-Bregman","repo_kind":"found_in_text","path":"weak_to_strong/common.py","file_url":"https://github.com/Gengze/W2SG-Bregman/blob/HEAD/weak_to_strong/common.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8254e3ba7a1ada12"}},{"code_sha256_prefix":"3e5a02dd2daa0d8a","entry":"load_dataset","repo":"Gengze/W2SG-Bregman","repo_kind":"found_in_text","path":"weak_to_strong/datasets.py","file_url":"https://github.com/Gengze/W2SG-Bregman/blob/HEAD/weak_to_strong/datasets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e5a02dd2daa0d8a"}},{"code_sha256_prefix":"b362575164f27b9f","entry":"load_multi_label_dataset","repo":"Gengze/W2SG-Bregman","repo_kind":"found_in_text","path":"weak_to_strong/datasets.py","file_url":"https://github.com/Gengze/W2SG-Bregman/blob/HEAD/weak_to_strong/datasets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b362575164f27b9f"}},{"code_sha256_prefix":"cf8df2a14a1dbda0","entry":"load_reward_dataset","repo":"Gengze/W2SG-Bregman","repo_kind":"found_in_text","path":"weak_to_strong/datasets.py","file_url":"https://github.com/Gengze/W2SG-Bregman/blob/HEAD/weak_to_strong/datasets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cf8df2a14a1dbda0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}