{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2606-13894","title":"Gefen: Optimized Stochastic Optimizer","arxiv_id":"2606.13894","date":"2026-06-11","proceeding":null,"authors":["Nadav Benedek","Tomer Koren","Ohad Fried"],"abstract":"AdamW is a default optimizer for modern deep learning, but its first and second moment states add roughly two parameter-sized buffers to training memory, increasing the already substantial cost of large-scale pretraining. We propose Gefen, a memory-efficient optimizer that automatically shares second-moment estimates across parameter blocks and quantizes the first moment using a learned codebook, thereby reducing AdamW's memory footprint by ~8x while maintaining the same performance, corresponding to a reduction of 6.5 GiB per billion parameters. The method is motivated by a theoretical result showing that large mixed Hessian entries constrain the ratio of squared gradients toward one, suggesting that Hessian-aligned parameters are natural candidates for sharing second-moment statistics. Since computing Hessians is impractical at scale, Gefen infers block structure from the initial squared gradients, requiring no architecture-specific metadata or hyperparameters beyond AdamW defaults. Gefen learns an exact histogram-based dynamic-programming quantization codebook and reuses the same blocks for first-moment scaling. Across diverse pretraining experiments, Gefen achieves the lowest peak optimizer memory among the compared AdamW-like methods while maintaining AdamW-level performance. In single-machine or distributed training, the reduced memory footprint enables larger microbatches and improves throughput significantly over AdamW, providing a practical drop-in replacement with lower memory usage that can increase throughput and enable training larger models or using larger global batch sizes. We provide the complete Python implementation, including fused CUDA kernels at https://github.com/ndvbd/Gefen","url_abs":"https://arxiv.org/abs/2606.13894","url_pdf":"https://arxiv.org/pdf/2606.13894","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2606.13894","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2606.13894"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/ndvbd/Gefen","reach":null}],"summary":{"ran":8,"unverified":5},"by_repo_kind":{"found_in_text":{"samples":13,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"a459724d9b450303","entry":"automatic_partition_reduce","repo":"ndvbd/Gefen","repo_kind":"found_in_text","path":"gefen.py","file_url":"https://github.com/ndvbd/Gefen/blob/HEAD/gefen.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a459724d9b450303"}},{"code_sha256_prefix":"73e82843f996b7f9","entry":"automatic_partition_view","repo":"ndvbd/Gefen","repo_kind":"found_in_text","path":"gefen.py","file_url":"https://github.com/ndvbd/Gefen/blob/HEAD/gefen.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"73e82843f996b7f9"}},{"code_sha256_prefix":"2f249a79bbebae58","entry":"average_within_block_variance_cpu","repo":"ndvbd/Gefen","repo_kind":"found_in_text","path":"partitioning.py","file_url":"https://github.com/ndvbd/Gefen/blob/HEAD/partitioning.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2f249a79bbebae58"}},{"code_sha256_prefix":"8aed1834de95f740","entry":"average_within_block_variance_torch","repo":"ndvbd/Gefen","repo_kind":"found_in_text","path":"partitioning.py","file_url":"https://github.com/ndvbd/Gefen/blob/HEAD/partitioning.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8aed1834de95f740"}},{"code_sha256_prefix":"91b2f1ad79df4e27","entry":"divisors","repo":"ndvbd/Gefen","repo_kind":"found_in_text","path":"partitioning.py","file_url":"https://github.com/ndvbd/Gefen/blob/HEAD/partitioning.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"91b2f1ad79df4e27"}},{"code_sha256_prefix":"bfeef597c145b8f0","entry":"gefen_state_dtype_from_grad","repo":"ndvbd/Gefen","repo_kind":"found_in_text","path":"gefen.py","file_url":"https://github.com/ndvbd/Gefen/blob/HEAD/gefen.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"bfeef597c145b8f0"}},{"code_sha256_prefix":"945f2dee1060aa76","entry":"run_with_progress_dots","repo":"ndvbd/Gefen","repo_kind":"found_in_text","path":"kernels/_build_notice.py","file_url":"https://github.com/ndvbd/Gefen/blob/HEAD/kernels/_build_notice.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"945f2dee1060aa76"}},{"code_sha256_prefix":"52bb733651591e96","entry":"should_verbose_build","repo":"ndvbd/Gefen","repo_kind":"found_in_text","path":"kernels/_build_notice.py","file_url":"https://github.com/ndvbd/Gefen/blob/HEAD/kernels/_build_notice.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"52bb733651591e96"}},{"code_sha256_prefix":"2e17814834cb001a","entry":"average_adjacent_log_block_mean_difference_cuda_kernel","repo":"ndvbd/Gefen","repo_kind":"found_in_text","path":"kernels/period_variance.py","file_url":"https://github.com/ndvbd/Gefen/blob/HEAD/kernels/period_variance.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2e17814834cb001a"}},{"code_sha256_prefix":"09e0c00f6ab906b6","entry":"average_within_block_coefficient_of_variation_cuda_kernel","repo":"ndvbd/Gefen","repo_kind":"found_in_text","path":"kernels/period_variance.py","file_url":"https://github.com/ndvbd/Gefen/blob/HEAD/kernels/period_variance.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"09e0c00f6ab906b6"}},{"code_sha256_prefix":"60eb1bde74d0a9e9","entry":"average_within_block_variance_cuda_kernel","repo":"ndvbd/Gefen","repo_kind":"found_in_text","path":"kernels/period_variance.py","file_url":"https://github.com/ndvbd/Gefen/blob/HEAD/kernels/period_variance.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"60eb1bde74d0a9e9"}},{"code_sha256_prefix":"f53263815a9b81ba","entry":"exact_dp","repo":"ndvbd/Gefen","repo_kind":"found_in_text","path":"quantization.py","file_url":"https://github.com/ndvbd/Gefen/blob/HEAD/quantization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f53263815a9b81ba"}},{"code_sha256_prefix":"1b85617e13599a5c","entry":"load_gefen_cuda_extension","repo":"ndvbd/Gefen","repo_kind":"found_in_text","path":"kernels/_build_notice.py","file_url":"https://github.com/ndvbd/Gefen/blob/HEAD/kernels/_build_notice.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1b85617e13599a5c"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,264 of the 9,581 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9581,"papers_checked":6264},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}