{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-11125","title":"Language Modeling with Hyperspherical Flows","arxiv_id":"2605.11125","date":"2026-05-11","proceeding":null,"authors":["Justin Deschenaux","Caglar Gulcehre"],"abstract":"Discrete Diffusion Language Models progressed rapidly as an alternative to autoregressive (AR) models, motivated by their parallel generation abilities. However, for tractability, discrete diffusion models sample from a factorized distribution, which is less expressive than AR. Recent Flow Language Models (FLMs) apply continuous flows to language, transporting noise to data with a deterministic ODE that avoids factorized sampling. FLMs operate on one-hot vectors whose dimension scales with the vocabulary size, making FLMs costly to train. Moreover, since all distinct one-hot embeddings are equidistant in $\\ell_2$, adding Gaussian noise does not have a clear semantic interpretation (unlike images, where Gaussian noise progressively degrades structure). We introduce $\\mathbb{S}$-FLM, a latent FLM in the hypersphere. $\\mathbb{S}$-FLM generates sequences by rotating vectors in $\\mathbb{S}^{d-1}$ along a velocity field learned with cross-entropy, avoiding the overhead of materializing one-hot vectors. Previous FLMs match AR in Generative Perplexity (Gen.\\ PPL), but samples with high likelihood are not necessarily correct in verifiable domains such as math and code. $\\mathbb{S}$-FLM substantially improves continuous flow language models on large-vocabulary reasoning and closes the gap to masked diffusion under standard-temperature sampling ($T=1$), while a gap remains under optimized low-temperature ($T=0.1$) decoding.","url_abs":"https://arxiv.org/abs/2605.11125","url_pdf":"https://arxiv.org/pdf/2605.11125","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2605.11125","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.11125"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/s-sahoo/duo","reach":null},{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/jdeschena/s-flm","reach":null}],"summary":{"ran":18,"ran_fixture":1,"ran_honours":2,"unverified":3},"by_repo_kind":{"found_in_text":{"samples":24,"ran":21,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"5d6407c4bc5302f2","entry":"alpha_to_gamma","repo":"jdeschena/s-flm","repo_kind":"found_in_text","path":"flm_utils.py","file_url":"https://github.com/jdeschena/s-flm/blob/HEAD/flm_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5d6407c4bc5302f2"}},{"code_sha256_prefix":"cb962a179d06cce8","entry":"attention_inner_heads_flash","repo":"s-sahoo/duo","repo_kind":"found_in_text","path":"models/unit_test_attention.py","file_url":"https://github.com/s-sahoo/duo/blob/HEAD/models/unit_test_attention.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cb962a179d06cce8"}},{"code_sha256_prefix":"fbbc6722b74bda27","entry":"bias_dropout_add_scale","repo":"jdeschena/s-flm","repo_kind":"found_in_text","path":"models/dit.py","file_url":"https://github.com/jdeschena/s-flm/blob/HEAD/models/dit.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fbbc6722b74bda27"}},{"code_sha256_prefix":"1c0ec5966835fcd3","entry":"build_luts","repo":"jdeschena/s-flm","repo_kind":"found_in_text","path":"flm_utils.py","file_url":"https://github.com/jdeschena/s-flm/blob/HEAD/flm_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1c0ec5966835fcd3"}},{"code_sha256_prefix":"99c19e9dd7632489","entry":"compute_alpha_exact","repo":"jdeschena/s-flm","repo_kind":"found_in_text","path":"flm_utils.py","file_url":"https://github.com/jdeschena/s-flm/blob/HEAD/flm_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"99c19e9dd7632489"}},{"code_sha256_prefix":"f6b944f50d3f15ae","entry":"count_parameters","repo":"s-sahoo/duo","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/s-sahoo/duo/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f6b944f50d3f15ae"}},{"code_sha256_prefix":"fdd6aad96accd488","entry":"default_init","repo":"s-sahoo/duo","repo_kind":"found_in_text","path":"models/unet.py","file_url":"https://github.com/s-sahoo/duo/blob/HEAD/models/unet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fdd6aad96accd488"}},{"code_sha256_prefix":"6f60d462bd6f4a0c","entry":"eval_suffix_nll","repo":"s-sahoo/duo","repo_kind":"found_in_text","path":"discrete_diffusion_harness.py","file_url":"https://github.com/s-sahoo/duo/blob/HEAD/discrete_diffusion_harness.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6f60d462bd6f4a0c"}},{"code_sha256_prefix":"d689900773b45e39","entry":"expected_rank","repo":"jdeschena/s-flm","repo_kind":"found_in_text","path":"candi_utils.py","file_url":"https://github.com/jdeschena/s-flm/blob/HEAD/candi_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d689900773b45e39"}},{"code_sha256_prefix":"dec1de00023a923b","entry":"fsspec_exists","repo":"s-sahoo/duo","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/s-sahoo/duo/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dec1de00023a923b"}},{"code_sha256_prefix":"06d2d8e50f91481f","entry":"get_bias_dropout_add_scale","repo":"jdeschena/s-flm","repo_kind":"found_in_text","path":"models/dit.py","file_url":"https://github.com/jdeschena/s-flm/blob/HEAD/models/dit.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"06d2d8e50f91481f"}},{"code_sha256_prefix":"b62f2c5c4a261f52","entry":"inference_sigmas","repo":"jdeschena/s-flm","repo_kind":"found_in_text","path":"candi_utils.py","file_url":"https://github.com/jdeschena/s-flm/blob/HEAD/candi_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b62f2c5c4a261f52"}},{"code_sha256_prefix":"32176fe5b679b415","entry":"justnorm","repo":"jdeschena/s-flm","repo_kind":"found_in_text","path":"models/sphere_arch.py","file_url":"https://github.com/jdeschena/s-flm/blob/HEAD/models/sphere_arch.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"32176fe5b679b415"}},{"code_sha256_prefix":"576e1b9b9bcae8ff","entry":"lm1b_detokenizer","repo":"jdeschena/s-flm","repo_kind":"found_in_text","path":"dataloader.py","file_url":"https://github.com/jdeschena/s-flm/blob/HEAD/dataloader.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"576e1b9b9bcae8ff"}},{"code_sha256_prefix":"78b555034273f3c7","entry":"modulate","repo":"jdeschena/s-flm","repo_kind":"found_in_text","path":"models/dit.py","file_url":"https://github.com/jdeschena/s-flm/blob/HEAD/models/dit.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"78b555034273f3c7"}},{"code_sha256_prefix":"9f7fbdbe7ca280e0","entry":"ptb_detokenizer","repo":"jdeschena/s-flm","repo_kind":"found_in_text","path":"dataloader.py","file_url":"https://github.com/jdeschena/s-flm/blob/HEAD/dataloader.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9f7fbdbe7ca280e0"}},{"code_sha256_prefix":"300194a15680602c","entry":"sample_categorical","repo":"s-sahoo/duo","repo_kind":"found_in_text","path":"trainer_base.py","file_url":"https://github.com/s-sahoo/duo/blob/HEAD/trainer_base.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"300194a15680602c"}},{"code_sha256_prefix":"3168aaa9a362be02","entry":"training_sigma_ve","repo":"jdeschena/s-flm","repo_kind":"found_in_text","path":"candi_utils.py","file_url":"https://github.com/jdeschena/s-flm/blob/HEAD/candi_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3168aaa9a362be02"}},{"code_sha256_prefix":"c40a24e67f67c90e","entry":"transformer_timestep_embedding","repo":"s-sahoo/duo","repo_kind":"found_in_text","path":"models/unet.py","file_url":"https://github.com/s-sahoo/duo/blob/HEAD/models/unet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c40a24e67f67c90e"}},{"code_sha256_prefix":"3ac11bfe0ae80195","entry":"variance_scaling","repo":"s-sahoo/duo","repo_kind":"found_in_text","path":"models/unet.py","file_url":"https://github.com/s-sahoo/duo/blob/HEAD/models/unet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3ac11bfe0ae80195"}},{"code_sha256_prefix":"46c10627206a7795","entry":"wt_detokenizer","repo":"jdeschena/s-flm","repo_kind":"found_in_text","path":"dataloader.py","file_url":"https://github.com/jdeschena/s-flm/blob/HEAD/dataloader.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"46c10627206a7795"}},{"code_sha256_prefix":"60415613b01cd2d8","entry":"dataclass","repo":"jdeschena/s-flm","repo_kind":"found_in_text","path":"dataclass_patch.py","file_url":"https://github.com/jdeschena/s-flm/blob/HEAD/dataclass_patch.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"60415613b01cd2d8"}},{"code_sha256_prefix":"833b4c254da7e2c9","entry":"eval_suffix_nll_ar","repo":"s-sahoo/duo","repo_kind":"found_in_text","path":"discrete_diffusion_harness.py","file_url":"https://github.com/s-sahoo/duo/blob/HEAD/discrete_diffusion_harness.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"833b4c254da7e2c9"}},{"code_sha256_prefix":"db0136eb80a50383","entry":"fsspec_listdir","repo":"s-sahoo/duo","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/s-sahoo/duo/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"db0136eb80a50383"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":"us.anthropic.claude-sonnet-4-5-20250929-v1:0","prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,795 of the 9,581 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9581,"papers_checked":6795},"entries":[],"not_placed":{"boards":1,"rejected_by_independent_check":1,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}