{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2603-17685","title":"Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints","arxiv_id":"2603.17685","date":"2026-03-18","proceeding":null,"authors":["Ting Gao","Stavros Orfanoudakis","Nan Lin","Winnie Daamen","Serge Hoogendoorn","Elvin Isufi"],"abstract":"Balancing policy expressiveness with the exploration-exploitation trade-off is a core challenge in online Reinforcement Learning (RL). While Stochastic Differential Equation (SDE)-based diffusion policies can represent complex, multimodal action distributions, they suffer from two critical limitations: their stochastic reverse processes render entropy intractable (necessitating heuristic exploration), and computing policy gradients through long denoising chains is expensive and unstable. In this work, we show that ODE-based flow matching inherently resolves these issues by enabling both simulation-free policy optimization and tractable entropy computation. Building on this, we introduce Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints (FMER). Our framework exploits this insight in three ways. First, we theoretically establish that minimizing an advantage-weighted conditional flow matching loss acts as a simulation-free surrogate for policy mirror descent. This steers the velocity field toward high-value regions while entirely avoiding backpropagation through the ODE solver. Second, we derive an analytic entropy objective that corrects for the density distortion caused by the $\\tanh$ transformation (mapping an unbounded latent space to bounded actions), thereby facilitating principled maximum-entropy optimization. Finally, we dynamically tune the mirror descent temperature based on the effective sample size to enforce a robust trust region during training. Empirical evaluations demonstrate that FMER achieves superior performance on the challenging sparse-reward FrankaKitchen environment, while maintaining competitive results across standard dense-reward MuJoCo benchmarks.","url_abs":"https://arxiv.org/abs/2603.17685","url_pdf":"https://arxiv.org/pdf/2603.17685","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2603.17685","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2603.17685"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/wadx2019/qvpo","reach":null},{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/mahaitongdae/diffusion_policy_online_rl","reach":null},{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/Elessar123/SAC-FLOW","reach":null},{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/lzqw/FLAME","reach":null}],"summary":{"ran":7,"ran_fixture":1,"unverified":21},"by_repo_kind":{"found_in_text":{"samples":29,"ran":8,"repositories":3}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":18,"samples":[{"code_sha256_prefix":"e3e7ea9dea530ccd","entry":"Critic","repo":"wadx2019/qvpo","repo_kind":"found_in_text","path":"agent/qvpo.py","file_url":"https://github.com/wadx2019/qvpo/blob/HEAD/agent/qvpo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e3e7ea9dea530ccd"}},{"code_sha256_prefix":"56ea6cda0fdbbddd","entry":"CriticQR","repo":"wadx2019/qvpo","repo_kind":"found_in_text","path":"agent/qvpo.py","file_url":"https://github.com/wadx2019/qvpo/blob/HEAD/agent/qvpo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"56ea6cda0fdbbddd"}},{"code_sha256_prefix":"0c81dda4b2c07710","entry":"Diffusion","repo":"wadx2019/qvpo","repo_kind":"found_in_text","path":"agent/qvpo.py","file_url":"https://github.com/wadx2019/qvpo/blob/HEAD/agent/qvpo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0c81dda4b2c07710"}},{"code_sha256_prefix":"10276c719fb6f855","entry":"Model","repo":"wadx2019/qvpo","repo_kind":"found_in_text","path":"agent/qvpo.py","file_url":"https://github.com/wadx2019/qvpo/blob/HEAD/agent/qvpo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"10276c719fb6f855"}},{"code_sha256_prefix":"52e215849471bdfc","entry":"SinusoidalPosEmb","repo":"wadx2019/qvpo","repo_kind":"found_in_text","path":"agent/qvpo.py","file_url":"https://github.com/wadx2019/qvpo/blob/HEAD/agent/qvpo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"52e215849471bdfc"}},{"code_sha256_prefix":"20545bf1089ab44e","entry":"WeightedL1","repo":"wadx2019/qvpo","repo_kind":"found_in_text","path":"agent/qvpo.py","file_url":"https://github.com/wadx2019/qvpo/blob/HEAD/agent/qvpo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"20545bf1089ab44e"}},{"code_sha256_prefix":"3b2b8b659d67b0af","entry":"WeightedL2","repo":"wadx2019/qvpo","repo_kind":"found_in_text","path":"agent/qvpo.py","file_url":"https://github.com/wadx2019/qvpo/blob/HEAD/agent/qvpo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3b2b8b659d67b0af"}},{"code_sha256_prefix":"3e12dfe5d2e56c8d","entry":"quantile_huber_loss","repo":"wadx2019/qvpo","repo_kind":"found_in_text","path":"agent/qvpo.py","file_url":"https://github.com/wadx2019/qvpo/blob/HEAD/agent/qvpo.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e12dfe5d2e56c8d"}},{"code_sha256_prefix":"cbe4b2a64b8a77d3","entry":"Actor","repo":"Elessar123/SAC-FLOW","repo_kind":"found_in_text","path":"from_scratch_code/SAC_flow_gru_jax.py","file_url":"https://github.com/Elessar123/SAC-FLOW/blob/HEAD/from_scratch_code/SAC_flow_gru_jax.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cbe4b2a64b8a77d3"}},{"code_sha256_prefix":"c7dd064a50cc4845","entry":"BatchRenorm","repo":"Elessar123/SAC-FLOW","repo_kind":"found_in_text","path":"from_scratch_code/SAC_flow_gru_jax.py","file_url":"https://github.com/Elessar123/SAC-FLOW/blob/HEAD/from_scratch_code/SAC_flow_gru_jax.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c7dd064a50cc4845"}},{"code_sha256_prefix":"fe3538c8acd17840","entry":"QVPO","repo":"wadx2019/qvpo","repo_kind":"found_in_text","path":"agent/qvpo.py","file_url":"https://github.com/wadx2019/qvpo/blob/HEAD/agent/qvpo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fe3538c8acd17840"}},{"code_sha256_prefix":"aab29e00b150b7b2","entry":"SinusoidalPosEmb","repo":"Elessar123/SAC-FLOW","repo_kind":"found_in_text","path":"from_scratch_code/SAC_flow_gru_jax.py","file_url":"https://github.com/Elessar123/SAC-FLOW/blob/HEAD/from_scratch_code/SAC_flow_gru_jax.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aab29e00b150b7b2"}},{"code_sha256_prefix":"728e142be8c63b8b","entry":"WeightedLoss","repo":"wadx2019/qvpo","repo_kind":"found_in_text","path":"agent/qvpo.py","file_url":"https://github.com/wadx2019/qvpo/blob/HEAD/agent/qvpo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"728e142be8c63b8b"}},{"code_sha256_prefix":"b98ba81a4ea31a61","entry":"build_table","repo":"lzqw/FLAME","repo_kind":"found_in_text","path":"analysis/make_exp37_tables.py","file_url":"https://github.com/lzqw/FLAME/blob/HEAD/analysis/make_exp37_tables.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b98ba81a4ea31a61"}},{"code_sha256_prefix":"8f52d69d86b15b42","entry":"classify","repo":"lzqw/FLAME","repo_kind":"found_in_text","path":"analysis/plot_exp37_main_figures.py","file_url":"https://github.com/lzqw/FLAME/blob/HEAD/analysis/plot_exp37_main_figures.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8f52d69d86b15b42"}},{"code_sha256_prefix":"1d2ce563da9d7683","entry":"entropy2d","repo":"lzqw/FLAME","repo_kind":"found_in_text","path":"analysis/plot_action_space_entropy.py","file_url":"https://github.com/lzqw/FLAME/blob/HEAD/analysis/plot_action_space_entropy.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1d2ce563da9d7683"}},{"code_sha256_prefix":"54adb6b865c1eab7","entry":"flatten_rollouts","repo":"lzqw/FLAME","repo_kind":"found_in_text","path":"analysis/plot_exp37_diagnostics.py","file_url":"https://github.com/lzqw/FLAME/blob/HEAD/analysis/plot_exp37_diagnostics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"54adb6b865c1eab7"}},{"code_sha256_prefix":"0a19232d2baba9b2","entry":"fmt_stat","repo":"lzqw/FLAME","repo_kind":"found_in_text","path":"analysis/make_exp37_tables.py","file_url":"https://github.com/lzqw/FLAME/blob/HEAD/analysis/make_exp37_tables.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0a19232d2baba9b2"}},{"code_sha256_prefix":"6f97de74414878c5","entry":"init_weights","repo":"wadx2019/qvpo","repo_kind":"found_in_text","path":"agent/qvpo.py","file_url":"https://github.com/wadx2019/qvpo/blob/HEAD/agent/qvpo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6f97de74414878c5"}},{"code_sha256_prefix":"5d7dd03a436d96f8","entry":"load_json","repo":"lzqw/FLAME","repo_kind":"found_in_text","path":"analysis/plot_three_obstacle_pullback_results.py","file_url":"https://github.com/lzqw/FLAME/blob/HEAD/analysis/plot_three_obstacle_pullback_results.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5d7dd03a436d96f8"}},{"code_sha256_prefix":"0385ed9d7ed29835","entry":"load_npz","repo":"lzqw/FLAME","repo_kind":"found_in_text","path":"analysis/plot_three_obstacle_pullback_results.py","file_url":"https://github.com/lzqw/FLAME/blob/HEAD/analysis/plot_three_obstacle_pullback_results.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0385ed9d7ed29835"}},{"code_sha256_prefix":"4fbcb926264f78c1","entry":"parse_meta","repo":"lzqw/FLAME","repo_kind":"found_in_text","path":"analysis/collect_exp37_results.py","file_url":"https://github.com/lzqw/FLAME/blob/HEAD/analysis/collect_exp37_results.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4fbcb926264f78c1"}},{"code_sha256_prefix":"502742af75564dca","entry":"parse_path_meta","repo":"lzqw/FLAME","repo_kind":"found_in_text","path":"analysis/plot_exp37_diagnostics.py","file_url":"https://github.com/lzqw/FLAME/blob/HEAD/analysis/plot_exp37_diagnostics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"502742af75564dca"}},{"code_sha256_prefix":"abbd0c4947647c0e","entry":"pearson_corr","repo":"lzqw/FLAME","repo_kind":"found_in_text","path":"analysis/plot_projection_critic_heatmaps.py","file_url":"https://github.com/lzqw/FLAME/blob/HEAD/analysis/plot_projection_critic_heatmaps.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"abbd0c4947647c0e"}},{"code_sha256_prefix":"1df489cbc7af1f28","entry":"percentile_clip","repo":"lzqw/FLAME","repo_kind":"found_in_text","path":"analysis/plot_projection_critic_heatmaps.py","file_url":"https://github.com/lzqw/FLAME/blob/HEAD/analysis/plot_projection_critic_heatmaps.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1df489cbc7af1f28"}},{"code_sha256_prefix":"c48ae4d3213b89a9","entry":"require_path","repo":"lzqw/FLAME","repo_kind":"found_in_text","path":"analysis/plot_double_integrator_pullback_results.py","file_url":"https://github.com/lzqw/FLAME/blob/HEAD/analysis/plot_double_integrator_pullback_results.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c48ae4d3213b89a9"}},{"code_sha256_prefix":"e54daa8eee4256d4","entry":"run_rollouts","repo":"lzqw/FLAME","repo_kind":"found_in_text","path":"analysis/plot_curvature_lqr.py","file_url":"https://github.com/lzqw/FLAME/blob/HEAD/analysis/plot_curvature_lqr.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e54daa8eee4256d4"}},{"code_sha256_prefix":"c43c9056d3743694","entry":"safe_eigvals","repo":"lzqw/FLAME","repo_kind":"found_in_text","path":"analysis/plot_curvature_lqr.py","file_url":"https://github.com/lzqw/FLAME/blob/HEAD/analysis/plot_curvature_lqr.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c43c9056d3743694"}},{"code_sha256_prefix":"7d29cbd982dfe045","entry":"spearman_corr","repo":"lzqw/FLAME","repo_kind":"found_in_text","path":"analysis/plot_projection_critic_heatmaps.py","file_url":"https://github.com/lzqw/FLAME/blob/HEAD/analysis/plot_projection_critic_heatmaps.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7d29cbd982dfe045"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}