{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/offline-model-based-optimization-via-policy","title":"Offline Model-Based Optimization via Policy-Guided Gradient Search","arxiv_id":"2405.05349","date":"2024-05-08","proceeding":null,"authors":["Yassine Chemingui","Aryan Deshwal","Trong Nghia Hoang","Janardhan Rao Doppa"],"abstract":"Offline optimization is an emerging problem in many experimental engineering domains including protein, drug or aircraft design, where online experimentation to collect evaluation data is too expensive or dangerous. To avoid that, one has to optimize an unknown function given only its offline evaluation at a fixed set of inputs. A naive solution to this problem is to learn a surrogate model of the unknown function and optimize this surrogate instead. However, such a naive optimizer is prone to erroneous overestimation of the surrogate (possibly due to over-fitting on a biased sample of function evaluation) on inputs outside the offline dataset. Prior approaches addressing this challenge have primarily focused on learning robust surrogate models. However, their search strategies are derived from the surrogate model rather than the actual offline data. To fill this important gap, we introduce a new learning-to-search perspective for offline optimization by reformulating it as an offline reinforcement learning problem. Our proposed policy-guided gradient search approach explicitly learns the best policy for a given surrogate model created from the offline data. Our empirical results on multiple benchmarks demonstrate that the learned optimization policy can be combined with existing offline surrogates to significantly improve the optimization performance.","url_abs":"https://arxiv.org/abs/2405.05349v1","url_pdf":"https://arxiv.org/pdf/2405.05349v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"offline-model-based-optimization-via-policy","repo_url":"https://github.com/yassinech/pgs","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.05349","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.05349"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/google-research/google-research","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/siddarthk97/bonet","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yassinech/pgs","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/yassineCh/PGS","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/younggeng/CQL","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}}],"summary":{"ran":13,"ran_draft_wrong":2,"unverified":3},"by_repo_kind":{"official":{"samples":17,"ran":14,"repositories":1},"found_in_text":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":17,"samples":[{"code_sha256_prefix":"3a2ee7569ed76374","entry":"ForwardModel","repo":"yassineCh/PGS","repo_kind":"official","path":"coms_cleaned/nets.py","file_url":"https://github.com/yassineCh/PGS/blob/HEAD/coms_cleaned/nets.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3a2ee7569ed76374"}},{"code_sha256_prefix":"32231080b4b40efc","entry":"batch_to_torch","repo":"yassineCh/PGS","repo_kind":"official","path":"replay_buffer.py","file_url":"https://github.com/yassineCh/PGS/blob/HEAD/replay_buffer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"32231080b4b40efc"}},{"code_sha256_prefix":"1e5f5f635bdf773a","entry":"extend_and_repeat","repo":"yassineCh/PGS","repo_kind":"official","path":"model.py","file_url":"https://github.com/yassineCh/PGS/blob/HEAD/model.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1e5f5f635bdf773a"}},{"code_sha256_prefix":"7712cf0435cff046","entry":"generate_trajectory","repo":"yassineCh/PGS","repo_kind":"official","path":"generate_trajectories.py","file_url":"https://github.com/yassineCh/PGS/blob/HEAD/generate_trajectories.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7712cf0435cff046"}},{"code_sha256_prefix":"753958f65f38b74a","entry":"get_grads","repo":"yassineCh/PGS","repo_kind":"official","path":"generate_trajectories.py","file_url":"https://github.com/yassineCh/PGS/blob/HEAD/generate_trajectories.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"753958f65f38b74a"}},{"code_sha256_prefix":"541df06efd45bb79","entry":"get_noninit_guided_config","repo":"google-research/google-research","repo_kind":"found_in_text","path":"gradient_based_tuning/guided_parameters.py","file_url":"https://github.com/google-research/google-research/blob/HEAD/gradient_based_tuning/guided_parameters.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"541df06efd45bb79"}},{"code_sha256_prefix":"c8b2fc80a3fc7424","entry":"index_batch","repo":"yassineCh/PGS","repo_kind":"official","path":"replay_buffer.py","file_url":"https://github.com/yassineCh/PGS/blob/HEAD/replay_buffer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c8b2fc80a3fc7424"}},{"code_sha256_prefix":"2847ff08c9fa8b2c","entry":"link","repo":"yassineCh/PGS","repo_kind":"official","path":"generate_trajectories.py","file_url":"https://github.com/yassineCh/PGS/blob/HEAD/generate_trajectories.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2847ff08c9fa8b2c"}},{"code_sha256_prefix":"8ce1498bb25b3523","entry":"load_embedded_data","repo":"yassineCh/PGS","repo_kind":"official","path":"knn_evaluate.py","file_url":"https://github.com/yassineCh/PGS/blob/HEAD/knn_evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8ce1498bb25b3523"}},{"code_sha256_prefix":"d06ff25d52ad20f5","entry":"multiple_action_q_function","repo":"yassineCh/PGS","repo_kind":"official","path":"model.py","file_url":"https://github.com/yassineCh/PGS/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d06ff25d52ad20f5"}},{"code_sha256_prefix":"7e4ca4d4478f4b49","entry":"normalize","repo":"yassineCh/PGS","repo_kind":"official","path":"knn_evaluate.py","file_url":"https://github.com/yassineCh/PGS/blob/HEAD/knn_evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7e4ca4d4478f4b49"}},{"code_sha256_prefix":"3f55f8769dd45c20","entry":"parition_batch_train_test","repo":"yassineCh/PGS","repo_kind":"official","path":"replay_buffer.py","file_url":"https://github.com/yassineCh/PGS/blob/HEAD/replay_buffer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3f55f8769dd45c20"}},{"code_sha256_prefix":"21c82f7926f182f1","entry":"prepare_dataloaders","repo":"yassineCh/PGS","repo_kind":"official","path":"surrogate.py","file_url":"https://github.com/yassineCh/PGS/blob/HEAD/surrogate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"21c82f7926f182f1"}},{"code_sha256_prefix":"45a7574afd665891","entry":"split_dataset","repo":"yassineCh/PGS","repo_kind":"official","path":"surrogate.py","file_url":"https://github.com/yassineCh/PGS/blob/HEAD/surrogate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"45a7574afd665891"}},{"code_sha256_prefix":"7c357874e6b59114","entry":"validate_dnn","repo":"yassineCh/PGS","repo_kind":"official","path":"surrogate.py","file_url":"https://github.com/yassineCh/PGS/blob/HEAD/surrogate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7c357874e6b59114"}},{"code_sha256_prefix":"5868f3063af26312","entry":"build_pipeline","repo":"yassineCh/PGS","repo_kind":"official","path":"data.py","file_url":"https://github.com/yassineCh/PGS/blob/HEAD/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5868f3063af26312"}},{"code_sha256_prefix":"2b3c0c162fb9aa5b","entry":"create_mlp","repo":"yassineCh/PGS","repo_kind":"official","path":"embed.py","file_url":"https://github.com/yassineCh/PGS/blob/HEAD/embed.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2b3c0c162fb9aa5b"}},{"code_sha256_prefix":"28b0766edda08356","entry":"load_top_observations","repo":"yassineCh/PGS","repo_kind":"official","path":"evaluate_policy.py","file_url":"https://github.com/yassineCh/PGS/blob/HEAD/evaluate_policy.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"28b0766edda08356"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}