{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/stochastic-bilevel-optimization-with-lower","title":"Contextual Bilevel Reinforcement Learning for Incentive Alignment","arxiv_id":"2406.01575","date":"2024-06-03","proceeding":null,"authors":["Vinzenz Thoma","Barna Pasztor","Andreas Krause","Giorgia Ramponi","Yifan Hu"],"abstract":"The optimal policy in various real-world strategic decision-making problems depends both on the environmental configuration and exogenous events. For these settings, we introduce Contextual Bilevel Reinforcement Learning (CB-RL), a stochastic bilevel decision-making model, where the lower level consists of solving a contextual Markov Decision Process (CMDP). CB-RL can be viewed as a Stackelberg Game where the leader and a random context beyond the leader's control together decide the setup of many MDPs that potentially multiple followers best respond to. This framework extends beyond traditional bilevel optimization and finds relevance in diverse fields such as RLHF, tax design, reward shaping, contract theory and mechanism design. We propose a stochastic Hyper Policy Gradient Descent (HPGD) algorithm to solve CB-RL, and demonstrate its convergence. Notably, HPGD uses stochastic hypergradient estimates, based on observations of the followers' trajectories. Therefore, it allows followers to use any training procedure and the leader to be agnostic of the specific algorithm, which aligns with various real-world scenarios. We further consider the setting when the leader can influence the training of followers and propose an accelerated algorithm. We empirically demonstrate the performance of our algorithm for reward shaping and tax design.","url_abs":"https://arxiv.org/abs/2406.01575v2","url_pdf":"https://arxiv.org/pdf/2406.01575v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"stochastic-bilevel-optimization-with-lower","repo_url":"https://github.com/lasgroup/hpgd","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"bilevel-optimization","task_name":"Bilevel Optimization"},{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.01575","atlas_url":"https://app.syntology.ai/?focus=2406.01575","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.01575"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lasgroup/hpgd","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":8,"unverified":5},"by_repo_kind":{"official":{"samples":13,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"90bf283f158bfaef","entry":"apply_greedy_policy","repo":"lasgroup/hpgd","repo_kind":"official","path":"train_stochastic_bilevel_opt.py","file_url":"https://github.com/lasgroup/hpgd/blob/HEAD/train_stochastic_bilevel_opt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"90bf283f158bfaef"}},{"code_sha256_prefix":"fcab5c91a12d6c1e","entry":"default_reward_function","repo":"lasgroup/hpgd","repo_kind":"official","path":"src/environments/utils.py","file_url":"https://github.com/lasgroup/hpgd/blob/HEAD/src/environments/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fcab5c91a12d6c1e"}},{"code_sha256_prefix":"e1d1608d2719c09d","entry":"incentive_transform","repo":"lasgroup/hpgd","repo_kind":"official","path":"src/models/IncentiveModel.py","file_url":"https://github.com/lasgroup/hpgd/blob/HEAD/src/models/IncentiveModel.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e1d1608d2719c09d"}},{"code_sha256_prefix":"1fa8c23d22d4dd92","entry":"normalize_discounted_rewards","repo":"lasgroup/hpgd","repo_kind":"official","path":"src/algorithms/utils.py","file_url":"https://github.com/lasgroup/hpgd/blob/HEAD/src/algorithms/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1fa8c23d22d4dd92"}},{"code_sha256_prefix":"cf39e222b919e4f0","entry":"random_normal_perturbations","repo":"lasgroup/hpgd","repo_kind":"official","path":"train_tax_design_zero_order.py","file_url":"https://github.com/lasgroup/hpgd/blob/HEAD/train_tax_design_zero_order.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cf39e222b919e4f0"}},{"code_sha256_prefix":"8aaaff5f138c1764","entry":"random_split_like_tree","repo":"lasgroup/hpgd","repo_kind":"official","path":"train_zero_order.py","file_url":"https://github.com/lasgroup/hpgd/blob/HEAD/train_zero_order.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8aaaff5f138c1764"}},{"code_sha256_prefix":"5e8fb421a069c1de","entry":"regularized_softmax","repo":"lasgroup/hpgd","repo_kind":"official","path":"train_stochastic_bilevel_opt.py","file_url":"https://github.com/lasgroup/hpgd/blob/HEAD/train_stochastic_bilevel_opt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5e8fb421a069c1de"}},{"code_sha256_prefix":"6a60b610c1b46d4e","entry":"sample_array","repo":"lasgroup/hpgd","repo_kind":"official","path":"src/environments/utils.py","file_url":"https://github.com/lasgroup/hpgd/blob/HEAD/src/environments/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6a60b610c1b46d4e"}},{"code_sha256_prefix":"75b917a4731eaf8a","entry":"clip_norm","repo":"lasgroup/hpgd","repo_kind":"official","path":"train_stochastic_bilevel_opt.py","file_url":"https://github.com/lasgroup/hpgd/blob/HEAD/train_stochastic_bilevel_opt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"75b917a4731eaf8a"}},{"code_sha256_prefix":"d811aaa5f794375d","entry":"get_actions_from_q_values","repo":"lasgroup/hpgd","repo_kind":"official","path":"src/algorithms/Regularized_DQN.py","file_url":"https://github.com/lasgroup/hpgd/blob/HEAD/src/algorithms/Regularized_DQN.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d811aaa5f794375d"}},{"code_sha256_prefix":"caa8707bc43176db","entry":"string_to_bool_map","repo":"lasgroup/hpgd","repo_kind":"official","path":"src/environments/utils.py","file_url":"https://github.com/lasgroup/hpgd/blob/HEAD/src/environments/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"caa8707bc43176db"}},{"code_sha256_prefix":"a944954ab6dc61d0","entry":"tree_random_normal_like","repo":"lasgroup/hpgd","repo_kind":"official","path":"train_zero_order.py","file_url":"https://github.com/lasgroup/hpgd/blob/HEAD/train_zero_order.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a944954ab6dc61d0"}},{"code_sha256_prefix":"45825551bc380031","entry":"update_value_functions","repo":"lasgroup/hpgd","repo_kind":"official","path":"train_amd.py","file_url":"https://github.com/lasgroup/hpgd/blob/HEAD/train_amd.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"45825551bc380031"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}