{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-to-represent-action-values-as-a-1","title":"Learning to Represent Action Values as a Hypergraph on the Action Vertices","arxiv_id":"2010.14680","date":"2020-10-28","proceeding":"ICLR 2021 1","authors":["Arash Tavakoli","Mehdi Fatemi","Petar Kormushev"],"abstract":"Action-value estimation is a critical component of many reinforcement learning (RL) methods whereby sample complexity relies heavily on how fast a good estimator for action value can be learned. By viewing this problem through the lens of representation learning, good representations of both state and action can facilitate action-value estimation. While advances in deep learning have seamlessly driven progress in learning state representations, given the specificity of the notion of agency to RL, little attention has been paid to learning action representations. We conjecture that leveraging the combinatorial structure of multi-dimensional action spaces is a key ingredient for learning good representations of action. To test this, we set forth the action hypergraph networks framework -- a class of functions for learning action representations in multi-dimensional discrete action spaces with a structural inductive bias. Using this framework we realise an agent class based on a combination with deep Q-networks, which we dub hypergraph Q-networks. We show the effectiveness of our approach on a myriad of domains: illustrative prediction problems under minimal confounding effects, Atari 2600 games, and discretised physical control benchmarks.","url_abs":"https://arxiv.org/abs/2010.14680v2","url_pdf":"https://arxiv.org/pdf/2010.14680v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-to-represent-action-values-as-a-1","repo_url":"https://github.com/atavakol/action-hypergraph-networks","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"atari-games","task_name":"Atari Games"},{"task_slug":"continuous-control","task_name":"Continuous Control"},{"task_slug":"general-reinforcement-learning","task_name":"General Reinforcement Learning"},{"task_slug":"inductive-bias","task_name":"Inductive Bias"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"specificity","task_name":"Specificity"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2010.14680","atlas_url":"https://app.syntology.ai/?focus=2010.14680","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2010.14680"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/atavakol/action-hypergraph-networks","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":5},"by_repo_kind":{"official":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"dead47a490130fee","entry":"ceil_rounder","repo":"atavakol/action-hypergraph-networks","repo_kind":"official","path":"hyperdopamine/agents/utils.py","file_url":"https://github.com/atavakol/action-hypergraph-networks/blob/HEAD/hyperdopamine/agents/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dead47a490130fee"}},{"code_sha256_prefix":"048d09cb919ee1a3","entry":"create_sum_order_1_map","repo":"atavakol/action-hypergraph-networks","repo_kind":"official","path":"hyperdopamine/agents/utils.py","file_url":"https://github.com/atavakol/action-hypergraph-networks/blob/HEAD/hyperdopamine/agents/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"048d09cb919ee1a3"}},{"code_sha256_prefix":"6baf0498610153c8","entry":"create_sum_order_2_map","repo":"atavakol/action-hypergraph-networks","repo_kind":"official","path":"hyperdopamine/agents/utils.py","file_url":"https://github.com/atavakol/action-hypergraph-networks/blob/HEAD/hyperdopamine/agents/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6baf0498610153c8"}},{"code_sha256_prefix":"350a26d90c79c966","entry":"get_latest_checkpoint_number","repo":"atavakol/action-hypergraph-networks","repo_kind":"official","path":"hyperdopamine/interfaces/checkpointer.py","file_url":"https://github.com/atavakol/action-hypergraph-networks/blob/HEAD/hyperdopamine/interfaces/checkpointer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"350a26d90c79c966"}},{"code_sha256_prefix":"b5d8677e13858924","entry":"read_data","repo":"atavakol/action-hypergraph-networks","repo_kind":"official","path":"plot_atari.py","file_url":"https://github.com/atavakol/action-hypergraph-networks/blob/HEAD/plot_atari.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b5d8677e13858924"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}