{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/deeptop-deep-threshold-optimal-policy-for","title":"DeepTOP: Deep Threshold-Optimal Policy for MDPs and RMABs","arxiv_id":"2209.08646","date":"2022-09-18","proceeding":null,"authors":["Khaled Nakhleh","I-Hong Hou"],"abstract":"We consider the problem of learning the optimal threshold policy for control problems. Threshold policies make control decisions by evaluating whether an element of the system state exceeds a certain threshold, whose value is determined by other elements of the system state. By leveraging the monotone property of threshold policies, we prove that their policy gradients have a surprisingly simple expression. We use this simple expression to build an off-policy actor-critic algorithm for learning the optimal threshold policy. Simulation results show that our policy significantly outperforms other reinforcement learning algorithms due to its ability to exploit the monotone property. In addition, we show that the Whittle index, a powerful tool for restless multi-armed bandit problems, is equivalent to the optimal threshold policy for an alternative problem. This observation leads to a simple algorithm that finds the Whittle index by learning the optimal threshold policy in the alternative problem. Simulation results show that our algorithm learns the Whittle index much faster than several recent studies that learn the Whittle index through indirect means.","url_abs":"https://arxiv.org/abs/2209.08646v2","url_pdf":"https://arxiv.org/pdf/2209.08646v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"deeptop-deep-threshold-optimal-policy-for","repo_url":"https://github.com/khalednakhleh/deeptop","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2209.08646","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2209.08646"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/khalednakhleh/deeptop","reach":null}],"summary":{"ran":3,"ran_honours":1,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3473662fd8d11551","entry":"Actor","repo":"khalednakhleh/deeptop","repo_kind":"official","path":"MDP/DeepTOP.py","file_url":"https://github.com/khalednakhleh/deeptop/blob/HEAD/MDP/DeepTOP.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3473662fd8d11551"}},{"code_sha256_prefix":"61ba3ab4aa8be9c0","entry":"Critic","repo":"khalednakhleh/deeptop","repo_kind":"official","path":"MDP/DeepTOP.py","file_url":"https://github.com/khalednakhleh/deeptop/blob/HEAD/MDP/DeepTOP.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"61ba3ab4aa8be9c0"}},{"code_sha256_prefix":"c62009e413091ff9","entry":"SequentialMemory","repo":"khalednakhleh/deeptop","repo_kind":"official","path":"MDP/DeepTOP.py","file_url":"https://github.com/khalednakhleh/deeptop/blob/HEAD/MDP/DeepTOP.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c62009e413091ff9"}},{"code_sha256_prefix":"f2ba9f82931525fe","entry":"sample_batch_indexes","repo":"khalednakhleh/deeptop","repo_kind":"official","path":"MDP/DeepTOP.py","file_url":"https://github.com/khalednakhleh/deeptop/blob/HEAD/MDP/DeepTOP.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f2ba9f82931525fe"}},{"code_sha256_prefix":"b291d6ddac717573","entry":"DeepTOP_MDP","repo":"khalednakhleh/deeptop","repo_kind":"official","path":"MDP/DeepTOP.py","file_url":"https://github.com/khalednakhleh/deeptop/blob/HEAD/MDP/DeepTOP.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b291d6ddac717573"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}