{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/more-efficient-randomized-exploration-for","title":"More Efficient Randomized Exploration for Reinforcement Learning via Approximate Sampling","arxiv_id":"2406.12241","date":"2024-06-18","proceeding":null,"authors":["Haque Ishfaq","Yixin Tan","Yu Yang","Qingfeng Lan","Jianfeng Lu","A. Rupam Mahmood","Doina Precup","Pan Xu"],"abstract":"Thompson sampling (TS) is one of the most popular exploration techniques in reinforcement learning (RL). However, most TS algorithms with theoretical guarantees are difficult to implement and not generalizable to Deep RL. While the emerging approximate sampling-based exploration schemes are promising, most existing algorithms are specific to linear Markov Decision Processes (MDP) with suboptimal regret bounds, or only use the most basic samplers such as Langevin Monte Carlo. In this work, we propose an algorithmic framework that incorporates different approximate sampling methods with the recently proposed Feel-Good Thompson Sampling (FGTS) approach (Zhang, 2022; Dann et al., 2021), which was previously known to be computationally intractable in general. When applied to linear MDPs, our regret analysis yields the best known dependency of regret on dimensionality, surpassing existing randomized algorithms. Additionally, we provide explicit sampling complexity for each employed sampler. Empirically, we show that in tasks where deep exploration is necessary, our proposed algorithms that combine FGTS and approximate sampling perform significantly better compared to other strong baselines. On several challenging games from the Atari 57 suite, our algorithms achieve performance that is either better than or on par with other strong baselines from the deep RL literature.","url_abs":"https://arxiv.org/abs/2406.12241v1","url_pdf":"https://arxiv.org/pdf/2406.12241v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"more-efficient-randomized-exploration-for","repo_url":"https://github.com/panxulab/lsvi-ase","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"thompson-sampling","task_name":"Thompson Sampling"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"ts","method_name":"TS"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.12241","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.12241"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/panxulab/lsvi-ase","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d3ea6d1370050c52","entry":"get_csv_result_dict","repo":"panxulab/lsvi-ase","repo_kind":"official","path":"atari_analysis.py","file_url":"https://github.com/panxulab/lsvi-ase/blob/HEAD/atari_analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d3ea6d1370050c52"}},{"code_sha256_prefix":"c087bcc4a878fdfe","entry":"get_process_result_dict","repo":"panxulab/lsvi-ase","repo_kind":"official","path":"atari_analysis.py","file_url":"https://github.com/panxulab/lsvi-ase/blob/HEAD/atari_analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c087bcc4a878fdfe"}},{"code_sha256_prefix":"e3514d05d0540a24","entry":"get_process_result_dict","repo":"panxulab/lsvi-ase","repo_kind":"official","path":"nchain_analysis.py","file_url":"https://github.com/panxulab/lsvi-ase/blob/HEAD/nchain_analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e3514d05d0540a24"}},{"code_sha256_prefix":"b956e8eaded5fc36","entry":"moving_average","repo":"panxulab/lsvi-ase","repo_kind":"official","path":"atari_results/plot_atari_individual.py","file_url":"https://github.com/panxulab/lsvi-ase/blob/HEAD/atari_results/plot_atari_individual.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b956e8eaded5fc36"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}