{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-large-scale-open-dataset-for-bandit","title":"Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluation","arxiv_id":"2008.07146","date":"2020-08-17","proceeding":null,"authors":["Yuta Saito","Shunsuke Aihara","Megumi Matsutani","Yusuke Narita"],"abstract":"Off-policy evaluation (OPE) aims to estimate the performance of hypothetical policies using data generated by a different policy. Because of its huge potential impact in practice, there has been growing research interest in this field. There is, however, no real-world public dataset that enables the evaluation of OPE, making its experimental studies unrealistic and irreproducible. With the goal of enabling realistic and reproducible OPE research, we present Open Bandit Dataset, a public logged bandit dataset collected on a large-scale fashion e-commerce platform, ZOZOTOWN. Our dataset is unique in that it contains a set of multiple logged bandit datasets collected by running different policies on the same platform. This enables experimental comparisons of different OPE estimators for the first time. We also develop Python software called Open Bandit Pipeline to streamline and standardize the implementation of batch bandit algorithms and OPE. Our open data and software will contribute to fair and transparent OPE research and help the community identify fruitful research directions. We provide extensive benchmark experiments of existing OPE estimators using our dataset and software. The results open up essential challenges and new avenues for future OPE research.","url_abs":"https://arxiv.org/abs/2008.07146v5","url_pdf":"https://arxiv.org/pdf/2008.07146v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-large-scale-open-dataset-for-bandit","repo_url":"https://github.com/st-tech/zr-obp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"a-large-scale-open-dataset-for-bandit","repo_url":"https://github.com/RetailMarketingAI/retailsynth-agentsim","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"a-large-scale-open-dataset-for-bandit","repo_url":"https://github.com/han20192019/newRL","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"a-large-scale-open-dataset-for-bandit","repo_url":"https://github.com/usaito/obp-demo","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"off-policy-evaluation","task_name":"Off-policy evaluation"}],"methods":[],"datasets_introduced":[{"slug":"obp","name":"OBP","full_name":"Open Bandit Dataset"}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2008.07146","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2008.07146"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/han20192019/newRL","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/RetailMarketingAI/retailsynth-agentsim","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/st-tech/zr-obp","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/usaito/obp-demo","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":2},"by_repo_kind":{"listed":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5ae885b38baf7e63","entry":"sample_action_fast","repo":"han20192019/newRL","repo_kind":"listed","path":"obp/utils.py","file_url":"https://github.com/han20192019/newRL/blob/HEAD/obp/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5ae885b38baf7e63"}},{"code_sha256_prefix":"f98fa584326b4b66","entry":"convert_to_action_dist","repo":"han20192019/newRL","repo_kind":"listed","path":"obp/utils.py","file_url":"https://github.com/han20192019/newRL/blob/HEAD/obp/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f98fa584326b4b66"}},{"code_sha256_prefix":"b0299ee348e0a723","entry":"estimate_confidence_interval_by_bootstrap","repo":"han20192019/newRL","repo_kind":"listed","path":"obp/utils.py","file_url":"https://github.com/han20192019/newRL/blob/HEAD/obp/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b0299ee348e0a723"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}