{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-framework-for-multi-armedbandit-testing","title":"A framework for Multi-A(rmed)/B(andit) testing with online FDR control","arxiv_id":"1706.05378","date":"2017-06-16","proceeding":"NeurIPS 2017 12","authors":["Fanny Yang","Aaditya Ramdas","Kevin Jamieson","Martin J. Wainwright"],"abstract":"We propose an alternative framework to existing setups for controlling false\nalarms when multiple A/B tests are run over time. This setup arises in many\npractical applications, e.g. when pharmaceutical companies test new treatment\noptions against control pills for different diseases, or when internet\ncompanies test their default webpages versus various alternatives over time.\nOur framework proposes to replace a sequence of A/B tests by a sequence of\nbest-arm MAB instances, which can be continuously monitored by the data\nscientist. When interleaving the MAB tests with an an online false discovery\nrate (FDR) algorithm, we can obtain the best of both worlds: low sample\ncomplexity and any time online FDR control. Our main contributions are: (i) to\npropose reasonable definitions of a null hypothesis for MAB instances; (ii) to\ndemonstrate how one can derive an always-valid sequential p-value that allows\ncontinuous monitoring of each MAB test; and (iii) to show that using rejection\nthresholds of online-FDR algorithms as the confidence levels for the MAB\nalgorithms results in both sample-optimality, high power and low FDR at any\npoint in time. We run extensive simulations to verify our claims, and also\nreport results on real data collected from the New Yorker Cartoon Caption\ncontest.","url_abs":"http://arxiv.org/abs/1706.05378v2","url_pdf":"http://arxiv.org/pdf/1706.05378v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-framework-for-multi-armedbandit-testing","repo_url":"https://github.com/fanny-yang/MABFDR","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":null,"task_name":"valid"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1706.05378","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}