{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/estimating-the-number-and-effect-sizes-of-non","title":"Estimating the number and effect sizes of non-null hypotheses","arxiv_id":"2002.07297","date":"2020-02-17","proceeding":"ICML 2020 1","authors":["Jennifer Brennan","Ramya Korlakai Vinayak","Kevin Jamieson"],"abstract":"We study the problem of estimating the distribution of effect sizes (the mean of the test statistic under the alternate hypothesis) in a multiple testing setting. Knowing this distribution allows us to calculate the power (type II error) of any experimental design. We show that it is possible to estimate this distribution using an inexpensive pilot experiment, which takes significantly fewer samples than would be required by an experiment that identified the discoveries. Our estimator can be used to guarantee the number of discoveries that will be made using a given experimental design in a future experiment. We prove that this simple and computationally efficient estimator enjoys a number of favorable theoretical properties, and demonstrate its effectiveness on data from a gene knockout experiment on influenza inhibition in Drosophila.","url_abs":"https://arxiv.org/abs/2002.07297v2","url_pdf":"https://arxiv.org/pdf/2002.07297v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"estimating-the-number-and-effect-sizes-of-non","repo_url":"https://github.com/jenniferbrennan/CountingDiscoveries","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"experimental-design","task_name":"Experimental Design"}],"methods":[{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2002.07297","atlas_url":"https://app.syntology.ai/?focus=2002.07297","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2002.07297"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jenniferbrennan/CountingDiscoveries","reach":null}],"summary":{"ran_fixture":1,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"0743825fb9f8131a","entry":"binarySearch","repo":"jenniferbrennan/CountingDiscoveries","repo_kind":"official","path":"estimator.py","file_url":"https://github.com/jenniferbrennan/CountingDiscoveries/blob/HEAD/estimator.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0743825fb9f8131a"}},{"code_sha256_prefix":"b0abbe5b267df1e6","entry":"estimateEntireLine","repo":"jenniferbrennan/CountingDiscoveries","repo_kind":"official","path":"estimator.py","file_url":"https://github.com/jenniferbrennan/CountingDiscoveries/blob/HEAD/estimator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b0abbe5b267df1e6"}},{"code_sha256_prefix":"197f9e4b172dfcc6","entry":"estimateEntireLine_oneJob","repo":"jenniferbrennan/CountingDiscoveries","repo_kind":"official","path":"estimator.py","file_url":"https://github.com/jenniferbrennan/CountingDiscoveries/blob/HEAD/estimator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"197f9e4b172dfcc6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}