{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/benchmarking-simulation-based-inference","title":"Benchmarking Simulation-Based Inference","arxiv_id":"2101.04653","date":"2021-01-12","proceeding":null,"authors":["Jan-Matthis Lueckmann","Jan Boelts","David S. Greenberg","Pedro J. Gonçalves","Jakob H. Macke"],"abstract":"Recent advances in probabilistic modelling have led to a large number of simulation-based inference algorithms which do not require numerical evaluation of likelihoods. However, a public benchmark with appropriate performance metrics for such 'likelihood-free' algorithms has been lacking. This has made it difficult to compare algorithms and identify their strengths and weaknesses. We set out to fill this gap: We provide a benchmark with inference tasks and suitable performance metrics, with an initial selection of algorithms including recent approaches employing neural networks and classical Approximate Bayesian Computation methods. We found that the choice of performance metric is critical, that even state-of-the-art algorithms have substantial room for improvement, and that sequential estimation improves sample efficiency. Neural network-based approaches generally exhibit better performance, but there is no uniformly best algorithm. We provide practical advice and highlight the potential of the benchmark to diagnose problems and improve algorithms. The results can be explored interactively on a companion website. All code is open source, making it possible to contribute further benchmark tasks and inference algorithms.","url_abs":"https://arxiv.org/abs/2101.04653v2","url_pdf":"https://arxiv.org/pdf/2101.04653v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"benchmarking-simulation-based-inference","repo_url":"https://github.com/sbi-benchmark/sbibm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"benchmarking-simulation-based-inference","repo_url":"https://github.com/bkmi/cnre","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2101.04653","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2101.04653"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sbi-benchmark/sbibm","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bkmi/cnre","reach":null}],"summary":{"ran_fixture":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"aeb1e589882d2515","entry":"c2st","repo":"sbi-benchmark/sbibm","repo_kind":"official","path":"sbibm/metrics/c2st.py","file_url":"https://github.com/sbi-benchmark/sbibm/blob/HEAD/sbibm/metrics/c2st.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aeb1e589882d2515"}},{"code_sha256_prefix":"1448fc76605fa1da","entry":"c2st_auc","repo":"sbi-benchmark/sbibm","repo_kind":"official","path":"sbibm/metrics/c2st.py","file_url":"https://github.com/sbi-benchmark/sbibm/blob/HEAD/sbibm/metrics/c2st.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1448fc76605fa1da"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}