{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/benchopt-reproducible-efficient-and","title":"Benchopt: Reproducible, efficient and collaborative optimization benchmarks","arxiv_id":"2206.13424","date":"2022-06-27","proceeding":null,"authors":["Thomas Moreau","Mathurin Massias","Alexandre Gramfort","Pierre Ablin","Pierre-Antoine Bannier","Benjamin Charlier","Mathieu Dagréou","Tom Dupré La Tour","Ghislain Durif","Cassio F. Dantas","Quentin Klopfenstein","Johan Larsson","En Lai","Tanguy Lefort","Benoit Malézieux","Badr Moufad","Binh T. Nguyen","Alain Rakotomamonjy","Zaccharie Ramzi","Joseph Salmon","Samuel Vaiter"],"abstract":"Numerical validation is at the core of machine learning research as it allows to assess the actual impact of new methods, and to confirm the agreement between theory and practice. Yet, the rapid development of the field poses several challenges: researchers are confronted with a profusion of methods to compare, limited transparency and consensus on best practices, as well as tedious re-implementation work. As a result, validation is often very partial, which can lead to wrong conclusions that slow down the progress of research. We propose Benchopt, a collaborative framework to automate, reproduce and publish optimization benchmarks in machine learning across programming languages and hardware architectures. Benchopt simplifies benchmarking for the community by providing an off-the-shelf tool for running, sharing and extending experiments. To demonstrate its broad usability, we showcase benchmarks on three standard learning tasks: $\\ell_2$-regularized logistic regression, Lasso, and ResNet18 training for image classification. These benchmarks highlight key practical findings that give a more nuanced view of the state-of-the-art for these problems, showing that for practical evaluation, the devil is in the details. We hope that Benchopt will foster collaborative work in the community hence improving the reproducibility of research findings.","url_abs":"https://arxiv.org/abs/2206.13424v3","url_pdf":"https://arxiv.org/pdf/2206.13424v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"benchopt-reproducible-efficient-and","repo_url":"https://github.com/benchopt/benchopt","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"benchopt-reproducible-efficient-and","repo_url":"https://github.com/deepmind/optax","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"benchopt-reproducible-efficient-and","repo_url":"https://github.com/google-deepmind/optax","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"stochastic-optimization","task_name":"Stochastic Optimization"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"bottleneck-residual-block","method_name":"Bottleneck Residual Block"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"kaiming-initialization","method_name":"Kaiming Initialization"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-block","method_name":"Residual Block"},{"method_slug":"residual-connection","method_name":"Residual Connection"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-classification-on-cifar-10","task":"Image Classification","dataset":"CIFAR-10","model":"ResNet-18","rank_in_archive_order":130,"of":265,"metrics":{"Percentage correct":"95.55"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-svhn","task":"Image Classification","dataset":"SVHN","model":"ResNet-18","rank_in_archive_order":38,"of":62,"metrics":{"Percentage error":"2.65"},"uses_additional_data":false},{"leaderboard":"/sota/stochastic-optimization-on-cifar-10-resnet-18","task":"Stochastic Optimization","dataset":"CIFAR-10 ResNet-18 - 200 Epochs","model":"SGD - cosine LR schedule","rank_in_archive_order":1,"of":4,"metrics":{"Accuracy":"95.55"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2206.13424","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.13424"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/benchopt/benchmark_logreg_","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/benchopt/benchmark_lasso","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/benchopt/benchmark_resnet_","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/benchopt/benchmark_ridge","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/benchopt/benchmark_mcp","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/deepmind/optax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/benchopt/benchopt","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google-deepmind/optax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":10},"by_repo_kind":{"official":{"samples":10,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5e54113b03b3d98a","entry":"buffer_iterator","repo":"benchopt/benchopt","repo_kind":"official","path":"benchopt/_generate_runs.py","file_url":"https://github.com/benchopt/benchopt/blob/HEAD/benchopt/_generate_runs.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"5e54113b03b3d98a"}},{"code_sha256_prefix":"9484e5cb8dc4812e","entry":"get_data_path","repo":"benchopt/benchopt","repo_kind":"official","path":"benchopt/config.py","file_url":"https://github.com/benchopt/benchopt/blob/HEAD/benchopt/config.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"9484e5cb8dc4812e"}},{"code_sha256_prefix":"91a5c920fcf2ab00","entry":"get_file_hash","repo":"benchopt/benchopt","repo_kind":"official","path":"benchopt/utils/dynamic_modules.py","file_url":"https://github.com/benchopt/benchopt/blob/HEAD/benchopt/utils/dynamic_modules.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"91a5c920fcf2ab00"}},{"code_sha256_prefix":"25ccf1c080455614","entry":"get_setting","repo":"benchopt/benchopt","repo_kind":"official","path":"benchopt/config.py","file_url":"https://github.com/benchopt/benchopt/blob/HEAD/benchopt/config.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"25ccf1c080455614"}},{"code_sha256_prefix":"550d9e693411887a","entry":"get_solver_kwargs","repo":"benchopt/benchopt","repo_kind":"official","path":"benchopt/_generate_runs.py","file_url":"https://github.com/benchopt/benchopt/blob/HEAD/benchopt/_generate_runs.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"550d9e693411887a"}},{"code_sha256_prefix":"76f11a61835fb1c6","entry":"parse_value","repo":"benchopt/benchopt","repo_kind":"official","path":"benchopt/config.py","file_url":"https://github.com/benchopt/benchopt/blob/HEAD/benchopt/config.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"76f11a61835fb1c6"}},{"code_sha256_prefix":"04cef2687413f0eb","entry":"propose_from_list","repo":"benchopt/benchopt","repo_kind":"official","path":"benchopt/cli/completion.py","file_url":"https://github.com/benchopt/benchopt/blob/HEAD/benchopt/cli/completion.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"04cef2687413f0eb"}},{"code_sha256_prefix":"7143d36d09390525","entry":"run_benchmark","repo":"benchopt/benchopt","repo_kind":"official","path":"benchopt/runner.py","file_url":"https://github.com/benchopt/benchopt/blob/HEAD/benchopt/runner.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"7143d36d09390525"}},{"code_sha256_prefix":"7552746dabc0de85","entry":"run_one_resolution","repo":"benchopt/benchopt","repo_kind":"official","path":"benchopt/runner.py","file_url":"https://github.com/benchopt/benchopt/blob/HEAD/benchopt/runner.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"7552746dabc0de85"}},{"code_sha256_prefix":"d601cf4f35f083ea","entry":"run_one_to_cvg","repo":"benchopt/benchopt","repo_kind":"official","path":"benchopt/runner.py","file_url":"https://github.com/benchopt/benchopt/blob/HEAD/benchopt/runner.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"d601cf4f35f083ea"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}