{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-cross-domain-benchmark-for-active-learning","title":"A Cross-Domain Benchmark for Active Learning","arxiv_id":"2408.00426","date":"2024-08-01","proceeding":null,"authors":["Thorben Werner","Johannes Burchert","Maximilian Stubbemann","Lars Schmidt-Thieme"],"abstract":"Active Learning (AL) deals with identifying the most informative samples for labeling to reduce data annotation costs for supervised learning tasks. AL research suffers from the fact that lifts from literature generalize poorly and that only a small number of repetitions of experiments are conducted. To overcome these obstacles, we propose CDALBench, the first active learning benchmark which includes tasks in computer vision, natural language processing and tabular learning. Furthermore, by providing an efficient, greedy oracle, CDALBench can be evaluated with 50 runs for each experiment. We show, that both the cross-domain character and a large amount of repetitions are crucial for sophisticated evaluation of AL research. Concretely, we show that the superiority of specific methods varies over the different domains, making it important to evaluate Active Learning with a cross-domain benchmark. Additionally, we show that having a large amount of runs is crucial. With only conducting three runs as often done in the literature, the superiority of specific methods can strongly vary with the specific runs. This effect is so strong, that, depending on the seed, even a well-established method's performance can be significantly better and significantly worse than random for the same dataset.","url_abs":"https://arxiv.org/abs/2408.00426v2","url_pdf":"https://arxiv.org/pdf/2408.00426v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-cross-domain-benchmark-for-active-learning","repo_url":"https://github.com/wernerth94/a-cross-domain-benchmark-for-active-learning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"active-learning","task_name":"Active Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2408.00426","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.00426"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wernerth94/a-cross-domain-benchmark-for-active-learning","reach":{"status":"ok"}}],"summary":{"ran":10,"unverified":1},"by_repo_kind":{"official":{"samples":11,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"9cd7f610b76f1d74","entry":"BCEAdjLoss","repo":"wernerth94/a-cross-domain-benchmark-for-active-learning","repo_kind":"official","path":"agents/core_gcn.py","file_url":"https://github.com/wernerth94/a-cross-domain-benchmark-for-active-learning/blob/HEAD/agents/core_gcn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9cd7f610b76f1d74"}},{"code_sha256_prefix":"21ea0fc5cdae1d1f","entry":"aff_to_adj","repo":"wernerth94/a-cross-domain-benchmark-for-active-learning","repo_kind":"official","path":"agents/core_gcn.py","file_url":"https://github.com/wernerth94/a-cross-domain-benchmark-for-active-learning/blob/HEAD/agents/core_gcn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"21ea0fc5cdae1d1f"}},{"code_sha256_prefix":"7a641086c1ff376f","entry":"compute_ranks_over_trials","repo":"wernerth94/a-cross-domain-benchmark-for-active-learning","repo_kind":"official","path":"core/evaluation.py","file_url":"https://github.com/wernerth94/a-cross-domain-benchmark-for-active-learning/blob/HEAD/core/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7a641086c1ff376f"}},{"code_sha256_prefix":"8cdb9dc547d4b794","entry":"get_init_pool_size","repo":"wernerth94/a-cross-domain-benchmark-for-active-learning","repo_kind":"official","path":"core/helper_functions.py","file_url":"https://github.com/wernerth94/a-cross-domain-benchmark-for-active-learning/blob/HEAD/core/helper_functions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8cdb9dc547d4b794"}},{"code_sha256_prefix":"af3ee80e6d3d3a29","entry":"kaiming_uniform_seeded","repo":"wernerth94/a-cross-domain-benchmark-for-active-learning","repo_kind":"official","path":"classifiers/seeded_layers.py","file_url":"https://github.com/wernerth94/a-cross-domain-benchmark-for-active-learning/blob/HEAD/classifiers/seeded_layers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"af3ee80e6d3d3a29"}},{"code_sha256_prefix":"d35bc6ccbb32273f","entry":"moving_avrg","repo":"wernerth94/a-cross-domain-benchmark-for-active-learning","repo_kind":"official","path":"core/helper_functions.py","file_url":"https://github.com/wernerth94/a-cross-domain-benchmark-for-active-learning/blob/HEAD/core/helper_functions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d35bc6ccbb32273f"}},{"code_sha256_prefix":"e3961104b9c7f2f5","entry":"normalize","repo":"wernerth94/a-cross-domain-benchmark-for-active-learning","repo_kind":"official","path":"core/data.py","file_url":"https://github.com/wernerth94/a-cross-domain-benchmark-for-active-learning/blob/HEAD/core/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e3961104b9c7f2f5"}},{"code_sha256_prefix":"d9288a12e176f28f","entry":"sort_according_to_reference","repo":"wernerth94/a-cross-domain-benchmark-for-active-learning","repo_kind":"official","path":"core/evaluation.py","file_url":"https://github.com/wernerth94/a-cross-domain-benchmark-for-active-learning/blob/HEAD/core/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d9288a12e176f28f"}},{"code_sha256_prefix":"755ef2a549fdd690","entry":"subsample_data","repo":"wernerth94/a-cross-domain-benchmark-for-active-learning","repo_kind":"official","path":"core/data.py","file_url":"https://github.com/wernerth94/a-cross-domain-benchmark-for-active-learning/blob/HEAD/core/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"755ef2a549fdd690"}},{"code_sha256_prefix":"707898a271c298d2","entry":"to_torch","repo":"wernerth94/a-cross-domain-benchmark-for-active-learning","repo_kind":"official","path":"core/data.py","file_url":"https://github.com/wernerth94/a-cross-domain-benchmark-for-active-learning/blob/HEAD/core/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"707898a271c298d2"}},{"code_sha256_prefix":"1928f50ad5ade08b","entry":"plot_upper_bound","repo":"wernerth94/a-cross-domain-benchmark-for-active-learning","repo_kind":"official","path":"core/helper_functions.py","file_url":"https://github.com/wernerth94/a-cross-domain-benchmark-for-active-learning/blob/HEAD/core/helper_functions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1928f50ad5ade08b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}