{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-robust-and-reproducible-active","title":"Towards Robust and Reproducible Active Learning Using Neural Networks","arxiv_id":"2002.09564","date":"2020-02-21","proceeding":"CVPR 2022 1","authors":["Prateek Munjal","Nasir Hayat","Munawar Hayat","Jamshid Sourati","Shadab Khan"],"abstract":"Active learning (AL) is a promising ML paradigm that has the potential to parse through large unlabeled data and help reduce annotation cost in domains where labeling data can be prohibitive. Recently proposed neural network based AL methods use different heuristics to accomplish this goal. In this study, we demonstrate that under identical experimental settings, different types of AL algorithms (uncertainty based, diversity based, and committee based) produce an inconsistent gain over random sampling baseline. Through a variety of experiments, controlling for sources of stochasticity, we show that variance in performance metrics achieved by AL algorithms can lead to results that are not consistent with the previously reported results. We also found that under strong regularization, AL methods show marginal or no advantage over the random sampling baseline under a variety of experimental conditions. Finally, we conclude with a set of recommendations on how to assess the results using a new AL algorithm to ensure results are reproducible and robust under changes in experimental conditions. We share our codes to facilitate AL evaluations. We believe our findings and recommendations will help advance reproducible research in AL using neural networks. We open source our code at https://github.com/PrateekMunjal/TorchAL","url_abs":"https://arxiv.org/abs/2002.09564v3","url_pdf":"https://arxiv.org/pdf/2002.09564v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-robust-and-reproducible-active","repo_url":"https://github.com/prateekmunjal/torchal","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"towards-robust-and-reproducible-active","repo_url":"https://github.com/acl21/deep-active-learning-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"active-learning","task_name":"Active Learning"},{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"image-classification","task_name":"Image Classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/active-learning-on-cifar10-10000","task":"Active Learning","dataset":"CIFAR10 (10,000)","model":"Random Baseline (Resnet18)","rank_in_archive_order":6,"of":7,"metrics":{"Accuracy":"88.45"},"uses_additional_data":false},{"leaderboard":"/sota/active-learning-on-cifar10-10000","task":"Active Learning","dataset":"CIFAR10 (10,000)","model":"Random Baseline (VGG16)","rank_in_archive_order":7,"of":7,"metrics":{"Accuracy":"85.09"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2002.09564","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2002.09564"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/acl21/deep-active-learning-pytorch","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/prateekmunjal/torchal","reach":null}],"summary":{"ran_draft_wrong":3,"ran_violates":2,"ran":1,"unverified":6},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":11,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2a80220dabcb742a","entry":"conv1x1","repo":"acl21/deep-active-learning-pytorch","repo_kind":"listed","path":"pycls/models/resnet.py","file_url":"https://github.com/acl21/deep-active-learning-pytorch/blob/HEAD/pycls/models/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2a80220dabcb742a"}},{"code_sha256_prefix":"600ff2c45e0de056","entry":"conv3x3","repo":"acl21/deep-active-learning-pytorch","repo_kind":"listed","path":"pycls/models/resnet.py","file_url":"https://github.com/acl21/deep-active-learning-pytorch/blob/HEAD/pycls/models/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"600ff2c45e0de056"}},{"code_sha256_prefix":"79583b4154444ddf","entry":"drop_connect","repo":"acl21/deep-active-learning-pytorch","repo_kind":"listed","path":"pycls/utils/net.py","file_url":"https://github.com/acl21/deep-active-learning-pytorch/blob/HEAD/pycls/utils/net.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"79583b4154444ddf"}},{"code_sha256_prefix":"24d376b7f841ed4c","entry":"is_eval_epoch","repo":"prateekmunjal/torchal","repo_kind":"official","path":"tools/train_al.py","file_url":"https://github.com/prateekmunjal/torchal/blob/HEAD/tools/train_al.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"24d376b7f841ed4c"}},{"code_sha256_prefix":"0bac0ac5d129be80","entry":"make_layers","repo":"acl21/deep-active-learning-pytorch","repo_kind":"listed","path":"pycls/models/vgg.py","file_url":"https://github.com/acl21/deep-active-learning-pytorch/blob/HEAD/pycls/models/vgg.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0bac0ac5d129be80"}},{"code_sha256_prefix":"8ba24560996e7d76","entry":"unwrap_model","repo":"acl21/deep-active-learning-pytorch","repo_kind":"listed","path":"pycls/core/net.py","file_url":"https://github.com/acl21/deep-active-learning-pytorch/blob/HEAD/pycls/core/net.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8ba24560996e7d76"}},{"code_sha256_prefix":"1d3e648204ca8f7f","entry":"alexnet","repo":"acl21/deep-active-learning-pytorch","repo_kind":"listed","path":"pycls/models/alexnet.py","file_url":"https://github.com/acl21/deep-active-learning-pytorch/blob/HEAD/pycls/models/alexnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1d3e648204ca8f7f"}},{"code_sha256_prefix":"08ca33c7fa25c77b","entry":"construct_optimizer","repo":"acl21/deep-active-learning-pytorch","repo_kind":"listed","path":"pycls/core/optimizer.py","file_url":"https://github.com/acl21/deep-active-learning-pytorch/blob/HEAD/pycls/core/optimizer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"08ca33c7fa25c77b"}},{"code_sha256_prefix":"b26a6d70c4c1b982","entry":"get_flat_weights","repo":"acl21/deep-active-learning-pytorch","repo_kind":"listed","path":"pycls/utils/net.py","file_url":"https://github.com/acl21/deep-active-learning-pytorch/blob/HEAD/pycls/utils/net.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b26a6d70c4c1b982"}},{"code_sha256_prefix":"2ea7d3298243fe68","entry":"lr_fun_exp","repo":"acl21/deep-active-learning-pytorch","repo_kind":"listed","path":"pycls/core/optimizer.py","file_url":"https://github.com/acl21/deep-active-learning-pytorch/blob/HEAD/pycls/core/optimizer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2ea7d3298243fe68"}},{"code_sha256_prefix":"89e97f7341666d97","entry":"lr_fun_steps","repo":"acl21/deep-active-learning-pytorch","repo_kind":"listed","path":"pycls/core/optimizer.py","file_url":"https://github.com/acl21/deep-active-learning-pytorch/blob/HEAD/pycls/core/optimizer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"89e97f7341666d97"}},{"code_sha256_prefix":"5f276f6de9406cd1","entry":"resnet18","repo":"acl21/deep-active-learning-pytorch","repo_kind":"listed","path":"pycls/models/resnet.py","file_url":"https://github.com/acl21/deep-active-learning-pytorch/blob/HEAD/pycls/models/resnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5f276f6de9406cd1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}