{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tune-it-or-don-t-use-it-benchmarking-data","title":"Tune It or Don't Use It: Benchmarking Data-Efficient Image Classification","arxiv_id":"2108.13122","date":"2021-08-30","proceeding":null,"authors":["Lorenzo Brigato","Björn Barz","Luca Iocchi","Joachim Denzler"],"abstract":"Data-efficient image classification using deep neural networks in settings, where only small amounts of labeled data are available, has been an active research area in the recent past. However, an objective comparison between published methods is difficult, since existing works use different datasets for evaluation and often compare against untuned baselines with default hyper-parameters. We design a benchmark for data-efficient image classification consisting of six diverse datasets spanning various domains (e.g., natural images, medical imagery, satellite data) and data types (RGB, grayscale, multispectral). Using this benchmark, we re-evaluate the standard cross-entropy baseline and eight methods for data-efficient deep learning published between 2017 and 2021 at renowned venues. For a fair and realistic comparison, we carefully tune the hyper-parameters of all methods on each dataset. Surprisingly, we find that tuning learning rate, weight decay, and batch size on a separate validation split results in a highly competitive baseline, which outperforms all but one specialized method and performs competitively to the remaining one.","url_abs":"https://arxiv.org/abs/2108.13122v1","url_pdf":"https://arxiv.org/pdf/2108.13122v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tune-it-or-don-t-use-it-benchmarking-data","repo_url":"https://github.com/cvjena/deic","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"small-data","task_name":"Small Data Image Classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[],"datasets_introduced":[{"slug":"deic-benchmark","name":"DEIC Benchmark","full_name":"Data-Efficient Image Classification Benchmark"},{"slug":"imagenet-50-samples-per-class","name":"ImageNet 50 samples per class","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/small-data-on-cub-200-2011-30-samples-per-1","task":"Small Data Image Classification","dataset":"CUB-200-2011, 30 samples per class","model":"Harmonic Networks (no pre-training)","rank_in_archive_order":2,"of":4,"metrics":{"Accuracy":"72.26"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-on-cub-200-2011-30-samples-per-1","task":"Small Data Image Classification","dataset":"CUB-200-2011, 30 samples per class","model":"Cross-entropy baseline (no pre-training)","rank_in_archive_order":3,"of":4,"metrics":{"Accuracy":"71.44"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-on-cub-200-2011-30-samples-per-1","task":"Small Data Image Classification","dataset":"CUB-200-2011, 30 samples per class","model":"DSK Networks (no pre-training)","rank_in_archive_order":4,"of":4,"metrics":{"Accuracy":"71.02"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-deic","task":"Small Data Image Classification","dataset":"DEIC Benchmark","model":"Harmonic Networks","rank_in_archive_order":1,"of":10,"metrics":{"Average Balanced Accuracy (across datasets)":"68.70"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-deic","task":"Small Data Image Classification","dataset":"DEIC Benchmark","model":"Cross-Entropy baseline","rank_in_archive_order":2,"of":10,"metrics":{"Average Balanced Accuracy (across datasets)":"67.90"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-deic","task":"Small Data Image Classification","dataset":"DEIC Benchmark","model":"Cosine + Cross-Entropy Loss","rank_in_archive_order":3,"of":10,"metrics":{"Average Balanced Accuracy (across datasets)":"64.92"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-deic","task":"Small Data Image Classification","dataset":"DEIC Benchmark","model":"T-vMF Similarity","rank_in_archive_order":4,"of":10,"metrics":{"Average Balanced Accuracy (across datasets)":"64.67"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-deic","task":"Small Data Image Classification","dataset":"DEIC Benchmark","model":"DSK Networks","rank_in_archive_order":5,"of":10,"metrics":{"Average Balanced Accuracy (across datasets)":"64.64"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-deic","task":"Small Data Image Classification","dataset":"DEIC Benchmark","model":"OLÉ","rank_in_archive_order":6,"of":10,"metrics":{"Average Balanced Accuracy (across datasets)":"64.15"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-deic","task":"Small Data Image Classification","dataset":"DEIC Benchmark","model":"Cosine Loss","rank_in_archive_order":7,"of":10,"metrics":{"Average Balanced Accuracy (across datasets)":"62.73"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-deic","task":"Small Data Image Classification","dataset":"DEIC Benchmark","model":"Full Convolution","rank_in_archive_order":8,"of":10,"metrics":{"Average Balanced Accuracy (across datasets)":"62.06"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-deic","task":"Small Data Image Classification","dataset":"DEIC Benchmark","model":"Deep Hybrid Networks","rank_in_archive_order":9,"of":10,"metrics":{"Average Balanced Accuracy (across datasets)":"60.33"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-deic","task":"Small Data Image Classification","dataset":"DEIC Benchmark","model":"Grad-l2 Penalty","rank_in_archive_order":10,"of":10,"metrics":{"Average Balanced Accuracy (across datasets)":"55.47"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-eurosat-50","task":"Small Data Image Classification","dataset":"EuroSAT 50 samples per class","model":"Harmonic Networks","rank_in_archive_order":1,"of":3,"metrics":{"Accuracy":"92.09"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-eurosat-50","task":"Small Data Image Classification","dataset":"EuroSAT 50 samples per class","model":"DSK Networks","rank_in_archive_order":2,"of":3,"metrics":{"Accuracy":"91.25"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-eurosat-50","task":"Small Data Image Classification","dataset":"EuroSAT 50 samples per class","model":"Deep Hybrid Networks","rank_in_archive_order":3,"of":3,"metrics":{"Accuracy":"91.15"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-imagenet","task":"Small Data Image Classification","dataset":"ImageNet 50 samples per class","model":"Harmonic Networks","rank_in_archive_order":1,"of":3,"metrics":{"1:1 Accuracy":"46.36"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-imagenet","task":"Small Data Image Classification","dataset":"ImageNet 50 samples per class","model":"DSK Networks","rank_in_archive_order":2,"of":3,"metrics":{"1:1 Accuracy":"45.21"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-imagenet","task":"Small Data Image Classification","dataset":"ImageNet 50 samples per class","model":"Cross-entropy baseline","rank_in_archive_order":3,"of":3,"metrics":{"1:1 Accuracy":"44.97"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-cifair-10-1","task":"Small Data Image Classification","dataset":"ciFAIR-10 50 samples per class","model":"Cross-entropy baseline","rank_in_archive_order":3,"of":5,"metrics":{"Accuracy":"58.22"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-cifair-10-1","task":"Small Data Image Classification","dataset":"ciFAIR-10 50 samples per class","model":"T-vMF Similarity","rank_in_archive_order":4,"of":5,"metrics":{"Accuracy":"57.50"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-cifair-10-1","task":"Small Data Image Classification","dataset":"ciFAIR-10 50 samples per class","model":"Harmonic Networks","rank_in_archive_order":5,"of":5,"metrics":{"Accuracy":"56.50"},"uses_additional_data":false}],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}