{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/do-cifar-10-classifiers-generalize-to-cifar","title":"Do CIFAR-10 Classifiers Generalize to CIFAR-10?","arxiv_id":"1806.00451","date":"2018-06-01","proceeding":null,"authors":["Benjamin Recht","Rebecca Roelofs","Ludwig Schmidt","Vaishaal Shankar"],"abstract":"Machine learning is currently dominated by largely experimental work focused\non improvements in a few key tasks. However, the impressive accuracy numbers of\nthe best performing models are questionable because the same test sets have\nbeen used to select these models for multiple years now. To understand the\ndanger of overfitting, we measure the accuracy of CIFAR-10 classifiers by\ncreating a new test set of truly unseen images. Although we ensure that the new\ntest set is as close to the original data distribution as possible, we find a\nlarge drop in accuracy (4% to 10%) for a broad range of deep learning models.\nYet more recent models with higher original accuracy show a smaller drop and\nbetter overall performance, indicating that this drop is likely not due to\noverfitting based on adaptivity. Instead, we view our results as evidence that\ncurrent accuracy numbers are brittle and susceptible to even minute natural\nvariations in the data distribution.","url_abs":"http://arxiv.org/abs/1806.00451v1","url_pdf":"http://arxiv.org/pdf/1806.00451v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"do-cifar-10-classifiers-generalize-to-cifar","repo_url":"https://github.com/aisosalo/CIFAR-10","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"do-cifar-10-classifiers-generalize-to-cifar","repo_url":"https://github.com/modestyachts/CIFAR-10.1","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"do-cifar-10-classifiers-generalize-to-cifar","repo_url":"https://github.com/yueatsprograms/ttt_cifar_release","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1806.00451","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1806.00451"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/modestyachts/CIFAR-10.1","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yueatsprograms/ttt_cifar_release","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aisosalo/CIFAR-10","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":1,"unverified":10},"by_repo_kind":{"listed":{"samples":11,"ran":1,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ae13c8ccf56adae8","entry":"np_to_png","repo":"modestyachts/CIFAR-10.1","repo_kind":"listed","path":"code/utils.py","file_url":"https://github.com/modestyachts/CIFAR-10.1/blob/HEAD/code/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ae13c8ccf56adae8"}},{"code_sha256_prefix":"ab5cf2c3305cebc8","entry":"aslist","repo":"aisosalo/CIFAR-10","repo_kind":"listed","path":"imageclassification/training/arguments.py","file_url":"https://github.com/aisosalo/CIFAR-10/blob/HEAD/imageclassification/training/arguments.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ab5cf2c3305cebc8"}},{"code_sha256_prefix":"e28e30f82310aaae","entry":"build_cifar100_metadata","repo":"aisosalo/CIFAR-10","repo_kind":"listed","path":"build_dataset.py","file_url":"https://github.com/aisosalo/CIFAR-10/blob/HEAD/build_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e28e30f82310aaae"}},{"code_sha256_prefix":"fb3c67bcbad48f16","entry":"build_cifar10_metadata","repo":"aisosalo/CIFAR-10","repo_kind":"listed","path":"build_dataset.py","file_url":"https://github.com/aisosalo/CIFAR-10/blob/HEAD/build_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fb3c67bcbad48f16"}},{"code_sha256_prefix":"5e66e1cf8a48f819","entry":"ev","repo":"aisosalo/CIFAR-10","repo_kind":"listed","path":"run_evaluation.py","file_url":"https://github.com/aisosalo/CIFAR-10/blob/HEAD/run_evaluation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5e66e1cf8a48f819"}},{"code_sha256_prefix":"d0941755b1d1eba6","entry":"init_dataset","repo":"aisosalo/CIFAR-10","repo_kind":"listed","path":"imageclassification/training/dataset.py","file_url":"https://github.com/aisosalo/CIFAR-10/blob/HEAD/imageclassification/training/dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d0941755b1d1eba6"}},{"code_sha256_prefix":"be2b61117e25938d","entry":"init_mean_std","repo":"aisosalo/CIFAR-10","repo_kind":"listed","path":"imageclassification/training/transformations.py","file_url":"https://github.com/aisosalo/CIFAR-10/blob/HEAD/imageclassification/training/transformations.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"be2b61117e25938d"}},{"code_sha256_prefix":"a9822666515c5938","entry":"load_distances_to_cifar10","repo":"modestyachts/CIFAR-10.1","repo_kind":"listed","path":"code/utils.py","file_url":"https://github.com/modestyachts/CIFAR-10.1/blob/HEAD/code/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a9822666515c5938"}},{"code_sha256_prefix":"c21e963cc74afdd7","entry":"load_new_test_data","repo":"modestyachts/CIFAR-10.1","repo_kind":"listed","path":"code/utils.py","file_url":"https://github.com/modestyachts/CIFAR-10.1/blob/HEAD/code/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c21e963cc74afdd7"}},{"code_sha256_prefix":"b8200f6c2f73edda","entry":"unpickle","repo":"aisosalo/CIFAR-10","repo_kind":"listed","path":"build_dataset.py","file_url":"https://github.com/aisosalo/CIFAR-10/blob/HEAD/build_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b8200f6c2f73edda"}},{"code_sha256_prefix":"54d88bd375890f88","entry":"unpickle","repo":"aisosalo/CIFAR-10","repo_kind":"listed","path":"imageclassification/training/dataset.py","file_url":"https://github.com/aisosalo/CIFAR-10/blob/HEAD/imageclassification/training/dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"54d88bd375890f88"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}