{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/benchmarking-test-time-adaptation-against","title":"Benchmarking Test-Time Adaptation against Distribution Shifts in Image Classification","arxiv_id":"2307.03133","date":"2023-07-06","proceeding":null,"authors":["Yongcan Yu","Lijun Sheng","Ran He","Jian Liang"],"abstract":"Test-time adaptation (TTA) is a technique aimed at enhancing the generalization performance of models by leveraging unlabeled samples solely during prediction. Given the need for robustness in neural network systems when faced with distribution shifts, numerous TTA methods have recently been proposed. However, evaluating these methods is often done under different settings, such as varying distribution shifts, backbones, and designing scenarios, leading to a lack of consistent and fair benchmarks to validate their effectiveness. To address this issue, we present a benchmark that systematically evaluates 13 prominent TTA methods and their variants on five widely used image classification datasets: CIFAR-10-C, CIFAR-100-C, ImageNet-C, DomainNet, and Office-Home. These methods encompass a wide range of adaptation scenarios (e.g. online adaptation v.s. offline adaptation, instance adaptation v.s. batch adaptation v.s. domain adaptation). Furthermore, we explore the compatibility of different TTA methods with diverse network backbones. To implement this benchmark, we have developed a unified framework in PyTorch, which allows for consistent evaluation and comparison of the TTA methods across the different datasets and network architectures. By establishing this benchmark, we aim to provide researchers and practitioners with a reliable means of assessing and comparing the effectiveness of TTA methods in improving model robustness and generalization performance. Our code is available at https://github.com/yuyongcan/Benchmark-TTA.","url_abs":"https://arxiv.org/abs/2307.03133v1","url_pdf":"https://arxiv.org/pdf/2307.03133v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"benchmarking-test-time-adaptation-against","repo_url":"https://github.com/yuyongcan/benchmark-tta","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"domain-adaptation","task_name":"Domain Adaptation"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"test-time-adaptation","task_name":"Test-time Adaptation"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2307.03133","atlas_url":"https://app.syntology.ai/?focus=2307.03133","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.03133"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yuyongcan/benchmark-tta","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":3,"ran":1,"ran_honours":1,"unverified":3},"by_repo_kind":{"official":{"samples":8,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"35c2d572ebd185f5","entry":"image_train","repo":"yuyongcan/benchmark-tta","repo_kind":"official","path":"train_source.py","file_url":"https://github.com/yuyongcan/benchmark-tta/blob/HEAD/train_source.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"35c2d572ebd185f5"}},{"code_sha256_prefix":"0b7ffc9f8b77529c","entry":"lr_scheduler","repo":"yuyongcan/benchmark-tta","repo_kind":"official","path":"train_source.py","file_url":"https://github.com/yuyongcan/benchmark-tta/blob/HEAD/train_source.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0b7ffc9f8b77529c"}},{"code_sha256_prefix":"4aedf5bc9100546f","entry":"make_custom_dataset","repo":"yuyongcan/benchmark-tta","repo_kind":"official","path":"robustbench/loaders.py","file_url":"https://github.com/yuyongcan/benchmark-tta/blob/HEAD/robustbench/loaders.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4aedf5bc9100546f"}},{"code_sha256_prefix":"93a11f62e4a129f0","entry":"op_copy","repo":"yuyongcan/benchmark-tta","repo_kind":"official","path":"train_source.py","file_url":"https://github.com/yuyongcan/benchmark-tta/blob/HEAD/train_source.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"93a11f62e4a129f0"}},{"code_sha256_prefix":"f321f54723433661","entry":"pil_loader","repo":"yuyongcan/benchmark-tta","repo_kind":"official","path":"robustbench/loaders.py","file_url":"https://github.com/yuyongcan/benchmark-tta/blob/HEAD/robustbench/loaders.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f321f54723433661"}},{"code_sha256_prefix":"404fb2b2daa1ae78","entry":"accimage_loader","repo":"yuyongcan/benchmark-tta","repo_kind":"official","path":"robustbench/loaders.py","file_url":"https://github.com/yuyongcan/benchmark-tta/blob/HEAD/robustbench/loaders.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"404fb2b2daa1ae78"}},{"code_sha256_prefix":"2ef65c34f0fa8042","entry":"load_cifar10","repo":"yuyongcan/benchmark-tta","repo_kind":"official","path":"robustbench/data.py","file_url":"https://github.com/yuyongcan/benchmark-tta/blob/HEAD/robustbench/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2ef65c34f0fa8042"}},{"code_sha256_prefix":"52acd5b81cee97b1","entry":"load_cifar100","repo":"yuyongcan/benchmark-tta","repo_kind":"official","path":"robustbench/data.py","file_url":"https://github.com/yuyongcan/benchmark-tta/blob/HEAD/robustbench/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"52acd5b81cee97b1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}