{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/spawrious-a-benchmark-for-fine-control-of","title":"Spawrious: A Benchmark for Fine Control of Spurious Correlation Biases","arxiv_id":"2303.05470","date":"2023-03-09","proceeding":null,"authors":["Aengus Lynch","Gbètondji J-S Dovonon","Jean Kaddour","Ricardo Silva"],"abstract":"The problem of spurious correlations (SCs) arises when a classifier relies on non-predictive features that happen to be correlated with the labels in the training data. For example, a classifier may misclassify dog breeds based on the background of dog images. This happens when the backgrounds are correlated with other breeds in the training data, leading to misclassifications during test time. Previous SC benchmark datasets suffer from varying issues, e.g., over-saturation or only containing one-to-one (O2O) SCs, but no many-to-many (M2M) SCs arising between groups of spurious attributes and classes. In this paper, we present \\benchmark-\\{O2O, M2M\\}-\\{Easy, Medium, Hard\\}, an image classification benchmark suite containing spurious correlations between classes and backgrounds. To create this dataset, we employ a text-to-image model to generate photo-realistic images and an image captioning model to filter out unsuitable ones. The resulting dataset is of high quality and contains approximately 152k images. Our experimental results demonstrate that state-of-the-art group robustness methods struggle with \\benchmark, most notably on the Hard-splits with none of them getting over $70\\%$ accuracy on the hardest split using a ResNet50 pretrained on ImageNet. By examining model misclassifications, we detect reliances on spurious backgrounds, demonstrating that our dataset provides a significant challenge.","url_abs":"https://arxiv.org/abs/2303.05470v3","url_pdf":"https://arxiv.org/pdf/2303.05470v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"spawrious-a-benchmark-for-fine-control-of","repo_url":"https://github.com/aengusl/spawrious","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"spawrious-a-benchmark-for-fine-control-of","repo_url":"https://github.com/facebookresearch/DomainBed","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":null,"method_name":"None"},{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2303.05470","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.05470"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/DomainBed","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aengusl/spawrious","reach":null}],"summary":{"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2521fe4e47b71cff","entry":"build_combination","repo":"aengusl/spawrious","repo_kind":"official","path":"spawrious/torch.py","file_url":"https://github.com/aengusl/spawrious/blob/HEAD/spawrious/torch.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2521fe4e47b71cff"}},{"code_sha256_prefix":"3219784454724dde","entry":"evaluate","repo":"aengusl/spawrious","repo_kind":"official","path":"example.py","file_url":"https://github.com/aengusl/spawrious/blob/HEAD/example.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3219784454724dde"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}