{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/what-a-mess-multi-domain-evaluation-of-zero","title":"What a MESS: Multi-Domain Evaluation of Zero-Shot Semantic Segmentation","arxiv_id":"2306.15521","date":"2023-06-27","proceeding":"NeurIPS 2023 11","authors":["Benedikt Blumenstiel","Johannes Jakubik","Hilde Kühne","Michael Vössing"],"abstract":"While semantic segmentation has seen tremendous improvements in the past, there are still significant labeling efforts necessary and the problem of limited generalization to classes that have not been present during training. To address this problem, zero-shot semantic segmentation makes use of large self-supervised vision-language models, allowing zero-shot transfer to unseen classes. In this work, we build a benchmark for Multi-domain Evaluation of Semantic Segmentation (MESS), which allows a holistic analysis of performance across a wide range of domain-specific datasets such as medicine, engineering, earth monitoring, biology, and agriculture. To do this, we reviewed 120 datasets, developed a taxonomy, and classified the datasets according to the developed taxonomy. We select a representative subset consisting of 22 datasets and propose it as the MESS benchmark. We evaluate eight recently published models on the proposed MESS benchmark and analyze characteristics for the performance of zero-shot transfer models. The toolkit is available at https://github.com/blumenstiel/MESS.","url_abs":"https://arxiv.org/abs/2306.15521v3","url_pdf":"https://arxiv.org/pdf/2306.15521v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"what-a-mess-multi-domain-evaluation-of-zero","repo_url":"https://github.com/blumenstiel/mess","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"zero-shot-semantic-segmentation","task_name":"Zero-Shot Semantic Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2306.15521","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.15521"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/blumenstiel/MESS","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/blumenstiel/mess","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a612d83c1febe16b","entry":"combine_paxray","repo":"blumenstiel/MESS","repo_kind":"official","path":"mess/evaluation/mess_evaluation.py","file_url":"https://github.com/blumenstiel/MESS/blob/HEAD/mess/evaluation/mess_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a612d83c1febe16b"}},{"code_sha256_prefix":"dd117dca963afb93","entry":"get_class_names","repo":"blumenstiel/MESS","repo_kind":"official","path":"mess/datasets/MMSegDataset.py","file_url":"https://github.com/blumenstiel/MESS/blob/HEAD/mess/datasets/MMSegDataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dd117dca963afb93"}},{"code_sha256_prefix":"080cb1a082ca2feb","entry":"get_tiles","repo":"blumenstiel/mess","repo_kind":"official","path":"mess/prepare_datasets/prepare_floodnet.py","file_url":"https://github.com/blumenstiel/mess/blob/HEAD/mess/prepare_datasets/prepare_floodnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"080cb1a082ca2feb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}