{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/chils-zero-shot-image-classification-with","title":"CHiLS: Zero-Shot Image Classification with Hierarchical Label Sets","arxiv_id":"2302.02551","date":"2023-02-06","proceeding":null,"authors":["Zachary Novack","Julian McAuley","Zachary C. Lipton","Saurabh Garg"],"abstract":"Open vocabulary models (e.g. CLIP) have shown strong performance on zero-shot classification through their ability generate embeddings for each class based on their (natural language) names. Prior work has focused on improving the accuracy of these models through prompt engineering or by incorporating a small amount of labeled downstream data (via finetuning). However, there has been little focus on improving the richness of the class names themselves, which can pose issues when class labels are coarsely-defined and are uninformative. We propose Classification with Hierarchical Label Sets (or CHiLS), an alternative strategy for zero-shot classification specifically designed for datasets with implicit semantic hierarchies. CHiLS proceeds in three steps: (i) for each class, produce a set of subclasses, using either existing label hierarchies or by querying GPT-3; (ii) perform the standard zero-shot CLIP procedure as though these subclasses were the labels of interest; (iii) map the predicted subclass back to its parent to produce the final prediction. Across numerous datasets with underlying hierarchical structure, CHiLS leads to improved accuracy in situations both with and without ground-truth hierarchical information. CHiLS is simple to implement within existing zero-shot pipelines and requires no additional training cost. Code is available at: https://github.com/acmi-lab/CHILS.","url_abs":"https://arxiv.org/abs/2302.02551v3","url_pdf":"https://arxiv.org/pdf/2302.02551v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"chils-zero-shot-image-classification-with","repo_url":"https://github.com/acmi-lab/chils","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"prompt-engineering","task_name":"Prompt Engineering"},{"task_slug":"zero-shot-image-classification","task_name":"Zero-Shot Image Classification"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"},{"task_slug":"image-classification","task_name":"image-classification"},{"task_slug":null,"task_name":"zero-shot-classification"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2302.02551","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2302.02551"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/acmi-lab/chils","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":10},"by_repo_kind":{"official":{"samples":11,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2129574ed34dcb74","entry":"dataset_with_indices","repo":"acmi-lab/chils","repo_kind":"official","path":"src/zshot_utils.py","file_url":"https://github.com/acmi-lab/chils/blob/HEAD/src/zshot_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2129574ed34dcb74"}},{"code_sha256_prefix":"5b57ee54bdf0b7a5","entry":"dataset_with_indices","repo":"acmi-lab/chils","repo_kind":"official","path":"src/data_utils.py","file_url":"https://github.com/acmi-lab/chils/blob/HEAD/src/data_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5b57ee54bdf0b7a5"}},{"code_sha256_prefix":"234ae9cbbfa4fe06","entry":"get_CLIP_inputs_from_dict","repo":"acmi-lab/chils","repo_kind":"official","path":"src/zshot_utils.py","file_url":"https://github.com/acmi-lab/chils/blob/HEAD/src/zshot_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"234ae9cbbfa4fe06"}},{"code_sha256_prefix":"1cca8cd1fc9f507f","entry":"get_class_name","repo":"acmi-lab/chils","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/acmi-lab/chils/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1cca8cd1fc9f507f"}},{"code_sha256_prefix":"6bdaa18bfcbe2a56","entry":"get_dict_hash","repo":"acmi-lab/chils","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/acmi-lab/chils/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6bdaa18bfcbe2a56"}},{"code_sha256_prefix":"f72e18409d2e2c29","entry":"get_model","repo":"acmi-lab/chils","repo_kind":"official","path":"src/model_utils.py","file_url":"https://github.com/acmi-lab/chils/blob/HEAD/src/model_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f72e18409d2e2c29"}},{"code_sha256_prefix":"48287a6b9a790584","entry":"list_files","repo":"acmi-lab/chils","repo_kind":"official","path":"src/simple_utils.py","file_url":"https://github.com/acmi-lab/chils/blob/HEAD/src/simple_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"48287a6b9a790584"}},{"code_sha256_prefix":"795a168f1c1503bb","entry":"load_config","repo":"acmi-lab/chils","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/acmi-lab/chils/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"795a168f1c1503bb"}},{"code_sha256_prefix":"29353942db28fc10","entry":"load_pickle","repo":"acmi-lab/chils","repo_kind":"official","path":"src/simple_utils.py","file_url":"https://github.com/acmi-lab/chils/blob/HEAD/src/simple_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"29353942db28fc10"}},{"code_sha256_prefix":"2dcfaad02e7565df","entry":"query_gpt_prompt","repo":"acmi-lab/chils","repo_kind":"official","path":"models/clip_models.py","file_url":"https://github.com/acmi-lab/chils/blob/HEAD/models/clip_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2dcfaad02e7565df"}},{"code_sha256_prefix":"b60aa51289c404bd","entry":"query_gpt_prompt","repo":"acmi-lab/chils","repo_kind":"official","path":"src/zshot_utils.py","file_url":"https://github.com/acmi-lab/chils/blob/HEAD/src/zshot_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b60aa51289c404bd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}