{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unconstrained-open-vocabulary-image","title":"Unconstrained Open Vocabulary Image Classification: Zero-Shot Transfer from Text to Image via CLIP Inversion","arxiv_id":"2407.11211","date":"2024-07-15","proceeding":null,"authors":["Philipp Allgeuer","Kyra Ahrens","Stefan Wermter"],"abstract":"We introduce NOVIC, an innovative real-time uNconstrained Open Vocabulary Image Classifier that uses an autoregressive transformer to generatively output classification labels as language. Leveraging the extensive knowledge of CLIP models, NOVIC harnesses the embedding space to enable zero-shot transfer from pure text to images. Traditional CLIP models, despite their ability for open vocabulary classification, require an exhaustive prompt of potential class labels, restricting their application to images of known content or context. To address this, we propose an \"object decoder\" model that is trained on a large-scale 92M-target dataset of templated object noun sets and LLM-generated captions to always output the object noun in question. This effectively inverts the CLIP text encoder and allows textual object labels from essentially the entire English language to be generated directly from image-derived embedding vectors, without requiring any a priori knowledge of the potential content of an image, and without any label biases. The trained decoders are tested on a mix of manually and web-curated datasets, as well as standard image classification benchmarks, and achieve fine-grained prompt-free prediction scores of up to 87.5%, a strong result considering the model must work for any conceivable image and without any contextual clues.","url_abs":"https://arxiv.org/abs/2407.11211v4","url_pdf":"https://arxiv.org/pdf/2407.11211v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unconstrained-open-vocabulary-image","repo_url":"https://github.com/pallgeuer/novic","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"unconstrained-open-vocabulary-image","repo_url":"https://github.com/pallgeuer/object_noun_dictionary","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"open-vocabulary-image-classification","task_name":"Open Vocabulary Image Classification"},{"task_slug":"open-vocabulary-object-detection","task_name":"Open Vocabulary Object Detection"},{"task_slug":"text-classification","task_name":"Text Classification"},{"task_slug":"zero-shot-image-classification","task_name":"Zero-Shot Image Classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[{"slug":"novic-caption-object-data","name":"NOVIC Caption-Object Data","full_name":""},{"slug":"ovic-datasets","name":"OVIC Datasets","full_name":"Open Vocabulary Image Classification Datasets"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"DFN-5B H/14-378 + PrefixedIter Decoder","rank_in_archive_order":56,"of":1060,"metrics":{"Top 1 Accuracy":"88.21%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"SigLIP B/16 + PrefixedIter Decoder","rank_in_archive_order":424,"of":1060,"metrics":{"Top 1 Accuracy":"83.46%"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-image-classification-on-ovic-2","task":"Open Vocabulary Image Classification","dataset":"OVIC Datasets (Val3K)","model":"SigLIP B/16 + PrefixedIter Decoder (FT6)","rank_in_archive_order":1,"of":2,"metrics":{"Prediction Score (mean of 3)":"76.50","Top 1 Accuracy (mean of 3)":"75.04"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-image-classification-on-ovic-2","task":"Open Vocabulary Image Classification","dataset":"OVIC Datasets (Val3K)","model":"SigLIP B/16 + PrefixedIter Decoder (FT2)","rank_in_archive_order":2,"of":2,"metrics":{"Prediction Score (mean of 3)":"74.35","Top 1 Accuracy (mean of 3)":"72.90"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-image-classification-on-ovic-1","task":"Open Vocabulary Image Classification","dataset":"OVIC Datasets (Wiki-H)","model":"DFN-5B H/14-378 + PrefixedIter Decoder (FT2)","rank_in_archive_order":1,"of":3,"metrics":{"Overall Score":"79.02","Prediction Score":"80.13","Top 1 Accuracy":"77.05"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-image-classification-on-ovic-1","task":"Open Vocabulary Image Classification","dataset":"OVIC Datasets (Wiki-H)","model":"DFN-5B H/14-378 + PrefixedIter Decoder (FT0)","rank_in_archive_order":2,"of":3,"metrics":{"Overall Score":"78.21","Prediction Score":"79.18","Top 1 Accuracy":"77.10"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-image-classification-on-ovic-1","task":"Open Vocabulary Image Classification","dataset":"OVIC Datasets (Wiki-H)","model":"SigLIP B/16 + PrefixedIter Decoder (FT2)","rank_in_archive_order":3,"of":3,"metrics":{"Prediction Score (mean of 3)":"72.03"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-image-classification-on-ovic-3","task":"Open Vocabulary Image Classification","dataset":"OVIC Datasets (Wiki-L)","model":"DFN-5B H/14-378 + PrefixedIter Decoder (FT2)","rank_in_archive_order":1,"of":2,"metrics":{"Prediction Score (mean of 3)":"74.88"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-image-classification-on-ovic-3","task":"Open Vocabulary Image Classification","dataset":"OVIC Datasets (Wiki-L)","model":"DFN-5B H/14-378 + PrefixedIter Decoder (FT0)","rank_in_archive_order":2,"of":2,"metrics":{"Prediction Score (mean of 3)":"74.48"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-image-classification-on-ovic","task":"Open Vocabulary Image Classification","dataset":"OVIC Datasets (World-H)","model":"SigLIP SO/14 + PrefixedIter Decoder (FT2)","rank_in_archive_order":1,"of":3,"metrics":{"Prediction Score (mean of 3)":"87.49"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-image-classification-on-ovic","task":"Open Vocabulary Image Classification","dataset":"OVIC Datasets (World-H)","model":"DFN-5B H/14-378 + PrefixedIter Decoder (FT2)","rank_in_archive_order":2,"of":3,"metrics":{"Overall Score":"87.13","Prediction Score":"87.94","Prediction Score (mean of 3)":"87.08","Top 1 Accuracy":"86.77"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-image-classification-on-ovic","task":"Open Vocabulary Image Classification","dataset":"OVIC Datasets (World-H)","model":"DFN-5B H/14-378 + PrefixedIter Decoder (FT0)","rank_in_archive_order":3,"of":3,"metrics":{"Overall Score":"87.90","Prediction Score":"88.27","Prediction Score (mean of 3)":"86.41","Top 1 Accuracy":"86.95"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}