{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/delving-into-multimodal-prompting-for-fine","title":"Delving into Multimodal Prompting for Fine-grained Visual Classification","arxiv_id":"2309.08912","date":"2023-09-16","proceeding":null,"authors":["Xin Jiang","Hao Tang","Junyao Gao","Xiaoyu Du","Shengfeng He","Zechao Li"],"abstract":"Fine-grained visual classification (FGVC) involves categorizing fine subdivisions within a broader category, which poses challenges due to subtle inter-class discrepancies and large intra-class variations. However, prevailing approaches primarily focus on uni-modal visual concepts. Recent advancements in pre-trained vision-language models have demonstrated remarkable performance in various high-level vision tasks, yet the applicability of such models to FGVC tasks remains uncertain. In this paper, we aim to fully exploit the capabilities of cross-modal description to tackle FGVC tasks and propose a novel multimodal prompting solution, denoted as MP-FGVC, based on the contrastive language-image pertaining (CLIP) model. Our MP-FGVC comprises a multimodal prompts scheme and a multimodal adaptation scheme. The former includes Subcategory-specific Vision Prompt (SsVP) and Discrepancy-aware Text Prompt (DaTP), which explicitly highlights the subcategory-specific discrepancies from the perspectives of both vision and language. The latter aligns the vision and text prompting elements in a common semantic space, facilitating cross-modal collaborative reasoning through a Vision-Language Fusion Module (VLFM) for further improvement on FGVC. Moreover, we tailor a two-stage optimization strategy for MP-FGVC to fully leverage the pre-trained CLIP model and expedite efficient adaptation for FGVC. Extensive experiments conducted on four FGVC datasets demonstrate the effectiveness of our MP-FGVC.","url_abs":"https://arxiv.org/abs/2309.08912v2","url_pdf":"https://arxiv.org/pdf/2309.08912v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"fine-grained-image-classification","task_name":"Fine-Grained Image Classification"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/fine-grained-image-classification-on-nabirds","task":"Fine-Grained Image Classification","dataset":"NABirds","model":"MP-FGVC","rank_in_archive_order":14,"of":30,"metrics":{"Accuracy":"91.0%"},"uses_additional_data":false},{"leaderboard":"/sota/fine-grained-image-classification-on-stanford-1","task":"Fine-Grained Image Classification","dataset":"Stanford Dogs","model":"MP-FGVC","rank_in_archive_order":16,"of":24,"metrics":{"Accuracy":"91.0%"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2309.08912","atlas_url":"https://app.syntology.ai/?focus=2309.08912","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}