{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hoiclip-efficient-knowledge-transfer-for-hoi","title":"HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Models","arxiv_id":"2303.15786","date":"2023-03-28","proceeding":"CVPR 2023 1","authors":["Shan Ning","Longtian Qiu","Yongfei Liu","Xuming He"],"abstract":"Human-Object Interaction (HOI) detection aims to localize human-object pairs and recognize their interactions. Recently, Contrastive Language-Image Pre-training (CLIP) has shown great potential in providing interaction prior for HOI detectors via knowledge distillation. However, such approaches often rely on large-scale training data and suffer from inferior performance under few/zero-shot scenarios. In this paper, we propose a novel HOI detection framework that efficiently extracts prior knowledge from CLIP and achieves better generalization. In detail, we first introduce a novel interaction decoder to extract informative regions in the visual feature map of CLIP via a cross-attention mechanism, which is then fused with the detection backbone by a knowledge integration block for more accurate human-object pair detection. In addition, prior knowledge in CLIP text encoder is leveraged to generate a classifier by embedding HOI descriptions. To distinguish fine-grained interactions, we build a verb classifier from training data via visual semantic arithmetic and a lightweight verb representation adapter. Furthermore, we propose a training-free enhancement to exploit global HOI predictions from CLIP. Extensive experiments demonstrate that our method outperforms the state of the art by a large margin on various settings, e.g. +4.04 mAP on HICO-Det. The source code is available in https://github.com/Artanic30/HOICLIP.","url_abs":"https://arxiv.org/abs/2303.15786v3","url_pdf":"https://arxiv.org/pdf/2303.15786v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hoiclip-efficient-knowledge-transfer-for-hoi","repo_url":"https://github.com/artanic30/hoiclip","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"human-object-interaction-detection","task_name":"Human-Object Interaction Detection"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"object","task_name":"Object"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/human-object-interaction-detection-on-hico","task":"Human-Object Interaction Detection","dataset":"HICO-DET","model":"HOICLIP","rank_in_archive_order":16,"of":55,"metrics":{"mAP":"34.69"},"uses_additional_data":false},{"leaderboard":"/sota/human-object-interaction-detection-on-v-coco","task":"Human-Object Interaction Detection","dataset":"V-COCO","model":"HOICLIP","rank_in_archive_order":8,"of":34,"metrics":{"AP(S1)":"63.50","AP(S2)":"64.81"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2303.15786","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}