{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/openkd-opening-prompt-diversity-for-zero-and","title":"OpenKD: Opening Prompt Diversity for Zero- and Few-shot Keypoint Detection","arxiv_id":"2409.19899","date":"2024-09-30","proceeding":null,"authors":["Changsheng Lu","Zheyuan Liu","Piotr Koniusz"],"abstract":"Exploiting the foundation models (e.g., CLIP) to build a versatile keypoint detector has gained increasing attention. Most existing models accept either the text prompt (e.g., ``the nose of a cat''), or the visual prompt (e.g., support image with keypoint annotations), to detect the corresponding keypoints in query image, thereby, exhibiting either zero-shot or few-shot detection ability. However, the research on taking multimodal prompt is still underexplored, and the prompt diversity in semantics and language is far from opened. For example, how to handle unseen text prompts for novel keypoint detection and the diverse text prompts like ``Can you detect the nose and ears of a cat?'' In this work, we open the prompt diversity from three aspects: modality, semantics (seen v.s. unseen), and language, to enable a more generalized zero- and few-shot keypoint detection (Z-FSKD). We propose a novel OpenKD model which leverages multimodal prototype set to support both visual and textual prompting. Further, to infer the keypoint location of unseen texts, we add the auxiliary keypoints and texts interpolated from visual and textual domains into training, which improves the spatial reasoning of our model and significantly enhances zero-shot novel keypoint detection. We also found large language model (LLM) is a good parser, which achieves over 96% accuracy to parse keypoints from texts. With LLM, OpenKD can handle diverse text prompts. Experimental results show that our method achieves state-of-the-art performance on Z-FSKD and initiates new ways to deal with unseen text and diverse texts. The source code and data are available at https://github.com/AlanLuSun/OpenKD.","url_abs":"https://arxiv.org/abs/2409.19899v1","url_pdf":"https://arxiv.org/pdf/2409.19899v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"openkd-opening-prompt-diversity-for-zero-and","repo_url":"https://github.com/alanlusun/openkd","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"keypoint-detection","task_name":"Keypoint Detection"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"spatial-reasoning","task_name":"Spatial Reasoning"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.19899","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.19899"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/AlanLuSun/OpenKD","reach":{"status":"ok"}}],"summary":{"ran":13,"unverified":1},"by_repo_kind":{"official":{"samples":14,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"5efc70da8ecd43b8","entry":"average_representations","repo":"AlanLuSun/OpenKD","repo_kind":"official","path":"network/models_gridms2.py","file_url":"https://github.com/AlanLuSun/OpenKD/blob/HEAD/network/models_gridms2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5efc70da8ecd43b8"}},{"code_sha256_prefix":"c3f0e5e4591095c6","entry":"compute_openkd_heatmap_loss","repo":"AlanLuSun/OpenKD","repo_kind":"official","path":"core/misc.py","file_url":"https://github.com/AlanLuSun/OpenKD/blob/HEAD/core/misc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c3f0e5e4591095c6"}},{"code_sha256_prefix":"a4318543e6349061","entry":"does_det_success","repo":"AlanLuSun/OpenKD","repo_kind":"official","path":"vis.py","file_url":"https://github.com/AlanLuSun/OpenKD/blob/HEAD/vis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a4318543e6349061"}},{"code_sha256_prefix":"bf44a6653f00c99e","entry":"draw_line","repo":"AlanLuSun/OpenKD","repo_kind":"official","path":"vis.py","file_url":"https://github.com/AlanLuSun/OpenKD/blob/HEAD/vis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bf44a6653f00c99e"}},{"code_sha256_prefix":"aede916dbfc2618a","entry":"even_sampling","repo":"AlanLuSun/OpenKD","repo_kind":"official","path":"network/openkd_model.py","file_url":"https://github.com/AlanLuSun/OpenKD/blob/HEAD/network/openkd_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aede916dbfc2618a"}},{"code_sha256_prefix":"0d9f8c3391250a0d","entry":"extract_texts","repo":"AlanLuSun/OpenKD","repo_kind":"official","path":"vis.py","file_url":"https://github.com/AlanLuSun/OpenKD/blob/HEAD/vis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0d9f8c3391250a0d"}},{"code_sha256_prefix":"cba0ea99fbeca472","entry":"get_align_kps_and_mask","repo":"AlanLuSun/OpenKD","repo_kind":"official","path":"eval_openkd.py","file_url":"https://github.com/AlanLuSun/OpenKD/blob/HEAD/eval_openkd.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cba0ea99fbeca472"}},{"code_sha256_prefix":"5bbf1fcd58116e75","entry":"get_loss_function","repo":"AlanLuSun/OpenKD","repo_kind":"official","path":"core/loss_lw.py","file_url":"https://github.com/AlanLuSun/OpenKD/blob/HEAD/core/loss_lw.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5bbf1fcd58116e75"}},{"code_sha256_prefix":"fefd2cc2cdad885f","entry":"get_optimizer","repo":"AlanLuSun/OpenKD","repo_kind":"official","path":"eval_openkd.py","file_url":"https://github.com/AlanLuSun/OpenKD/blob/HEAD/eval_openkd.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fefd2cc2cdad885f"}},{"code_sha256_prefix":"abb30f0c095fc86e","entry":"heatmap_normalize_layer","repo":"AlanLuSun/OpenKD","repo_kind":"official","path":"network/lw_model.py","file_url":"https://github.com/AlanLuSun/OpenKD/blob/HEAD/network/lw_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"abb30f0c095fc86e"}},{"code_sha256_prefix":"938f923090279c87","entry":"progressive_sampling","repo":"AlanLuSun/OpenKD","repo_kind":"official","path":"network/openkd_model.py","file_url":"https://github.com/AlanLuSun/OpenKD/blob/HEAD/network/openkd_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"938f923090279c87"}},{"code_sha256_prefix":"ce17a5df5b544fcf","entry":"recover_kps","repo":"AlanLuSun/OpenKD","repo_kind":"official","path":"eval_diverse_prompts.py","file_url":"https://github.com/AlanLuSun/OpenKD/blob/HEAD/eval_diverse_prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ce17a5df5b544fcf"}},{"code_sha256_prefix":"17cc8e9898559474","entry":"split_main_aux_heatmaps","repo":"AlanLuSun/OpenKD","repo_kind":"official","path":"core/misc.py","file_url":"https://github.com/AlanLuSun/OpenKD/blob/HEAD/core/misc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"17cc8e9898559474"}},{"code_sha256_prefix":"09dab04abdb1a64c","entry":"get_lw_base_model","repo":"AlanLuSun/OpenKD","repo_kind":"official","path":"network/lw_model.py","file_url":"https://github.com/AlanLuSun/OpenKD/blob/HEAD/network/lw_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"09dab04abdb1a64c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}