{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-semantic-proxies-from-visual-prompts","title":"Learning Semantic Proxies from Visual Prompts for Parameter-Efficient Fine-Tuning in Deep Metric Learning","arxiv_id":"2402.02340","date":"2024-02-04","proceeding":null,"authors":["Li Ren","Chen Chen","Liqiang Wang","Kien Hua"],"abstract":"Deep Metric Learning (DML) has long attracted the attention of the machine learning community as a key objective. Existing solutions concentrate on fine-tuning the pre-trained models on conventional image datasets. As a result of the success of recent pre-trained models trained from larger-scale datasets, it is challenging to adapt the model to the DML tasks in the local data domain while retaining the previously gained knowledge. In this paper, we investigate parameter-efficient methods for fine-tuning the pre-trained model for DML tasks. In particular, we propose a novel and effective framework based on learning Visual Prompts (VPT) in the pre-trained Vision Transformers (ViT). Based on the conventional proxy-based DML paradigm, we augment the proxy by incorporating the semantic information from the input image and the ViT, in which we optimize the visual prompts for each class. We demonstrate that our new approximations with semantic information are superior to representative capabilities, thereby improving metric learning performance. We conduct extensive experiments to demonstrate that our proposed framework is effective and efficient by evaluating popular DML benchmarks. In particular, we demonstrate that our fine-tuning method achieves comparable or even better performance than recent state-of-the-art full fine-tuning works of DML while tuning only a small percentage of total parameters.","url_abs":"https://arxiv.org/abs/2402.02340v2","url_pdf":"https://arxiv.org/pdf/2402.02340v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-semantic-proxies-from-visual-prompts","repo_url":"https://github.com/noahsark/parameterefficient-dml","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-retrieval","task_name":"Image Retrieval"},{"task_slug":"metric-learning","task_name":"Metric Learning"},{"task_slug":"parameter-efficient-fine-tuning","task_name":"parameter-efficient fine-tuning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-retrieval-on-inaturalist","task":"Image Retrieval","dataset":"iNaturalist","model":"EfficientDML-VPTSP-G/512","rank_in_archive_order":2,"of":10,"metrics":{"R@1":"84.5"},"uses_additional_data":false},{"leaderboard":"/sota/metric-learning-on-cars196","task":"Metric Learning","dataset":"CARS196","model":"EfficientDML-VPTSP-G/512","rank_in_archive_order":6,"of":36,"metrics":{"R@1":"91.2"},"uses_additional_data":true},{"leaderboard":"/sota/metric-learning-on-cub-200-2011","task":"Metric Learning","dataset":"CUB-200-2011","model":"EfficientDML-VPTSP-G/512","rank_in_archive_order":2,"of":30,"metrics":{"R@1":"88.5"},"uses_additional_data":true},{"leaderboard":"/sota/metric-learning-on-in-shop-1","task":"Metric Learning","dataset":"In-Shop","model":"EfficientDML-VPTSP-G/512","rank_in_archive_order":10,"of":15,"metrics":{"R@1":"92.1"},"uses_additional_data":true}],"syntology":{"syntology_url":"https://syntology.ai/paper/2402.02340","atlas_url":"https://app.syntology.ai/?focus=2402.02340","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.02340"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/noahsark/parameterefficient-dml","reach":null}],"summary":{"ran":3,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f7ef1a9769aba46a","entry":"AdaptBlock","repo":"noahsark/parameterefficient-dml","repo_kind":"official","path":"architectures/peft_vit.py","file_url":"https://github.com/noahsark/parameterefficient-dml/blob/HEAD/architectures/peft_vit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f7ef1a9769aba46a"}},{"code_sha256_prefix":"4aa93defe46d50e4","entry":"AdapterLayer","repo":"noahsark/parameterefficient-dml","repo_kind":"official","path":"architectures/peft_vit.py","file_url":"https://github.com/noahsark/parameterefficient-dml/blob/HEAD/architectures/peft_vit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4aa93defe46d50e4"}},{"code_sha256_prefix":"60eea9f2574a41e5","entry":"Simple_GRUCell","repo":"noahsark/parameterefficient-dml","repo_kind":"official","path":"architectures/peft_vit.py","file_url":"https://github.com/noahsark/parameterefficient-dml/blob/HEAD/architectures/peft_vit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"60eea9f2574a41e5"}},{"code_sha256_prefix":"4b7aae5784242bac","entry":"Network","repo":"noahsark/parameterefficient-dml","repo_kind":"official","path":"architectures/peft_vit.py","file_url":"https://github.com/noahsark/parameterefficient-dml/blob/HEAD/architectures/peft_vit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4b7aae5784242bac"}},{"code_sha256_prefix":"269d33dde5785561","entry":"init_bert_weights","repo":"noahsark/parameterefficient-dml","repo_kind":"official","path":"architectures/peft_vit.py","file_url":"https://github.com/noahsark/parameterefficient-dml/blob/HEAD/architectures/peft_vit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"269d33dde5785561"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}