{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/let-all-be-whitened-multi-teacher","title":"Let All be Whitened: Multi-teacher Distillation for Efficient Visual Retrieval","arxiv_id":"2312.09716","date":"2023-12-15","proceeding":null,"authors":["Zhe Ma","Jianfeng Dong","Shouling Ji","Zhenguang Liu","Xuhong Zhang","Zonghui Wang","Sifeng He","Feng Qian","Xiaobo Zhang","Lei Yang"],"abstract":"Visual retrieval aims to search for the most relevant visual items, e.g., images and videos, from a candidate gallery with a given query item. Accuracy and efficiency are two competing objectives in retrieval tasks. Instead of crafting a new method pursuing further improvement on accuracy, in this paper we propose a multi-teacher distillation framework Whiten-MTD, which is able to transfer knowledge from off-the-shelf pre-trained retrieval models to a lightweight student model for efficient visual retrieval. Furthermore, we discover that the similarities obtained by different retrieval models are diversified and incommensurable, which makes it challenging to jointly distill knowledge from multiple models. Therefore, we propose to whiten the output of teacher models before fusion, which enables effective multi-teacher distillation for retrieval models. Whiten-MTD is conceptually simple and practically effective. Extensive experiments on two landmark image retrieval datasets and one video retrieval dataset demonstrate the effectiveness of our proposed method, and its good balance of retrieval performance and efficiency. Our source code is released at https://github.com/Maryeon/whiten_mtd.","url_abs":"https://arxiv.org/abs/2312.09716v1","url_pdf":"https://arxiv.org/pdf/2312.09716v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"let-all-be-whitened-multi-teacher","repo_url":"https://github.com/maryeon/whiten_mtd","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"all","task_name":"All"},{"task_slug":"image-retrieval","task_name":"Image Retrieval"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.09716","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.09716"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/maryeon/whiten_mtd","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":2,"ran":1,"unverified":4},"by_repo_kind":{"official":{"samples":7,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2a80220dabcb742a","entry":"conv1x1","repo":"maryeon/whiten_mtd","repo_kind":"official","path":"models/resnet.py","file_url":"https://github.com/maryeon/whiten_mtd/blob/HEAD/models/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2a80220dabcb742a"}},{"code_sha256_prefix":"600ff2c45e0de056","entry":"conv3x3","repo":"maryeon/whiten_mtd","repo_kind":"official","path":"models/resnet.py","file_url":"https://github.com/maryeon/whiten_mtd/blob/HEAD/models/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"600ff2c45e0de056"}},{"code_sha256_prefix":"bfa2f8e8174388e2","entry":"multiscale_feature","repo":"maryeon/whiten_mtd","repo_kind":"official","path":"oxford_paris_eval.py","file_url":"https://github.com/maryeon/whiten_mtd/blob/HEAD/oxford_paris_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bfa2f8e8174388e2"}},{"code_sha256_prefix":"0ecc3a2da5f14cd5","entry":"resnet101_ap_gem","repo":"maryeon/whiten_mtd","repo_kind":"official","path":"models/teachers.py","file_url":"https://github.com/maryeon/whiten_mtd/blob/HEAD/models/teachers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0ecc3a2da5f14cd5"}},{"code_sha256_prefix":"7141d634226594fb","entry":"resnet101_gem","repo":"maryeon/whiten_mtd","repo_kind":"official","path":"models/teachers.py","file_url":"https://github.com/maryeon/whiten_mtd/blob/HEAD/models/teachers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7141d634226594fb"}},{"code_sha256_prefix":"3b19e8957a27e58d","entry":"resnet101_solar","repo":"maryeon/whiten_mtd","repo_kind":"official","path":"models/teachers.py","file_url":"https://github.com/maryeon/whiten_mtd/blob/HEAD/models/teachers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3b19e8957a27e58d"}},{"code_sha256_prefix":"3163964c502a4162","entry":"resnet18","repo":"maryeon/whiten_mtd","repo_kind":"official","path":"models/resnet.py","file_url":"https://github.com/maryeon/whiten_mtd/blob/HEAD/models/resnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3163964c502a4162"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}