{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/zero-shot-everything-sketch-based-image","title":"Zero-Shot Everything Sketch-Based Image Retrieval, and in Explainable Style","arxiv_id":"2303.14348","date":"2023-03-25","proceeding":"CVPR 2023 1","authors":["Fengyin Lin","Mingkang Li","Da Li","Timothy Hospedales","Yi-Zhe Song","Yonggang Qi"],"abstract":"This paper studies the problem of zero-short sketch-based image retrieval (ZS-SBIR), however with two significant differentiators to prior art (i) we tackle all variants (inter-category, intra-category, and cross datasets) of ZS-SBIR with just one network (``everything''), and (ii) we would really like to understand how this sketch-photo matching operates (``explainable''). Our key innovation lies with the realization that such a cross-modal matching problem could be reduced to comparisons of groups of key local patches -- akin to the seasoned ``bag-of-words'' paradigm. Just with this change, we are able to achieve both of the aforementioned goals, with the added benefit of no longer requiring external semantic knowledge. Technically, ours is a transformer-based cross-modal network, with three novel components (i) a self-attention module with a learnable tokenizer to produce visual tokens that correspond to the most informative local regions, (ii) a cross-attention module to compute local correspondences between the visual tokens across two modalities, and finally (iii) a kernel-based relation network to assemble local putative matches and produce an overall similarity metric for a sketch-photo pair. Experiments show ours indeed delivers superior performances across all ZS-SBIR settings. The all important explainable goal is elegantly achieved by visualizing cross-modal token correspondences, and for the first time, via sketch to photo synthesis by universal replacement of all matched photo patches. Code and model are available at \\url{https://github.com/buptLinfy/ZSE-SBIR}.","url_abs":"https://arxiv.org/abs/2303.14348v1","url_pdf":"https://arxiv.org/pdf/2303.14348v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"zero-shot-everything-sketch-based-image","repo_url":"https://github.com/buptlinfy/zse-sbir","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"image-retrieval","task_name":"Image Retrieval"},{"task_slug":"relation-network","task_name":"Relation Network"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"sketch-based-image-retrieval","task_name":"Sketch-Based Image Retrieval"}],"methods":[{"method_slug":"concatenated-skip-connection","method_name":"Concatenated Skip Connection"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2303.14348","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.14348"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/buptlinfy/zse-sbir","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/buptLinfy/ZSE-SBIR","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":10,"ran_draft_wrong":2,"unverified":4},"by_repo_kind":{"official":{"samples":16,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"afe81aa89974190e","entry":"AddAndNorm","repo":"buptLinfy/ZSE-SBIR","repo_kind":"official","path":"model/model.py","file_url":"https://github.com/buptLinfy/ZSE-SBIR/blob/HEAD/model/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"afe81aa89974190e"}},{"code_sha256_prefix":"88b2eaf4a0201422","entry":"Encoder","repo":"buptLinfy/ZSE-SBIR","repo_kind":"official","path":"model/model.py","file_url":"https://github.com/buptLinfy/ZSE-SBIR/blob/HEAD/model/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"88b2eaf4a0201422"}},{"code_sha256_prefix":"bfdf0d1287b4d60a","entry":"Encoder1DBlock","repo":"buptLinfy/ZSE-SBIR","repo_kind":"official","path":"model/model.py","file_url":"https://github.com/buptLinfy/ZSE-SBIR/blob/HEAD/model/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bfdf0d1287b4d60a"}},{"code_sha256_prefix":"6d34c0d54b4cd5cc","entry":"EncoderLayer","repo":"buptLinfy/ZSE-SBIR","repo_kind":"official","path":"model/model.py","file_url":"https://github.com/buptLinfy/ZSE-SBIR/blob/HEAD/model/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6d34c0d54b4cd5cc"}},{"code_sha256_prefix":"9f9b1604d01394b4","entry":"LayerNorm","repo":"buptLinfy/ZSE-SBIR","repo_kind":"official","path":"model/model.py","file_url":"https://github.com/buptLinfy/ZSE-SBIR/blob/HEAD/model/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9f9b1604d01394b4"}},{"code_sha256_prefix":"31fe7ef87068a002","entry":"MultiHeadDotProductAttention","repo":"buptLinfy/ZSE-SBIR","repo_kind":"official","path":"model/model.py","file_url":"https://github.com/buptLinfy/ZSE-SBIR/blob/HEAD/model/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"31fe7ef87068a002"}},{"code_sha256_prefix":"8b6a335cd0939a66","entry":"MultiHeadedAttention","repo":"buptLinfy/ZSE-SBIR","repo_kind":"official","path":"model/model.py","file_url":"https://github.com/buptLinfy/ZSE-SBIR/blob/HEAD/model/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8b6a335cd0939a66"}},{"code_sha256_prefix":"2d97255bca4d10bc","entry":"PositionwiseFeedForward","repo":"buptLinfy/ZSE-SBIR","repo_kind":"official","path":"model/model.py","file_url":"https://github.com/buptLinfy/ZSE-SBIR/blob/HEAD/model/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2d97255bca4d10bc"}},{"code_sha256_prefix":"f25396c41187c618","entry":"Relation_Network","repo":"buptLinfy/ZSE-SBIR","repo_kind":"official","path":"model/model.py","file_url":"https://github.com/buptLinfy/ZSE-SBIR/blob/HEAD/model/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f25396c41187c618"}},{"code_sha256_prefix":"71497b370f236dea","entry":"Scale_Embedding","repo":"buptLinfy/ZSE-SBIR","repo_kind":"official","path":"model/model.py","file_url":"https://github.com/buptLinfy/ZSE-SBIR/blob/HEAD/model/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"71497b370f236dea"}},{"code_sha256_prefix":"3eed1a7a6d48923a","entry":"attention","repo":"buptLinfy/ZSE-SBIR","repo_kind":"official","path":"model/model.py","file_url":"https://github.com/buptLinfy/ZSE-SBIR/blob/HEAD/model/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3eed1a7a6d48923a"}},{"code_sha256_prefix":"a15a8757a1c2f059","entry":"cos_similar","repo":"buptLinfy/ZSE-SBIR","repo_kind":"official","path":"model/model.py","file_url":"https://github.com/buptLinfy/ZSE-SBIR/blob/HEAD/model/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a15a8757a1c2f059"}},{"code_sha256_prefix":"f1822a4207bad8bd","entry":"Cross_Attention","repo":"buptLinfy/ZSE-SBIR","repo_kind":"official","path":"model/model.py","file_url":"https://github.com/buptLinfy/ZSE-SBIR/blob/HEAD/model/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f1822a4207bad8bd"}},{"code_sha256_prefix":"36976ee282990c6e","entry":"Model","repo":"buptLinfy/ZSE-SBIR","repo_kind":"official","path":"model/model.py","file_url":"https://github.com/buptLinfy/ZSE-SBIR/blob/HEAD/model/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"36976ee282990c6e"}},{"code_sha256_prefix":"83ad54e98125113e","entry":"Self_Attention","repo":"buptLinfy/ZSE-SBIR","repo_kind":"official","path":"model/model.py","file_url":"https://github.com/buptLinfy/ZSE-SBIR/blob/HEAD/model/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"83ad54e98125113e"}},{"code_sha256_prefix":"5077295b3497cdc4","entry":"ViTPatch","repo":"buptLinfy/ZSE-SBIR","repo_kind":"official","path":"model/model.py","file_url":"https://github.com/buptLinfy/ZSE-SBIR/blob/HEAD/model/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5077295b3497cdc4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}