{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/part2whole-iteratively-enrich-detail-for","title":"Ask&Confirm: Active Detail Enriching for Cross-Modal Retrieval with Partial Query","arxiv_id":"2103.01654","date":"2021-03-02","proceeding":"ICCV 2021 10","authors":["Guanyu Cai","Jun Zhang","Xinyang Jiang","Yifei Gong","Lianghua He","Fufu Yu","Pai Peng","Xiaowei Guo","Feiyue Huang","Xing Sun"],"abstract":"Text-based image retrieval has seen considerable progress in recent years. However, the performance of existing methods suffers in real life since the user is likely to provide an incomplete description of an image, which often leads to results filled with false positives that fit the incomplete description. In this work, we introduce the partial-query problem and extensively analyze its influence on text-based image retrieval. Previous interactive methods tackle the problem by passively receiving users' feedback to supplement the incomplete query iteratively, which is time-consuming and requires heavy user effort. Instead, we propose a novel retrieval framework that conducts the interactive process in an Ask-and-Confirm fashion, where AI actively searches for discriminative details missing in the current query, and users only need to confirm AI's proposal. Specifically, we propose an object-based interaction to make the interactive retrieval more user-friendly and present a reinforcement-learning-based policy to search for discriminative objects. Furthermore, since fully-supervised training is often infeasible due to the difficulty of obtaining human-machine dialog data, we present a weakly-supervised training strategy that needs no human-annotated dialogs other than a text-image dataset. Experiments show that our framework significantly improves the performance of text-based image retrieval. Code is avaiable at https://github.com/CuthbertCai/Ask-Confirm.","url_abs":"https://arxiv.org/abs/2103.01654v2","url_pdf":"https://arxiv.org/pdf/2103.01654v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"part2whole-iteratively-enrich-detail-for","repo_url":"https://github.com/cuthbertcai/ask-confirm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"cross-modal-retrieval","task_name":"Cross-Modal Retrieval"},{"task_slug":"image-retrieval","task_name":"Image Retrieval"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2103.01654","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2103.01654"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/CuthbertCai/Ask-Confirm","reach":null}],"summary":{"ran":3,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"97473241dd1125fa","entry":"ConcatContextEncoder","repo":"CuthbertCai/Ask-Confirm","repo_kind":"official","path":"models/policy_model.py","file_url":"https://github.com/CuthbertCai/Ask-Confirm/blob/HEAD/models/policy_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"97473241dd1125fa"}},{"code_sha256_prefix":"cba9937fc7120054","entry":"ConcatPolicyNet","repo":"CuthbertCai/Ask-Confirm","repo_kind":"official","path":"models/policy_model.py","file_url":"https://github.com/CuthbertCai/Ask-Confirm/blob/HEAD/models/policy_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cba9937fc7120054"}},{"code_sha256_prefix":"58212868f8739f75","entry":"ConcatUpdater","repo":"CuthbertCai/Ask-Confirm","repo_kind":"official","path":"models/policy_model.py","file_url":"https://github.com/CuthbertCai/Ask-Confirm/blob/HEAD/models/policy_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"58212868f8739f75"}},{"code_sha256_prefix":"8a7f1e20f75ba4f3","entry":"Actor","repo":"CuthbertCai/Ask-Confirm","repo_kind":"official","path":"models/policy_model.py","file_url":"https://github.com/CuthbertCai/Ask-Confirm/blob/HEAD/models/policy_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8a7f1e20f75ba4f3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}