{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/asking-multimodal-clarifying-questions-in","title":"Asking Multimodal Clarifying Questions in Mixed-Initiative Conversational Search","arxiv_id":"2402.07742","date":"2024-02-12","proceeding":null,"authors":["Yifei Yuan","Clemencia Siro","Mohammad Aliannejadi","Maarten de Rijke","Wai Lam"],"abstract":"In mixed-initiative conversational search systems, clarifying questions are used to help users who struggle to express their intentions in a single query. These questions aim to uncover user's information needs and resolve query ambiguities. We hypothesize that in scenarios where multimodal information is pertinent, the clarification process can be improved by using non-textual information. Therefore, we propose to add images to clarifying questions and formulate the novel task of asking multimodal clarifying questions in open-domain, mixed-initiative conversational search systems. To facilitate research into this task, we collect a dataset named Melon that contains over 4k multimodal clarifying questions, enriched with over 14k images. We also propose a multimodal query clarification model named Marto and adopt a prompt-based, generative fine-tuning strategy to perform the training of different stages with different prompts. Several analyses are conducted to understand the importance of multimodal contents during the query clarification phase. Experimental results indicate that the addition of images leads to significant improvements of up to 90% in retrieval performance when selecting the relevant images. Extensive analyses are also performed to show the superiority of Marto compared with discriminative baselines in terms of effectiveness and efficiency.","url_abs":"https://arxiv.org/abs/2402.07742v1","url_pdf":"https://arxiv.org/pdf/2402.07742v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"asking-multimodal-clarifying-questions-in","repo_url":"https://github.com/yfyuan01/mqc","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"4k","task_name":"4k"},{"task_slug":"conversational-search","task_name":"Conversational Search"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2402.07742","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.07742"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yfyuan01/mqc","reach":{"status":"ok"}}],"summary":{"ran":7,"unverified":1},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"68e41c28a6dfe81d","entry":"collate_fn","repo":"yfyuan01/mqc","repo_kind":"official","path":"VL-T5/feature_extraction/refcocog_mattnet.py","file_url":"https://github.com/yfyuan01/mqc/blob/HEAD/VL-T5/feature_extraction/refcocog_mattnet.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"68e41c28a6dfe81d"}},{"code_sha256_prefix":"d35ee4a3f75ba6dd","entry":"data_split","repo":"yfyuan01/mqc","repo_kind":"official","path":"facet_data/train_split.py","file_url":"https://github.com/yfyuan01/mqc/blob/HEAD/facet_data/train_split.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d35ee4a3f75ba6dd"}},{"code_sha256_prefix":"e21bb85b8e1b4836","entry":"do_nms","repo":"yfyuan01/mqc","repo_kind":"official","path":"VL-T5/VL-T5/inference/modeling_frcnn.py","file_url":"https://github.com/yfyuan01/mqc/blob/HEAD/VL-T5/VL-T5/inference/modeling_frcnn.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e21bb85b8e1b4836"}},{"code_sha256_prefix":"16c0e181ff805fb7","entry":"norm_box","repo":"yfyuan01/mqc","repo_kind":"official","path":"VL-T5/VL-T5/inference/modeling_frcnn.py","file_url":"https://github.com/yfyuan01/mqc/blob/HEAD/VL-T5/VL-T5/inference/modeling_frcnn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"16c0e181ff805fb7"}},{"code_sha256_prefix":"54a2fff601881b32","entry":"pad_list_tensors","repo":"yfyuan01/mqc","repo_kind":"official","path":"VL-T5/VL-T5/inference/modeling_frcnn.py","file_url":"https://github.com/yfyuan01/mqc/blob/HEAD/VL-T5/VL-T5/inference/modeling_frcnn.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"54a2fff601881b32"}},{"code_sha256_prefix":"b2840eb2211dadb4","entry":"subbatch","repo":"yfyuan01/mqc","repo_kind":"official","path":"cedr/cedr/modeling_util.py","file_url":"https://github.com/yfyuan01/mqc/blob/HEAD/cedr/cedr/modeling_util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b2840eb2211dadb4"}},{"code_sha256_prefix":"bb05ea380c1dec92","entry":"un_subbatch","repo":"yfyuan01/mqc","repo_kind":"official","path":"cedr/cedr/modeling_util.py","file_url":"https://github.com/yfyuan01/mqc/blob/HEAD/cedr/cedr/modeling_util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bb05ea380c1dec92"}},{"code_sha256_prefix":"f7845afac24bc7a7","entry":"stem_tokenize","repo":"yfyuan01/mqc","repo_kind":"official","path":"first_phase_retrieval/bm25.py","file_url":"https://github.com/yfyuan01/mqc/blob/HEAD/first_phase_retrieval/bm25.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f7845afac24bc7a7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}