{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/interactive-mobile-app-navigation-with","title":"A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility","arxiv_id":"2202.02312","date":"2022-02-04","proceeding":null,"authors":["Andrea Burns","Deniz Arsan","Sanjna Agrawal","Ranjitha Kumar","Kate Saenko","Bryan A. Plummer"],"abstract":"Vision-language navigation (VLN), in which an agent follows language instruction in a visual environment, has been studied under the premise that the input command is fully feasible in the environment. Yet in practice, a request may not be possible due to language ambiguity or environment changes. To study VLN with unknown command feasibility, we introduce a new dataset Mobile app Tasks with Iterative Feedback (MoTIF), where the goal is to complete a natural language command in a mobile app. Mobile apps provide a scalable domain to study real downstream uses of VLN methods. Moreover, mobile app commands provide instruction for interactive navigation, as they result in action sequences with state changes via clicking, typing, or swiping. MoTIF is the first to include feasibility annotations, containing both binary feasibility labels and fine-grained labels for why tasks are unsatisfiable. We further collect follow-up questions for ambiguous queries to enable research on task uncertainty resolution. Equipped with our dataset, we propose the new problem of feasibility prediction, in which a natural language instruction and multimodal app environment are used to predict command feasibility. MoTIF provides a more realistic app dataset as it contains many diverse environments, high-level goals, and longer action sequences than prior work. We evaluate interactive VLN methods using MoTIF, quantify the generalization ability of current approaches to new app environments, and measure the effect of task feasibility on navigation performance.","url_abs":"https://arxiv.org/abs/2202.02312v3","url_pdf":"https://arxiv.org/pdf/2202.02312v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"interactive-mobile-app-navigation-with","repo_url":"https://github.com/aburns4/MoTIF","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"common-sense-reasoning","task_name":"Common Sense Reasoning"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"vision-language-navigation","task_name":"Vision-Language Navigation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2202.02312","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2202.02312"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aburns4/MoTIF","reach":null}],"summary":{"ran_draft_wrong":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"976126e787d6441b","entry":"load_traces","repo":"aburns4/MoTIF","repo_kind":"official","path":"task_automation/seq2act/motif_data_generation/all_in_one_motif_preprocess.py","file_url":"https://github.com/aburns4/MoTIF/blob/HEAD/task_automation/seq2act/motif_data_generation/all_in_one_motif_preprocess.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"976126e787d6441b"}},{"code_sha256_prefix":"2b2dd12878867ae0","entry":"remove_dups","repo":"aburns4/MoTIF","repo_kind":"official","path":"task_automation/seq2act/motif_data_generation/all_in_one_motif_preprocess.py","file_url":"https://github.com/aburns4/MoTIF/blob/HEAD/task_automation/seq2act/motif_data_generation/all_in_one_motif_preprocess.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2b2dd12878867ae0"}},{"code_sha256_prefix":"2e0dcf09857da7c8","entry":"ui_obj_to_str","repo":"aburns4/MoTIF","repo_kind":"official","path":"task_automation/seq2act/motif_data_generation/all_in_one_motif_preprocess.py","file_url":"https://github.com/aburns4/MoTIF/blob/HEAD/task_automation/seq2act/motif_data_generation/all_in_one_motif_preprocess.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2e0dcf09857da7c8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}