{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/can-foundation-models-watch-talk-and-guide","title":"Can Foundation Models Watch, Talk and Guide You Step by Step to Make a Cake?","arxiv_id":"2311.00738","date":"2023-11-01","proceeding":null,"authors":["Yuwei Bao","Keunwoo Peter Yu","Yichi Zhang","Shane Storks","Itamar Bar-Yossef","Alexander De La Iglesia","Megan Su","Xiao Lin Zheng","Joyce Chai"],"abstract":"Despite tremendous advances in AI, it remains a significant challenge to develop interactive task guidance systems that can offer situated, personalized guidance and assist humans in various tasks. These systems need to have a sophisticated understanding of the user as well as the environment, and make timely accurate decisions on when and what to say. To address this issue, we created a new multimodal benchmark dataset, Watch, Talk and Guide (WTaG) based on natural interaction between a human user and a human instructor. We further proposed two tasks: User and Environment Understanding, and Instructor Decision Making. We leveraged several foundation models to study to what extent these models can be quickly adapted to perceptually enabled task guidance. Our quantitative, qualitative, and human evaluation results show that these models can demonstrate fair performances in some cases with no task-specific training, but a fast and reliable adaptation remains a significant challenge. Our benchmark and baselines will provide a stepping stone for future work on situated task guidance.","url_abs":"https://arxiv.org/abs/2311.00738v1","url_pdf":"https://arxiv.org/pdf/2311.00738v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"can-foundation-models-watch-talk-and-guide","repo_url":"https://github.com/sled-group/watch-talk-and-guide","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"decision-making","task_name":"Decision Making"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2311.00738","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.00738"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sled-group/watch-talk-and-guide","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":2,"unverified":2},"by_repo_kind":{"official":{"samples":4,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"508259e833b104dd","entry":"get_prompt","repo":"sled-group/watch-talk-and-guide","repo_kind":"official","path":"src/util.py","file_url":"https://github.com/sled-group/watch-talk-and-guide/blob/HEAD/src/util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"508259e833b104dd"}},{"code_sha256_prefix":"44ac8726a21dbf5e","entry":"parse_text_time","repo":"sled-group/watch-talk-and-guide","repo_kind":"official","path":"src/util.py","file_url":"https://github.com/sled-group/watch-talk-and-guide/blob/HEAD/src/util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"44ac8726a21dbf5e"}},{"code_sha256_prefix":"00b98c2fa15a0c90","entry":"call_BLIP2","repo":"sled-group/watch-talk-and-guide","repo_kind":"official","path":"src/util.py","file_url":"https://github.com/sled-group/watch-talk-and-guide/blob/HEAD/src/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"00b98c2fa15a0c90"}},{"code_sha256_prefix":"d42b1e78850b73d0","entry":"genAndCall","repo":"sled-group/watch-talk-and-guide","repo_kind":"official","path":"src/pipeline.py","file_url":"https://github.com/sled-group/watch-talk-and-guide/blob/HEAD/src/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d42b1e78850b73d0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}