{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multimedia-generative-script-learning-for","title":"Multimedia Generative Script Learning for Task Planning","arxiv_id":"2208.12306","date":"2022-08-25","proceeding":null,"authors":["Qingyun Wang","Manling Li","Hou Pong Chan","Lifu Huang","Julia Hockenmaier","Girish Chowdhary","Heng Ji"],"abstract":"Goal-oriented generative script learning aims to generate subsequent steps to reach a particular goal, which is an essential task to assist robots or humans in performing stereotypical activities. An important aspect of this process is the ability to capture historical states visually, which provides detailed information that is not covered by text and will guide subsequent steps. Therefore, we propose a new task, Multimedia Generative Script Learning, to generate subsequent steps by tracking historical states in both text and vision modalities, as well as presenting the first benchmark containing 5,652 tasks and 79,089 multimedia steps. This task is challenging in three aspects: the multimedia challenge of capturing the visual states in images, the induction challenge of performing unseen tasks, and the diversity challenge of covering different information in individual steps. We propose to encode visual state changes through a selective multimedia encoder to address the multimedia challenge, transfer knowledge from previously observed tasks using a retrieval-augmented decoder to overcome the induction challenge, and further present distinct information at each step by optimizing a diversity-oriented contrastive learning objective. We define metrics to evaluate both generation and inductive quality. Experiment results demonstrate that our approach significantly outperforms strong baselines.","url_abs":"https://arxiv.org/abs/2208.12306v3","url_pdf":"https://arxiv.org/pdf/2208.12306v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"multimedia-generative-script-learning-for","repo_url":"https://github.com/EagleW/Multimedia-Generative-Script-Learning-for-Task-Planning","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"descriptive","task_name":"Descriptive"},{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"multimedia-generative-script-learning","task_name":"Multimedia Generative Script Learning"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"task-planning","task_name":"Task Planning"},{"task_slug":"multimodal-generation","task_name":"multimodal generation"}],"methods":[{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"}],"datasets_introduced":[{"slug":"multimedia-goal-oriented-generative-script","name":"Multimedia Goal-oriented Generative Script Learning Dataset","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2208.12306","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2208.12306"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/EagleW/Multimedia-Generative-Script-Learning-for-Task-Planning","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":1,"ran_draft_wrong":1,"ran":1,"unverified":6},"by_repo_kind":{"official":{"samples":9,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f6b944f50d3f15ae","entry":"count_parameters","repo":"EagleW/Multimedia-Generative-Script-Learning-for-Task-Planning","repo_kind":"official","path":"utils_.py","file_url":"https://github.com/EagleW/Multimedia-Generative-Script-Learning-for-Task-Planning/blob/HEAD/utils_.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f6b944f50d3f15ae"}},{"code_sha256_prefix":"ec4448373d72d636","entry":"precook","repo":"EagleW/Multimedia-Generative-Script-Learning-for-Task-Planning","repo_kind":"official","path":"bleu/bleu_scorer.py","file_url":"https://github.com/EagleW/Multimedia-Generative-Script-Learning-for-Task-Planning/blob/HEAD/bleu/bleu_scorer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec4448373d72d636"}},{"code_sha256_prefix":"44f4152cc7eb1ddd","entry":"reduce_dict","repo":"EagleW/Multimedia-Generative-Script-Learning-for-Task-Planning","repo_kind":"official","path":"utils_.py","file_url":"https://github.com/EagleW/Multimedia-Generative-Script-Learning-for-Task-Planning/blob/HEAD/utils_.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"44f4152cc7eb1ddd"}},{"code_sha256_prefix":"23f6c2e30fb38e32","entry":"AttnClsGate","repo":"EagleW/Multimedia-Generative-Script-Learning-for-Task-Planning","repo_kind":"official","path":"modules/utils.py","file_url":"https://github.com/EagleW/Multimedia-Generative-Script-Learning-for-Task-Planning/blob/HEAD/modules/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"23f6c2e30fb38e32"}},{"code_sha256_prefix":"87ffedc75fe54075","entry":"AttnGate","repo":"EagleW/Multimedia-Generative-Script-Learning-for-Task-Planning","repo_kind":"official","path":"modules/utils.py","file_url":"https://github.com/EagleW/Multimedia-Generative-Script-Learning-for-Task-Planning/blob/HEAD/modules/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"87ffedc75fe54075"}},{"code_sha256_prefix":"7586e1d32f7325f7","entry":"cook_refs","repo":"EagleW/Multimedia-Generative-Script-Learning-for-Task-Planning","repo_kind":"official","path":"bleu/bleu_scorer.py","file_url":"https://github.com/EagleW/Multimedia-Generative-Script-Learning-for-Task-Planning/blob/HEAD/bleu/bleu_scorer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7586e1d32f7325f7"}},{"code_sha256_prefix":"7ee8aa480d13ee01","entry":"cook_test","repo":"EagleW/Multimedia-Generative-Script-Learning-for-Task-Planning","repo_kind":"official","path":"bleu/bleu_scorer.py","file_url":"https://github.com/EagleW/Multimedia-Generative-Script-Learning-for-Task-Planning/blob/HEAD/bleu/bleu_scorer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7ee8aa480d13ee01"}},{"code_sha256_prefix":"b739bbac3e47c758","entry":"get_input","repo":"EagleW/Multimedia-Generative-Script-Learning-for-Task-Planning","repo_kind":"official","path":"wikihow_contrastive_data.py","file_url":"https://github.com/EagleW/Multimedia-Generative-Script-Learning-for-Task-Planning/blob/HEAD/wikihow_contrastive_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b739bbac3e47c758"}},{"code_sha256_prefix":"5a21ebce306df07c","entry":"load_state_dict","repo":"EagleW/Multimedia-Generative-Script-Learning-for-Task-Planning","repo_kind":"official","path":"utils_.py","file_url":"https://github.com/EagleW/Multimedia-Generative-Script-Learning-for-Task-Planning/blob/HEAD/utils_.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5a21ebce306df07c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}