{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/aligning-step-by-step-instructional-diagrams","title":"Aligning Step-by-Step Instructional Diagrams to Video Demonstrations","arxiv_id":"2303.13800","date":"2023-03-24","proceeding":"CVPR 2023 1","authors":["Jiahao Zhang","Anoop Cherian","Yanbin Liu","Yizhak Ben-Shabat","Cristian Rodriguez","Stephen Gould"],"abstract":"Multimodal alignment facilitates the retrieval of instances from one modality when queried using another. In this paper, we consider a novel setting where such an alignment is between (i) instruction steps that are depicted as assembly diagrams (commonly seen in Ikea assembly manuals) and (ii) video segments from in-the-wild videos; these videos comprising an enactment of the assembly actions in the real world. To learn this alignment, we introduce a novel supervised contrastive learning method that learns to align videos with the subtle details in the assembly diagrams, guided by a set of novel losses. To study this problem and demonstrate the effectiveness of our method, we introduce a novel dataset: IAW for Ikea assembly in the wild consisting of 183 hours of videos from diverse furniture assembly collections and nearly 8,300 illustrations from their associated instruction manuals and annotated for their ground truth alignments. We define two tasks on this dataset: First, nearest neighbor retrieval between video segments and illustrations, and, second, alignment of instruction steps and the segments for each video. Extensive experiments on IAW demonstrate superior performances of our approach against alternatives.","url_abs":"https://arxiv.org/abs/2303.13800v5","url_pdf":"https://arxiv.org/pdf/2303.13800v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"aligning-step-by-step-instructional-diagrams","repo_url":"https://github.com/DavidZhang73/AssemblyVideoManualAlignment","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"image-retrieval","task_name":"Image Retrieval"},{"task_slug":"video-alignment","task_name":"Video Alignment"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"}],"methods":[{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"}],"datasets_introduced":[{"slug":"ikea-assembly-in-the-wild-dataset","name":"IAW Dataset","full_name":"Ikea Assembly In The Wild Dataset"}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2303.13800","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.13800"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/DavidZhang73/AssemblyVideoManualAlignment","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":2},"by_repo_kind":{"official":{"samples":2,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"28ba364f804e5341","entry":"make_resnet50","repo":"DavidZhang73/AssemblyVideoManualAlignment","repo_kind":"official","path":"src/models/resnet.py","file_url":"https://github.com/DavidZhang73/AssemblyVideoManualAlignment/blob/HEAD/src/models/resnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"28ba364f804e5341"}},{"code_sha256_prefix":"8564c1ce2632bd6e","entry":"sinkhorn","repo":"DavidZhang73/AssemblyVideoManualAlignment","repo_kind":"official","path":"src/models/optimal_transport.py","file_url":"https://github.com/DavidZhang73/AssemblyVideoManualAlignment/blob/HEAD/src/models/optimal_transport.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8564c1ce2632bd6e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}