{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-21625","title":"Flat-Pack Bench: Evaluating Spatio-Temporal Understanding in Large Vision-Language Models through Furniture Assembly","arxiv_id":"2605.21625","date":"2026-05-20","proceeding":null,"authors":["Aditya Chetan","Eric Cai","Peeyush Kushwaha","Bharath Raj Nagoor Kani","Utkarsh Mall","Qianqian Wang","Noah Snavely","Bharath Hariharan"],"abstract":"The emergence of Large Vision-Language Models (LVLMs) has significantly advanced video understanding capabilities. However, existing benchmarks focus predominantly on coarse-grained tasks such as action segmentation, classification, captioning, and retrieval. Furthermore, these benchmarks often rely on entities that can be easily identified verbally, like household objects, animals, human subjects, etc., limiting their applicability to complex, in-the-wild video scenarios. But, many applications such as furniture assembly, cooking, etc., require step-by-step fine-grained spatio-temporal understanding of the video, which is not sufficiently evaluated in current benchmarks. To address this gap, we introduce Flat-Pack Bench, a novel benchmark centered on furniture assembly tasks. Our benchmark evaluates LVLMs on nuanced tasks, including temporal ordering of assembly actions, temporal localization of assembly state, understanding part mating, and tracking, using multiple-choice questions paired with visual prompts highlighting relevant parts as references for fine-grained questions. Our experiments reveal that state-of-the-art LVLMs struggle significantly with fine-grained spatio-temporal reasoning, highlighting their limitations in effectively leveraging temporal information from videos, limited tracking ability, and understanding of spatial interactions like physical contact.","url_abs":"https://arxiv.org/abs/2605.21625","url_pdf":"https://arxiv.org/pdf/2605.21625","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.21625"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/justachetan/flat-pack-bench","reach":null}],"summary":{"ran":7,"unverified":3},"by_repo_kind":{"found_in_text":{"samples":10,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e65a02da3772dafb","entry":"hierarchical_bootstrap_video_only","repo":"justachetan/flat-pack-bench","repo_kind":"found_in_text","path":"src/eval/tabulate_results.py","file_url":"https://github.com/justachetan/flat-pack-bench/blob/HEAD/src/eval/tabulate_results.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e65a02da3772dafb"}},{"code_sha256_prefix":"d48ae27bc618f675","entry":"identity_post_process","repo":"justachetan/flat-pack-bench","repo_kind":"found_in_text","path":"src/eval/evaluate.py","file_url":"https://github.com/justachetan/flat-pack-bench/blob/HEAD/src/eval/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d48ae27bc618f675"}},{"code_sha256_prefix":"c5546ca0fdd844f1","entry":"infer_mime","repo":"justachetan/flat-pack-bench","repo_kind":"found_in_text","path":"src/tva/render_logs.py","file_url":"https://github.com/justachetan/flat-pack-bench/blob/HEAD/src/tva/render_logs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c5546ca0fdd844f1"}},{"code_sha256_prefix":"e4daf562577d6d35","entry":"load_events","repo":"justachetan/flat-pack-bench","repo_kind":"found_in_text","path":"src/tva/render_logs.py","file_url":"https://github.com/justachetan/flat-pack-bench/blob/HEAD/src/tva/render_logs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e4daf562577d6d35"}},{"code_sha256_prefix":"a35daeb638210cc1","entry":"post_process_response","repo":"justachetan/flat-pack-bench","repo_kind":"found_in_text","path":"src/eval/models/aria.py","file_url":"https://github.com/justachetan/flat-pack-bench/blob/HEAD/src/eval/models/aria.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a35daeb638210cc1"}},{"code_sha256_prefix":"e58b8265b30b9e1b","entry":"run_all_questions","repo":"justachetan/flat-pack-bench","repo_kind":"found_in_text","path":"src/tva/agent.py","file_url":"https://github.com/justachetan/flat-pack-bench/blob/HEAD/src/tva/agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e58b8265b30b9e1b"}},{"code_sha256_prefix":"1e3dbbbcbeeedd85","entry":"slugify","repo":"justachetan/flat-pack-bench","repo_kind":"found_in_text","path":"src/tva/render_logs.py","file_url":"https://github.com/justachetan/flat-pack-bench/blob/HEAD/src/tva/render_logs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1e3dbbbcbeeedd85"}},{"code_sha256_prefix":"5179e1f01539c74e","entry":"evaluate","repo":"justachetan/flat-pack-bench","repo_kind":"found_in_text","path":"src/tva/tva_judge.py","file_url":"https://github.com/justachetan/flat-pack-bench/blob/HEAD/src/tva/tva_judge.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5179e1f01539c74e"}},{"code_sha256_prefix":"d1874206317a9fbb","entry":"read_video_pyav","repo":"justachetan/flat-pack-bench","repo_kind":"found_in_text","path":"src/eval/models/base_model.py","file_url":"https://github.com/justachetan/flat-pack-bench/blob/HEAD/src/eval/models/base_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d1874206317a9fbb"}},{"code_sha256_prefix":"2fe0bfd180ec4a21","entry":"resolve_response_schema","repo":"justachetan/flat-pack-bench","repo_kind":"found_in_text","path":"src/eval/models/gemini.py","file_url":"https://github.com/justachetan/flat-pack-bench/blob/HEAD/src/eval/models/gemini.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2fe0bfd180ec4a21"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}