{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2509-02807","title":"PixFoundation 2.0: Do Video Multi-Modal LLMs Use Motion in Visual Grounding?","arxiv_id":"2509.02807","date":"2025-09-02","proceeding":"NeurIPS","authors":["Mennatullah Siam"],"abstract":"Multi-modal large language models (MLLMs) have shown impressive generalization across tasks using images and text modalities. While their extension to video has enabled tasks such as video question answering and video captioning, their pixel-level visual grounding abilities are less studied. In this work, we raise the pertinent question of whether motion is used in pixel-level visual grounding and whether video MLLMs can segment objects based on natural language expressions describing their motion patterns. We identify the shortcomings in the current benchmarks, where we show that a single frame can often suffice for capturing the motion referring expression without any temporal reasoning. To address this, we introduce four motion-centric probing techniques, particularly designed for the visual grounding task, to study video MLLMs' ability to identify true motion from a fake one and their ability to grasp the motion order. Consequently, we provide a motion-centric benchmark, MoCentric-Bench. It ensures that video MLLMs are evaluated towards leveraging the interaction between motion and language rather than being dominated by static appearance cues emphasized in existing visual grounding datasets. We further establish strong single-image baselines that are on par with or outperform prior methods. Finally, we explore simple motion-centric adaptation techniques that provide state-of-the-art performance on our MoCentric-Bench. Our motion-centric benchmark, evaluation and findings challenge future models to improve dense spatiotemporal grounding and pixel-level understanding within videos. Code and datasets will be made publicly available at https://github.com/MSiam/PixFoundation-2.0.git.","url_abs":"https://arxiv.org/abs/2509.02807","url_pdf":"https://arxiv.org/pdf/2509.02807","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2509.02807","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2509.02807"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/MSiam/PixFoundation-2.0","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":12},"by_repo_kind":{"found_in_text":{"samples":12,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3e9c9d59fae40671","entry":"async_func","repo":"MSiam/PixFoundation-2.0","repo_kind":"found_in_text","path":"infer/infer_sa2va.py","file_url":"https://github.com/MSiam/PixFoundation-2.0/blob/HEAD/infer/infer_sa2va.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e9c9d59fae40671"}},{"code_sha256_prefix":"5bc1d4fdb144f905","entry":"bounding_box","repo":"MSiam/PixFoundation-2.0","repo_kind":"found_in_text","path":"datasets_/mevis_variants/mevis_dataset.py","file_url":"https://github.com/MSiam/PixFoundation-2.0/blob/HEAD/datasets_/mevis_variants/mevis_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5bc1d4fdb144f905"}},{"code_sha256_prefix":"31ef988499e81ea5","entry":"db_eval_boundary","repo":"MSiam/PixFoundation-2.0","repo_kind":"found_in_text","path":"eval/metrics.py","file_url":"https://github.com/MSiam/PixFoundation-2.0/blob/HEAD/eval/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"31ef988499e81ea5"}},{"code_sha256_prefix":"9d36806ccbb1f7d0","entry":"db_eval_iou","repo":"MSiam/PixFoundation-2.0","repo_kind":"found_in_text","path":"eval/metrics.py","file_url":"https://github.com/MSiam/PixFoundation-2.0/blob/HEAD/eval/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9d36806ccbb1f7d0"}},{"code_sha256_prefix":"ae6c2bb194221137","entry":"encode_image_to_base64","repo":"MSiam/PixFoundation-2.0","repo_kind":"found_in_text","path":"infer/infer_gpt.py","file_url":"https://github.com/MSiam/PixFoundation-2.0/blob/HEAD/infer/infer_gpt.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ae6c2bb194221137"}},{"code_sha256_prefix":"0b47c87d65457a57","entry":"f_measure","repo":"MSiam/PixFoundation-2.0","repo_kind":"found_in_text","path":"eval/metrics.py","file_url":"https://github.com/MSiam/PixFoundation-2.0/blob/HEAD/eval/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0b47c87d65457a57"}},{"code_sha256_prefix":"a21a61e41d986b6a","entry":"filter_images_with_only_crowd_annotations","repo":"MSiam/PixFoundation-2.0","repo_kind":"found_in_text","path":"datasets_/mevis_variants/mevis_build.py","file_url":"https://github.com/MSiam/PixFoundation-2.0/blob/HEAD/datasets_/mevis_variants/mevis_build.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a21a61e41d986b6a"}},{"code_sha256_prefix":"e5fcd7e40a0b8a73","entry":"give_options","repo":"MSiam/PixFoundation-2.0","repo_kind":"found_in_text","path":"infer/infer_rga.py","file_url":"https://github.com/MSiam/PixFoundation-2.0/blob/HEAD/infer/infer_rga.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e5fcd7e40a0b8a73"}},{"code_sha256_prefix":"a52ef2a02d4a77a1","entry":"harmonic_mean","repo":"MSiam/PixFoundation-2.0","repo_kind":"found_in_text","path":"eval/eval_mevis_variants.py","file_url":"https://github.com/MSiam/PixFoundation-2.0/blob/HEAD/eval/eval_mevis_variants.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a52ef2a02d4a77a1"}},{"code_sha256_prefix":"6f845c821f670c91","entry":"merge","repo":"MSiam/PixFoundation-2.0","repo_kind":"found_in_text","path":"infer/infer_sa2va.py","file_url":"https://github.com/MSiam/PixFoundation-2.0/blob/HEAD/infer/infer_sa2va.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6f845c821f670c91"}},{"code_sha256_prefix":"1ff5615c24ab4c88","entry":"process","repo":"MSiam/PixFoundation-2.0","repo_kind":"found_in_text","path":"infer/infer_rga.py","file_url":"https://github.com/MSiam/PixFoundation-2.0/blob/HEAD/infer/infer_rga.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1ff5615c24ab4c88"}},{"code_sha256_prefix":"6ca5450169996a48","entry":"retrieve_closest_timestamp","repo":"MSiam/PixFoundation-2.0","repo_kind":"found_in_text","path":"infer/infer_vanilla_mllm_baselines.py","file_url":"https://github.com/MSiam/PixFoundation-2.0/blob/HEAD/infer/infer_vanilla_mllm_baselines.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6ca5450169996a48"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CV","source":"arxiv_api"},"syntology_extracted_results":null}