{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pite-pixel-temporal-alignment-for-large-video","title":"PiTe: Pixel-Temporal Alignment for Large Video-Language Model","arxiv_id":"2409.07239","date":"2024-09-11","proceeding":null,"authors":["Yang Liu","Pengxiang Ding","Siteng Huang","Min Zhang","Han Zhao","Donglin Wang"],"abstract":"Fueled by the Large Language Models (LLMs) wave, Large Visual-Language Models (LVLMs) have emerged as a pivotal advancement, bridging the gap between image and text. However, video making it challenging for LVLMs to perform adequately due to the complexity of the relationship between language and spatial-temporal data structure. Recent Large Video-Language Models (LVidLMs) align feature of static visual data like image into latent space of language feature, by general multi-modal tasks to leverage abilities of LLMs sufficiently. In this paper, we explore fine-grained alignment approach via object trajectory for different modalities across both spatial and temporal dimensions simultaneously. Thus, we propose a novel LVidLM by trajectory-guided Pixel-Temporal Alignment, dubbed PiTe, that exhibits promising applicable model property. To achieve fine-grained video-language alignment, we curate a multi-modal pre-training dataset PiTe-143k, the dataset provision of moving trajectories in pixel level for all individual objects, that appear and mention in the video and caption both, by our automatic annotation pipeline. Meanwhile, PiTe demonstrates astounding capabilities on myriad video-related multi-modal tasks through beat the state-of-the-art methods by a large margin.","url_abs":"https://arxiv.org/abs/2409.07239v1","url_pdf":"https://arxiv.org/pdf/2409.07239v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pite-pixel-temporal-alignment-for-large-video","repo_url":"https://github.com/yliu-cs/pite","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2409.07239","atlas_url":"https://app.syntology.ai/?focus=2409.07239","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.07239"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yliu-cs/pite","reach":null}],"summary":{"ran":3,"unverified":3},"by_repo_kind":{"official":{"samples":6,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"be40a76c20e3f8ec","entry":"PiTeCausalLMOutputWithPast","repo":"yliu-cs/pite","repo_kind":"official","path":"pite.py","file_url":"https://github.com/yliu-cs/pite/blob/HEAD/pite.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"be40a76c20e3f8ec"}},{"code_sha256_prefix":"e18ba9a061b670cf","entry":"PiTeConfig","repo":"yliu-cs/pite","repo_kind":"official","path":"pite.py","file_url":"https://github.com/yliu-cs/pite/blob/HEAD/pite.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e18ba9a061b670cf"}},{"code_sha256_prefix":"d0e020640fd9f259","entry":"PiTeLlamaModel","repo":"yliu-cs/pite","repo_kind":"official","path":"pite.py","file_url":"https://github.com/yliu-cs/pite/blob/HEAD/pite.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d0e020640fd9f259"}},{"code_sha256_prefix":"7908d013a2872a99","entry":"PiTeLlamaForCausalLM","repo":"yliu-cs/pite","repo_kind":"official","path":"pite.py","file_url":"https://github.com/yliu-cs/pite/blob/HEAD/pite.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7908d013a2872a99"}},{"code_sha256_prefix":"7819b34c6b48a64d","entry":"PiTeMetaForCausalLM","repo":"yliu-cs/pite","repo_kind":"official","path":"pite.py","file_url":"https://github.com/yliu-cs/pite/blob/HEAD/pite.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7819b34c6b48a64d"}},{"code_sha256_prefix":"dc6db90271c2e693","entry":"PiTeMetaModel","repo":"yliu-cs/pite","repo_kind":"official","path":"pite.py","file_url":"https://github.com/yliu-cs/pite/blob/HEAD/pite.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dc6db90271c2e693"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}