{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/larp-tokenizing-videos-with-a-learned-1","title":"LARP: Tokenizing Videos with a Learned Autoregressive Generative Prior","arxiv_id":"2410.21264","date":"2024-10-28","proceeding":null,"authors":["Hanyu Wang","Saksham Suri","Yixuan Ren","Hao Chen","Abhinav Shrivastava"],"abstract":"We present LARP, a novel video tokenizer designed to overcome limitations in current video tokenization methods for autoregressive (AR) generative models. Unlike traditional patchwise tokenizers that directly encode local visual patches into discrete tokens, LARP introduces a holistic tokenization scheme that gathers information from the visual content using a set of learned holistic queries. This design allows LARP to capture more global and semantic representations, rather than being limited to local patch-level information. Furthermore, it offers flexibility by supporting an arbitrary number of discrete tokens, enabling adaptive and efficient tokenization based on the specific requirements of the task. To align the discrete token space with downstream AR generation tasks, LARP integrates a lightweight AR transformer as a training-time prior model that predicts the next token on its discrete latent space. By incorporating the prior model during training, LARP learns a latent space that is not only optimized for video reconstruction but is also structured in a way that is more conducive to autoregressive generation. Moreover, this process defines a sequential order for the discrete tokens, progressively pushing them toward an optimal configuration during training, ensuring smoother and more accurate AR generation at inference time. Comprehensive experiments demonstrate LARP's strong performance, achieving state-of-the-art FVD on the UCF101 class-conditional video generation benchmark. LARP enhances the compatibility of AR models with videos and opens up the potential to build unified high-fidelity multimodal large language models (MLLMs).","url_abs":"https://arxiv.org/abs/2410.21264v1","url_pdf":"https://arxiv.org/pdf/2410.21264v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"larp-tokenizing-videos-with-a-learned-1","repo_url":"https://github.com/hywang66/LARP","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"video-generation","task_name":"Video Generation"},{"task_slug":"video-reconstruction","task_name":"Video Reconstruction"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-generation-on-ucf-101","task":"Video Generation","dataset":"UCF-101","model":"LARP","rank_in_archive_order":3,"of":48,"metrics":{"FVD16":"57"},"uses_additional_data":false},{"leaderboard":"/sota/video-prediction-on-kinetics-600-12-frames","task":"Video Prediction","dataset":"Kinetics-600 12 frames, 64x64","model":"LARP","rank_in_archive_order":4,"of":16,"metrics":{"Cond":"5","FVD":"5.1","Pred":"11"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.21264","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.21264"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hywang66/LARP","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":1,"ran_honours":1,"ran_draft_wrong":2,"unverified":9},"by_repo_kind":{"listed":{"samples":13,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"87577b3ff9d32712","entry":"drop_path","repo":"hywang66/LARP","repo_kind":"listed","path":"models/larp_ar.py","file_url":"https://github.com/hywang66/LARP/blob/HEAD/models/larp_ar.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"87577b3ff9d32712"}},{"code_sha256_prefix":"f6ff7671338c9c92","entry":"find_multiple","repo":"hywang66/LARP","repo_kind":"listed","path":"models/larp_ar.py","file_url":"https://github.com/hywang66/LARP/blob/HEAD/models/larp_ar.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f6ff7671338c9c92"}},{"code_sha256_prefix":"22a83df891d750c6","entry":"hinge_d_loss","repo":"hywang66/LARP","repo_kind":"listed","path":"models/loss.py","file_url":"https://github.com/hywang66/LARP/blob/HEAD/models/loss.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"22a83df891d750c6"}},{"code_sha256_prefix":"44c3aa4c610ea3f7","entry":"register","repo":"hywang66/LARP","repo_kind":"listed","path":"models/models.py","file_url":"https://github.com/hywang66/LARP/blob/HEAD/models/models.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"44c3aa4c610ea3f7"}},{"code_sha256_prefix":"b4701e8bb46ce4c9","entry":"entropy_loss","repo":"hywang66/LARP","repo_kind":"listed","path":"models/bottleneck.py","file_url":"https://github.com/hywang66/LARP/blob/HEAD/models/bottleneck.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b4701e8bb46ce4c9"}},{"code_sha256_prefix":"17acd1617dd8a808","entry":"get_2d_sincos_pos_embed","repo":"hywang66/LARP","repo_kind":"listed","path":"models/embed.py","file_url":"https://github.com/hywang66/LARP/blob/HEAD/models/embed.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"17acd1617dd8a808"}},{"code_sha256_prefix":"fd6ac2b105d8a7b5","entry":"get_2d_sincos_pos_embed_from_grid","repo":"hywang66/LARP","repo_kind":"listed","path":"models/embed.py","file_url":"https://github.com/hywang66/LARP/blob/HEAD/models/embed.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fd6ac2b105d8a7b5"}},{"code_sha256_prefix":"e8364ecd23bce54d","entry":"get_3d_sincos_pos_embed","repo":"hywang66/LARP","repo_kind":"listed","path":"models/embed.py","file_url":"https://github.com/hywang66/LARP/blob/HEAD/models/embed.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e8364ecd23bce54d"}},{"code_sha256_prefix":"f158589f38e59deb","entry":"get_optimizer_parameter_set","repo":"hywang66/LARP","repo_kind":"listed","path":"trainers/larp_tokenizer_trainer.py","file_url":"https://github.com/hywang66/LARP/blob/HEAD/trainers/larp_tokenizer_trainer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f158589f38e59deb"}},{"code_sha256_prefix":"1bc068182cddc396","entry":"get_orig_module","repo":"hywang66/LARP","repo_kind":"listed","path":"models/larp_tokenizer.py","file_url":"https://github.com/hywang66/LARP/blob/HEAD/models/larp_tokenizer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1bc068182cddc396"}},{"code_sha256_prefix":"9260cf8e6d12812c","entry":"lecam_reg","repo":"hywang66/LARP","repo_kind":"listed","path":"models/loss.py","file_url":"https://github.com/hywang66/LARP/blob/HEAD/models/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9260cf8e6d12812c"}},{"code_sha256_prefix":"8003b350e3563bc9","entry":"make","repo":"hywang66/LARP","repo_kind":"listed","path":"models/models.py","file_url":"https://github.com/hywang66/LARP/blob/HEAD/models/models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8003b350e3563bc9"}},{"code_sha256_prefix":"4ba231e52b8114c1","entry":"r1_gradient_penalty","repo":"hywang66/LARP","repo_kind":"listed","path":"models/loss.py","file_url":"https://github.com/hywang66/LARP/blob/HEAD/models/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4ba231e52b8114c1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}