{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/vamos-versatile-action-models-for-video","title":"Vamos: Versatile Action Models for Video Understanding","arxiv_id":"2311.13627","date":"2023-11-22","proceeding":null,"authors":["Shijie Wang","Qi Zhao","Minh Quan Do","Nakul Agarwal","Kwonjoon Lee","Chen Sun"],"abstract":"What makes good representations for video understanding, such as anticipating future activities, or answering video-conditioned questions? While earlier approaches focus on end-to-end learning directly from video pixels, we propose to revisit text-based representations, such as general-purpose video captions, which are interpretable and can be directly consumed by large language models (LLMs). Intuitively, different video understanding tasks may require representations that are complementary and at different granularity. To this end, we propose versatile action models (Vamos), a learning framework powered by a large language model as the ``reasoner'', and can flexibly leverage visual embedding and free-form text descriptions as its input. To interpret the important text evidence for question answering, we generalize the concept bottleneck model to work with tokens and nonlinear models, which uses hard attention to select a small subset of tokens from the free-form text as inputs to the LLM reasoner. We evaluate Vamos on five complementary benchmarks, Ego4D, NeXT-QA, IntentQA, Spacewalk-18, and EgoSchema, on its capability to model temporal dynamics, encode visual history, and perform reasoning. Surprisingly, we observe that text-based representations consistently achieve competitive performance on all benchmarks, and that visual embeddings provide marginal or no performance improvement, demonstrating the effectiveness of text-based video representation in the LLM era. We also demonstrate that our token bottleneck model is able to select relevant evidence from free-form text, support test-time intervention, and achieves nearly 5 times inference speedup while keeping a competitive question answering performance. Code and models are publicly released at https://brown-palm.github.io/Vamos/","url_abs":"https://arxiv.org/abs/2311.13627v3","url_pdf":"https://arxiv.org/pdf/2311.13627v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"vamos-versatile-action-models-for-video","repo_url":"https://github.com/brown-palm/Vamos","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":null,"task_name":"EgoSchema"},{"task_slug":"hard-attention","task_name":"Hard Attention"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"video-question-answering","task_name":"Video Question Answering"},{"task_slug":"video-understanding","task_name":"Video Understanding"},{"task_slug":"zeroshot-video-question-answer","task_name":"Zero-Shot Video Question Answer"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-question-answering-on-next-qa","task":"Video Question Answering","dataset":"NExT-QA","model":"Vamos","rank_in_archive_order":21,"of":47,"metrics":{"Accuracy":"77.3"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-video-question-answer-on-egoschema-1","task":"Zero-Shot Video Question Answer","dataset":"EgoSchema (fullset)","model":"Vamos (GPT-4o)","rank_in_archive_order":15,"of":29,"metrics":{"Accuracy":"53.6"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-video-question-answer-on-egoschema-1","task":"Zero-Shot Video Question Answer","dataset":"EgoSchema (fullset)","model":"Vamos (GPT-4)","rank_in_archive_order":19,"of":29,"metrics":{"Accuracy":"48.3"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-video-question-answer-on-egoschema-1","task":"Zero-Shot Video Question Answer","dataset":"EgoSchema (fullset)","model":"Vamos (13B)","rank_in_archive_order":22,"of":29,"metrics":{"Accuracy":"36.7"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2311.13627","atlas_url":"https://app.syntology.ai/?focus=2311.13627","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.13627"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/brown-palm/Vamos","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":1,"ran":3,"ran_violates":1,"ran_fixture":1,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bd3578806ecf5ac2","entry":"adjust_learning_rate","repo":"brown-palm/Vamos","repo_kind":"official","path":"finetune/util/lr_sched.py","file_url":"https://github.com/brown-palm/Vamos/blob/HEAD/finetune/util/lr_sched.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd3578806ecf5ac2"}},{"code_sha256_prefix":"3dc19396537db789","entry":"all_reduce_mean","repo":"brown-palm/Vamos","repo_kind":"official","path":"finetune/util/misc.py","file_url":"https://github.com/brown-palm/Vamos/blob/HEAD/finetune/util/misc.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3dc19396537db789"}},{"code_sha256_prefix":"9c4265cdca5af37c","entry":"getCount","repo":"brown-palm/Vamos","repo_kind":"official","path":"finetune/util/misc.py","file_url":"https://github.com/brown-palm/Vamos/blob/HEAD/finetune/util/misc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9c4265cdca5af37c"}},{"code_sha256_prefix":"14a84c2cbfebc413","entry":"precompute_freqs_cis","repo":"brown-palm/Vamos","repo_kind":"official","path":"finetune/llama/model.py","file_url":"https://github.com/brown-palm/Vamos/blob/HEAD/finetune/llama/model.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"14a84c2cbfebc413"}},{"code_sha256_prefix":"70bf6ebaafd266c4","entry":"reshape_for_broadcast","repo":"brown-palm/Vamos","repo_kind":"official","path":"finetune/llama/model.py","file_url":"https://github.com/brown-palm/Vamos/blob/HEAD/finetune/llama/model.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"70bf6ebaafd266c4"}},{"code_sha256_prefix":"8845976729f4c4ee","entry":"sample_top_p","repo":"brown-palm/Vamos","repo_kind":"official","path":"finetune/llama/generation.py","file_url":"https://github.com/brown-palm/Vamos/blob/HEAD/finetune/llama/generation.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8845976729f4c4ee"}},{"code_sha256_prefix":"a65fdcad6fda1883","entry":"set_precision","repo":"brown-palm/Vamos","repo_kind":"official","path":"finetune/llama/model.py","file_url":"https://github.com/brown-palm/Vamos/blob/HEAD/finetune/llama/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a65fdcad6fda1883"}},{"code_sha256_prefix":"ba1356e8ceb654d2","entry":"get_grad_norm_","repo":"brown-palm/Vamos","repo_kind":"official","path":"finetune/util/misc.py","file_url":"https://github.com/brown-palm/Vamos/blob/HEAD/finetune/util/misc.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ba1356e8ceb654d2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}