{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/llavaction-evaluating-and-training-multi","title":"LLaVAction: evaluating and training multi-modal large language models for action recognition","arxiv_id":"2503.18712","date":"2025-03-24","proceeding":null,"authors":["Shaokai Ye","Haozhe Qi","Alexander Mathis","Mackenzie W. Mathis"],"abstract":"Understanding human behavior requires measuring behavioral actions. Due to its complexity, behavior is best mapped onto a rich, semantic structure such as language. The recent development of multi-modal large language models (MLLMs) is a promising candidate for a wide range of action understanding tasks. In this work, we focus on evaluating and then improving MLLMs to perform action recognition. We reformulate EPIC-KITCHENS-100, one of the largest and most challenging egocentric action datasets, to the form of video multiple question answering (EPIC-KITCHENS-100-MQA). We show that when we sample difficult incorrect answers as distractors, leading MLLMs struggle to recognize the correct actions. We propose a series of methods that greatly improve the MLLMs' ability to perform action recognition, achieving state-of-the-art on both the EPIC-KITCHENS-100 validation set, as well as outperforming GPT-4o by 21 points in accuracy on EPIC-KITCHENS-100-MQA. Lastly, we show improvements on other action-related video benchmarks such as EgoSchema, PerceptionTest, LongVideoBench, VideoMME and MVBench, suggesting that MLLMs are a promising path forward for complex action tasks. Code and models are available at: https://github.com/AdaptiveMotorControlLab/LLaVAction.","url_abs":"https://arxiv.org/abs/2503.18712v1","url_pdf":"https://arxiv.org/pdf/2503.18712v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"llavaction-evaluating-and-training-multi","repo_url":"https://github.com/adaptivemotorcontrollab/llavaction","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"action-understanding","task_name":"Action Understanding"},{"task_slug":null,"task_name":"EgoSchema"},{"task_slug":null,"task_name":"MVBench"},{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-recognition-on-epic-kitchens-100","task":"Action Recognition","dataset":"EPIC-KITCHENS-100","model":"LLaVAction","rank_in_archive_order":1,"of":32,"metrics":{"Action@1":"58.3","Noun@1":"69","Verb@1":"76"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.18712","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}