{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/flashsloth-lightning-multimodal-large","title":"FlashSloth: Lightning Multimodal Large Language Models via Embedded Visual Compression","arxiv_id":"2412.04317","date":"2024-12-05","proceeding":null,"authors":["Bo Tong","Bokai Lai","Yiyi Zhou","Gen Luo","Yunhang Shen","Ke Li","Xiaoshuai Sun","Rongrong Ji"],"abstract":"Despite a big leap forward in capability, multimodal large language models (MLLMs) tend to behave like a sloth in practical use, i.e., slow response and large latency. Recent efforts are devoted to building tiny MLLMs for better efficiency, but the plethora of visual tokens still used limit their actual speedup. In this paper, we propose a powerful and fast tiny MLLM called FlashSloth. Different from previous efforts, FlashSloth focuses on improving the descriptive power of visual tokens in the process of compressing their redundant semantics. In particular, FlashSloth introduces embedded visual compression designs to capture both visually salient and instruction-related image information, so as to achieving superior multimodal performance with fewer visual tokens. Extensive experiments are conducted to validate the proposed FlashSloth, and a bunch of tiny but strong MLLMs are also comprehensively compared, e.g., InternVL2, MiniCPM-V2 and Qwen2-VL. The experimental results show that compared with these advanced tiny MLLMs, our FlashSloth can greatly reduce the number of visual tokens, training memory and computation complexity while retaining high performance on various VL tasks.","url_abs":"https://arxiv.org/abs/2412.04317v1","url_pdf":"https://arxiv.org/pdf/2412.04317v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"flashsloth-lightning-multimodal-large","repo_url":"https://github.com/codefanw/flashsloth","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"descriptive","task_name":"Descriptive"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-question-answering-on-mm-vet","task":"Visual Question Answering","dataset":"MM-Vet","model":"FlashSloth-HD","rank_in_archive_order":69,"of":231,"metrics":{"GPT-4 score":"49.0","Params":"3.2B"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-mm-vet","task":"Visual Question Answering","dataset":"MM-Vet","model":"FlashSloth","rank_in_archive_order":100,"of":231,"metrics":{"GPT-4 score":"41.9","Params":"3.2B"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.04317","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.04317"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/codefanw/flashsloth","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":3,"ran_fixture":1,"ran_violates":1,"ran_honours":1,"ran":1},"by_repo_kind":{"official":{"samples":7,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"592b3c1a88f93d7c","entry":"expand2square","repo":"codefanw/flashsloth","repo_kind":"official","path":"flashsloth/mm_utils.py","file_url":"https://github.com/codefanw/flashsloth/blob/HEAD/flashsloth/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"592b3c1a88f93d7c"}},{"code_sha256_prefix":"b0a851cae92754ff","entry":"extract_patches","repo":"codefanw/flashsloth","repo_kind":"official","path":"flashsloth/mm_utils.py","file_url":"https://github.com/codefanw/flashsloth/blob/HEAD/flashsloth/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b0a851cae92754ff"}},{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"codefanw/flashsloth","repo_kind":"official","path":"flashsloth/eval/model_vqa.py","file_url":"https://github.com/codefanw/flashsloth/blob/HEAD/flashsloth/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"bae18947b56f2be1","entry":"is_none","repo":"codefanw/flashsloth","repo_kind":"official","path":"flashsloth/eval/model_vqa_mmbench.py","file_url":"https://github.com/codefanw/flashsloth/blob/HEAD/flashsloth/eval/model_vqa_mmbench.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bae18947b56f2be1"}},{"code_sha256_prefix":"9b3c1cb391672ccb","entry":"load_image","repo":"codefanw/flashsloth","repo_kind":"official","path":"flashsloth/eval/model_vqa_qbench.py","file_url":"https://github.com/codefanw/flashsloth/blob/HEAD/flashsloth/eval/model_vqa_qbench.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9b3c1cb391672ccb"}},{"code_sha256_prefix":"c3ee9d07c900dd55","entry":"load_image_from_base64","repo":"codefanw/flashsloth","repo_kind":"official","path":"flashsloth/mm_utils.py","file_url":"https://github.com/codefanw/flashsloth/blob/HEAD/flashsloth/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c3ee9d07c900dd55"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"codefanw/flashsloth","repo_kind":"official","path":"flashsloth/eval/model_vqa.py","file_url":"https://github.com/codefanw/flashsloth/blob/HEAD/flashsloth/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}