{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multimodal-table-understanding","title":"Multimodal Table Understanding","arxiv_id":"2406.08100","date":"2024-06-12","proceeding":null,"authors":["Mingyu Zheng","Xinwei Feng","Qingyi Si","Qiaoqiao She","Zheng Lin","Wenbin Jiang","Weiping Wang"],"abstract":"Although great progress has been made by previous table understanding methods including recent approaches based on large language models (LLMs), they rely heavily on the premise that given tables must be converted into a certain text sequence (such as Markdown or HTML) to serve as model input. However, it is difficult to access such high-quality textual table representations in some real-world scenarios, and table images are much more accessible. Therefore, how to directly understand tables using intuitive visual information is a crucial and urgent challenge for developing more practical applications. In this paper, we propose a new problem, multimodal table understanding, where the model needs to generate correct responses to various table-related requests based on the given table image. To facilitate both the model training and evaluation, we construct a large-scale dataset named MMTab, which covers a wide spectrum of table images, instructions and tasks. On this basis, we develop Table-LLaVA, a generalist tabular multimodal large language model (MLLM), which significantly outperforms recent open-source MLLM baselines on 23 benchmarks under held-in and held-out settings. The code and data is available at this https://github.com/SpursGoZmy/Table-LLaVA","url_abs":"https://arxiv.org/abs/2406.08100v1","url_pdf":"https://arxiv.org/pdf/2406.08100v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"multimodal-table-understanding","repo_url":"https://github.com/spursgozmy/table-llava","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"multimodal-large-language-model","task_name":"Multimodal Large Language Model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.08100","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.08100"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/spursgozmy/table-llava","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/SpursGoZmy/Table-LLaVA","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":3,"ran_violates":1,"ran_honours":1,"ran":3,"unverified":2},"by_repo_kind":{"official":{"samples":10,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"592b3c1a88f93d7c","entry":"expand2square","repo":"spursgozmy/table-llava","repo_kind":"official","path":"llava/mm_utils.py","file_url":"https://github.com/spursgozmy/table-llava/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"592b3c1a88f93d7c"}},{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"spursgozmy/table-llava","repo_kind":"official","path":"llava/eval/model_vqa.py","file_url":"https://github.com/spursgozmy/table-llava/blob/HEAD/llava/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"bae18947b56f2be1","entry":"is_none","repo":"spursgozmy/table-llava","repo_kind":"official","path":"llava/eval/model_vqa_mmbench.py","file_url":"https://github.com/spursgozmy/table-llava/blob/HEAD/llava/eval/model_vqa_mmbench.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bae18947b56f2be1"}},{"code_sha256_prefix":"9b3c1cb391672ccb","entry":"load_image","repo":"spursgozmy/table-llava","repo_kind":"official","path":"llava/eval/model_vqa_qbench.py","file_url":"https://github.com/spursgozmy/table-llava/blob/HEAD/llava/eval/model_vqa_qbench.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9b3c1cb391672ccb"}},{"code_sha256_prefix":"c3ee9d07c900dd55","entry":"load_image_from_base64","repo":"spursgozmy/table-llava","repo_kind":"official","path":"llava/mm_utils.py","file_url":"https://github.com/spursgozmy/table-llava/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c3ee9d07c900dd55"}},{"code_sha256_prefix":"7beeb01acc36919e","entry":"read_jsonl","repo":"spursgozmy/table-llava","repo_kind":"official","path":"llava/eval/generate_webpage_data_from_table.py","file_url":"https://github.com/spursgozmy/table-llava/blob/HEAD/llava/eval/generate_webpage_data_from_table.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7beeb01acc36919e"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"spursgozmy/table-llava","repo_kind":"official","path":"llava/eval/model_vqa.py","file_url":"https://github.com/spursgozmy/table-llava/blob/HEAD/llava/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}},{"code_sha256_prefix":"1d45bc90fd0cd90c","entry":"trim_hanging_lines","repo":"spursgozmy/table-llava","repo_kind":"official","path":"llava/eval/generate_webpage_data_from_table.py","file_url":"https://github.com/spursgozmy/table-llava/blob/HEAD/llava/eval/generate_webpage_data_from_table.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1d45bc90fd0cd90c"}},{"code_sha256_prefix":"2f7959e2c2e658be","entry":"LlavaMetaModel","repo":"SpursGoZmy/Table-LLaVA","repo_kind":"official","path":"llava/model/llava_arch.py","file_url":"https://github.com/SpursGoZmy/Table-LLaVA/blob/HEAD/llava/model/llava_arch.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2f7959e2c2e658be"}},{"code_sha256_prefix":"344dff4791fd1381","entry":"process_images","repo":"spursgozmy/table-llava","repo_kind":"official","path":"llava/mm_utils.py","file_url":"https://github.com/spursgozmy/table-llava/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"344dff4791fd1381"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}