{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/livexiv-a-multi-modal-live-benchmark-based-on","title":"LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Content","arxiv_id":"2410.10783","date":"2024-10-14","proceeding":null,"authors":["Nimrod Shabtay","Felipe Maia Polo","Sivan Doveh","Wei Lin","M. Jehanzeb Mirza","Leshem Chosen","Mikhail Yurochkin","Yuekai Sun","Assaf Arbelle","Leonid Karlinsky","Raja Giryes"],"abstract":"The large-scale training of multi-modal models on data scraped from the web has shown outstanding utility in infusing these models with the required world knowledge to perform effectively on multiple downstream tasks. However, one downside of scraping data from the web can be the potential sacrifice of the benchmarks on which the abilities of these models are often evaluated. To safeguard against test data contamination and to truly test the abilities of these foundation models we propose LiveXiv: A scalable evolving live benchmark based on scientific ArXiv papers. LiveXiv accesses domain-specific manuscripts at any given timestamp and proposes to automatically generate visual question-answer pairs (VQA). This is done without any human-in-the-loop, using the multi-modal content in the manuscripts, like graphs, charts, and tables. Moreover, we introduce an efficient evaluation approach that estimates the performance of all models on the evolving benchmark using evaluations of only a subset of models. This significantly reduces the overall evaluation cost. We benchmark multiple open and proprietary Large Multi-modal Models (LMMs) on the first version of our benchmark, showing its challenging nature and exposing the models true abilities, avoiding contamination. Lastly, in our commitment to high quality, we have collected and evaluated a manually verified subset. By comparing its overall results to our automatic annotations, we have found that the performance variance is indeed minimal (<2.5%). Our dataset is available online on HuggingFace, and our code will be available here.","url_abs":"https://arxiv.org/abs/2410.10783v2","url_pdf":"https://arxiv.org/pdf/2410.10783v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"livexiv-a-multi-modal-live-benchmark-based-on","repo_url":"https://github.com/nimrodshabtay/livexiv","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"world-knowledge","task_name":"World Knowledge"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.10783","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.10783"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nimrodshabtay/livexiv","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":1,"ran":16,"unverified":2},"by_repo_kind":{"official":{"samples":19,"ran":17,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"20e4f665698a3d18","entry":"collate_fn","repo":"nimrodshabtay/livexiv","repo_kind":"official","path":"vqa_generation/llava_eval.py","file_url":"https://github.com/nimrodshabtay/livexiv/blob/HEAD/vqa_generation/llava_eval.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"20e4f665698a3d18"}},{"code_sha256_prefix":"55c5aa2c82585ebd","entry":"collect_all_questions","repo":"nimrodshabtay/livexiv","repo_kind":"official","path":"vqa_generation/evaluation_utils.py","file_url":"https://github.com/nimrodshabtay/livexiv/blob/HEAD/vqa_generation/evaluation_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"55c5aa2c82585ebd"}},{"code_sha256_prefix":"730b80d67018d2aa","entry":"create_prompt","repo":"nimrodshabtay/livexiv","repo_kind":"official","path":"vqa_generation/filter_utils.py","file_url":"https://github.com/nimrodshabtay/livexiv/blob/HEAD/vqa_generation/filter_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"730b80d67018d2aa"}},{"code_sha256_prefix":"beb4056b7230f14f","entry":"encode_image","repo":"nimrodshabtay/livexiv","repo_kind":"official","path":"vqa_generation/model_utils/claude_utils.py","file_url":"https://github.com/nimrodshabtay/livexiv/blob/HEAD/vqa_generation/model_utils/claude_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"beb4056b7230f14f"}},{"code_sha256_prefix":"826cad294a32186a","entry":"encode_image","repo":"nimrodshabtay/livexiv","repo_kind":"official","path":"vqa_generation/model_utils/gpt_utils.py","file_url":"https://github.com/nimrodshabtay/livexiv/blob/HEAD/vqa_generation/model_utils/gpt_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"826cad294a32186a"}},{"code_sha256_prefix":"cda3c93c1984bd12","entry":"extract_gt","repo":"nimrodshabtay/livexiv","repo_kind":"official","path":"vqa_generation/evaluation_utils.py","file_url":"https://github.com/nimrodshabtay/livexiv/blob/HEAD/vqa_generation/evaluation_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cda3c93c1984bd12"}},{"code_sha256_prefix":"cc0b40f9f77cdf10","entry":"get_all_img_cap_pairs_above_length_th","repo":"nimrodshabtay/livexiv","repo_kind":"official","path":"vqa_generation/document_utils.py","file_url":"https://github.com/nimrodshabtay/livexiv/blob/HEAD/vqa_generation/document_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cc0b40f9f77cdf10"}},{"code_sha256_prefix":"9908efa6f99f8c6f","entry":"get_general_description_prompt","repo":"nimrodshabtay/livexiv","repo_kind":"official","path":"vqa_generation/model_utils/claude_utils.py","file_url":"https://github.com/nimrodshabtay/livexiv/blob/HEAD/vqa_generation/model_utils/claude_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9908efa6f99f8c6f"}},{"code_sha256_prefix":"b00f8fb2fa7ece30","entry":"group_by","repo":"nimrodshabtay/livexiv","repo_kind":"official","path":"vqa_generation/filter_utils.py","file_url":"https://github.com/nimrodshabtay/livexiv/blob/HEAD/vqa_generation/filter_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b00f8fb2fa7ece30"}},{"code_sha256_prefix":"ec7e7e421e348522","entry":"irt_forward","repo":"nimrodshabtay/livexiv","repo_kind":"official","path":"livexiv/efficient_eval.py","file_url":"https://github.com/nimrodshabtay/livexiv/blob/HEAD/livexiv/efficient_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ec7e7e421e348522"}},{"code_sha256_prefix":"fde558d67cd0c6b6","entry":"load_description_from_image_path","repo":"nimrodshabtay/livexiv","repo_kind":"official","path":"vqa_generation/document_utils.py","file_url":"https://github.com/nimrodshabtay/livexiv/blob/HEAD/vqa_generation/document_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fde558d67cd0c6b6"}},{"code_sha256_prefix":"3a18110e5cc4b51e","entry":"loss_matrix","repo":"nimrodshabtay/livexiv","repo_kind":"official","path":"livexiv/efficient_eval.py","file_url":"https://github.com/nimrodshabtay/livexiv/blob/HEAD/livexiv/efficient_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3a18110e5cc4b51e"}},{"code_sha256_prefix":"4f8e0c5e2a5c0e2f","entry":"nearest_multiple_crop","repo":"nimrodshabtay/livexiv","repo_kind":"official","path":"vqa_generation/model_utils/gpt_utils.py","file_url":"https://github.com/nimrodshabtay/livexiv/blob/HEAD/vqa_generation/model_utils/gpt_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4f8e0c5e2a5c0e2f"}},{"code_sha256_prefix":"9e6dbf5093b8e74f","entry":"prepare_questions_for_eval","repo":"nimrodshabtay/livexiv","repo_kind":"official","path":"vqa_generation/evaluation_utils.py","file_url":"https://github.com/nimrodshabtay/livexiv/blob/HEAD/vqa_generation/evaluation_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9e6dbf5093b8e74f"}},{"code_sha256_prefix":"b69516ff4137a498","entry":"replace_delimiters","repo":"nimrodshabtay/livexiv","repo_kind":"official","path":"vqa_generation/generate_raw_questions.py","file_url":"https://github.com/nimrodshabtay/livexiv/blob/HEAD/vqa_generation/generate_raw_questions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b69516ff4137a498"}},{"code_sha256_prefix":"4ac30ea009f8b618","entry":"resize_to_closest_ar","repo":"nimrodshabtay/livexiv","repo_kind":"official","path":"vqa_generation/model_utils/claude_utils.py","file_url":"https://github.com/nimrodshabtay/livexiv/blob/HEAD/vqa_generation/model_utils/claude_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4ac30ea009f8b618"}},{"code_sha256_prefix":"bbe6df2991cd4f9c","entry":"without_duplicates","repo":"nimrodshabtay/livexiv","repo_kind":"official","path":"vqa_generation/filter_utils.py","file_url":"https://github.com/nimrodshabtay/livexiv/blob/HEAD/vqa_generation/filter_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bbe6df2991cd4f9c"}},{"code_sha256_prefix":"b8e78f199e05bc35","entry":"EfficEval","repo":"nimrodshabtay/livexiv","repo_kind":"official","path":"livexiv/efficient_eval.py","file_url":"https://github.com/nimrodshabtay/livexiv/blob/HEAD/livexiv/efficient_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b8e78f199e05bc35"}},{"code_sha256_prefix":"199fbacfdb5ba1ef","entry":"classify_figure_type","repo":"nimrodshabtay/livexiv","repo_kind":"official","path":"vqa_generation/classify_figures.py","file_url":"https://github.com/nimrodshabtay/livexiv/blob/HEAD/vqa_generation/classify_figures.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"199fbacfdb5ba1ef"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}