{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/colpali-efficient-document-retrieval-with","title":"ColPali: Efficient Document Retrieval with Vision Language Models","arxiv_id":"2407.01449","date":"2024-06-27","proceeding":null,"authors":["Manuel Faysse","Hugues Sibille","Tony Wu","Bilel Omrani","Gautier Viaud","Céline Hudelot","Pierre Colombo"],"abstract":"Documents are visually rich structures that convey information through text, but also figures, page layouts, tables, or even fonts. Since modern retrieval systems mainly rely on the textual information they extract from document pages to index documents -often through lengthy and brittle processes-, they struggle to exploit key visual cues efficiently. This limits their capabilities in many practical document retrieval applications such as Retrieval Augmented Generation (RAG). To benchmark current systems on visually rich document retrieval, we introduce the Visual Document Retrieval Benchmark ViDoRe, composed of various page-level retrieval tasks spanning multiple domains, languages, and practical settings. The inherent complexity and performance shortcomings of modern systems motivate a new concept; doing document retrieval by directly embedding the images of the document pages. We release ColPali, a Vision Language Model trained to produce high-quality multi-vector embeddings from images of document pages. Combined with a late interaction matching mechanism, ColPali largely outperforms modern document retrieval pipelines while being drastically simpler, faster and end-to-end trainable. We release models, data, code and benchmarks under open licenses at https://hf.co/vidore.","url_abs":"https://arxiv.org/abs/2407.01449v6","url_pdf":"https://arxiv.org/pdf/2407.01449v6.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"colpali-efficient-document-retrieval-with","repo_url":"https://github.com/JingfenQiao/ColPali-Reproducibility","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"colpali-efficient-document-retrieval-with","repo_url":"https://github.com/abhijeet3922/vision-RAG","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"colpali-efficient-document-retrieval-with","repo_url":"https://github.com/illuin-tech/colpali","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"colpali-efficient-document-retrieval-with","repo_url":"https://github.com/illuin-tech/grouse","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"colpali-efficient-document-retrieval-with","repo_url":"https://github.com/illuin-tech/vidore-benchmark","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"rag","task_name":"RAG"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"retrieval-augmented-generation","task_name":"Retrieval-augmented Generation"},{"task_slug":"text-matching","task_name":"Text Matching"},{"task_slug":"document-understanding","task_name":"document understanding"}],"methods":[],"datasets_introduced":[{"slug":"vidore","name":"Vidore","full_name":"Visual Document Retrieval Benchmark"}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.01449","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.01449"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/illuin-tech/grouse","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/abhijeet3922/vision-RAG","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/JingfenQiao/ColPali-Reproducibility","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/illuin-tech/vidore-benchmark","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/illuin-tech/colpali","reach":null}],"summary":{"ran":2,"unverified":2},"by_repo_kind":{"listed":{"samples":4,"ran":2,"repositories":3}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bd8677ff14659624","entry":"ColPali","repo":"illuin-tech/colpali","repo_kind":"listed","path":"colpali_engine/models/paligemma/colpali/modeling_colpali.py","file_url":"https://github.com/illuin-tech/colpali/blob/HEAD/colpali_engine/models/paligemma/colpali/modeling_colpali.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd8677ff14659624"}},{"code_sha256_prefix":"21d0bddd3d24f1b9","entry":"ColPali","repo":"JingfenQiao/ColPali-Reproducibility","repo_kind":"listed","path":"colpali_engine/models/paligemma/colpali/modeling_colpali.py","file_url":"https://github.com/JingfenQiao/ColPali-Reproducibility/blob/HEAD/colpali_engine/models/paligemma/colpali/modeling_colpali.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"21d0bddd3d24f1b9"}},{"code_sha256_prefix":"dd905f8c3636c008","entry":"build_circle_legend","repo":"illuin-tech/grouse","repo_kind":"listed","path":"grouse/plot.py","file_url":"https://github.com/illuin-tech/grouse/blob/HEAD/grouse/plot.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dd905f8c3636c008"}},{"code_sha256_prefix":"fa815f8a869b3a27","entry":"nan_to_none","repo":"illuin-tech/grouse","repo_kind":"listed","path":"grouse/utils.py","file_url":"https://github.com/illuin-tech/grouse/blob/HEAD/grouse/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fa815f8a869b3a27"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}