{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-picture-is-worth-more-than-77-text-tokens","title":"A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions","arxiv_id":"2312.08578","date":"2023-12-14","proceeding":"CVPR 2024 1","authors":["Jack Urbanek","Florian Bordes","Pietro Astolfi","Mary Williamson","Vasu Sharma","Adriana Romero-Soriano"],"abstract":"Curation methods for massive vision-language datasets trade off between dataset size and quality. However, even the highest quality of available curated captions are far too short to capture the rich visual detail in an image. To show the value of dense and highly-aligned image-text pairs, we collect the Densely Captioned Images (DCI) dataset, containing 7805 natural images human-annotated with mask-aligned descriptions averaging above 1000 words each. With precise and reliable captions associated with specific parts of an image, we can evaluate vision-language models' (VLMs) understanding of image content with a novel task that matches each caption with its corresponding subcrop. As current models are often limited to 77 text tokens, we also introduce a summarized version (sDCI) in which each caption length is limited. We show that modern techniques that make progress on standard benchmarks do not correspond with significant improvement on our sDCI based benchmark. Lastly, we finetune CLIP using sDCI and show significant improvements over the baseline despite a small training set. By releasing the first human annotated dense image captioning dataset, we hope to enable the development of new benchmarks or fine-tuning recipes for the next generation of VLMs to come.","url_abs":"https://arxiv.org/abs/2312.08578v2","url_pdf":"https://arxiv.org/pdf/2312.08578v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-picture-is-worth-more-than-77-text-tokens","repo_url":"https://github.com/facebookresearch/dci","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"image-captioning","task_name":"Image Captioning"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2312.08578","atlas_url":"https://app.syntology.ai/?focus=2312.08578","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.08578"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/dci","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":8,"unverified":4},"by_repo_kind":{"official":{"samples":12,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"1cee1f63cec4466a","entry":"different_enough","repo":"facebookresearch/dci","repo_kind":"official","path":"dataset/densely_captioned_images/dataset/spacy_negs.py","file_url":"https://github.com/facebookresearch/dci/blob/HEAD/dataset/densely_captioned_images/dataset/spacy_negs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1cee1f63cec4466a"}},{"code_sha256_prefix":"ffb19f5fbd27f6b3","entry":"extract_final_data","repo":"facebookresearch/dci","repo_kind":"official","path":"reproduction/crowdsourcing/annotate/export_data.py","file_url":"https://github.com/facebookresearch/dci/blob/HEAD/reproduction/crowdsourcing/annotate/export_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ffb19f5fbd27f6b3"}},{"code_sha256_prefix":"2690cac287c66746","entry":"extract_stats_data","repo":"facebookresearch/dci","repo_kind":"official","path":"reproduction/crowdsourcing/annotate/get_worker_stats.py","file_url":"https://github.com/facebookresearch/dci/blob/HEAD/reproduction/crowdsourcing/annotate/get_worker_stats.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2690cac287c66746"}},{"code_sha256_prefix":"e61c479b6d80bf20","entry":"get_all_words","repo":"facebookresearch/dci","repo_kind":"official","path":"reproduction/crowdsourcing/annotate/get_worker_stats.py","file_url":"https://github.com/facebookresearch/dci/blob/HEAD/reproduction/crowdsourcing/annotate/get_worker_stats.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e61c479b6d80bf20"}},{"code_sha256_prefix":"8bfc1fd87426be36","entry":"get_grouped_diag","repo":"facebookresearch/dci","repo_kind":"official","path":"dataset/densely_captioned_images/dataset/loss.py","file_url":"https://github.com/facebookresearch/dci/blob/HEAD/dataset/densely_captioned_images/dataset/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8bfc1fd87426be36"}},{"code_sha256_prefix":"89e4607c1001611d","entry":"get_pooled_diag","repo":"facebookresearch/dci","repo_kind":"official","path":"dataset/densely_captioned_images/dataset/loss.py","file_url":"https://github.com/facebookresearch/dci/blob/HEAD/dataset/densely_captioned_images/dataset/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"89e4607c1001611d"}},{"code_sha256_prefix":"fa46a5068238794a","entry":"get_pooled_groups","repo":"facebookresearch/dci","repo_kind":"official","path":"dataset/densely_captioned_images/dataset/loss.py","file_url":"https://github.com/facebookresearch/dci/blob/HEAD/dataset/densely_captioned_images/dataset/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"fa46a5068238794a"}},{"code_sha256_prefix":"90a8e73fa6de72a4","entry":"levenshtein_distance","repo":"facebookresearch/dci","repo_kind":"official","path":"dataset/densely_captioned_images/dataset/spacy_negs.py","file_url":"https://github.com/facebookresearch/dci/blob/HEAD/dataset/densely_captioned_images/dataset/spacy_negs.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"90a8e73fa6de72a4"}},{"code_sha256_prefix":"31cd14e7516ac2be","entry":"build_tasks","repo":"facebookresearch/dci","repo_kind":"official","path":"reproduction/crowdsourcing/annotate/run_task.py","file_url":"https://github.com/facebookresearch/dci/blob/HEAD/reproduction/crowdsourcing/annotate/run_task.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"31cd14e7516ac2be"}},{"code_sha256_prefix":"3aa3ce7d8f3a139f","entry":"get_antonyms","repo":"facebookresearch/dci","repo_kind":"official","path":"dataset/densely_captioned_images/dataset/spacy_negs.py","file_url":"https://github.com/facebookresearch/dci/blob/HEAD/dataset/densely_captioned_images/dataset/spacy_negs.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"3aa3ce7d8f3a139f"}},{"code_sha256_prefix":"62c8d5458ed48728","entry":"get_clip_token_length","repo":"facebookresearch/dci","repo_kind":"official","path":"dataset/densely_captioned_images/dataset/utils.py","file_url":"https://github.com/facebookresearch/dci/blob/HEAD/dataset/densely_captioned_images/dataset/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"62c8d5458ed48728"}},{"code_sha256_prefix":"8bf1cef2fb35e633","entry":"truncate_long_captions","repo":"facebookresearch/dci","repo_kind":"official","path":"dataset/densely_captioned_images/dataset/utils.py","file_url":"https://github.com/facebookresearch/dci/blob/HEAD/dataset/densely_captioned_images/dataset/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8bf1cef2fb35e633"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}