{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/clip4str-a-simple-baseline-for-scene-text-1","title":"CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Model","arxiv_id":"2305.14014","date":"2023-05-23","proceeding":null,"authors":["Shuai Zhao","Ruijie Quan","Linchao Zhu","Yi Yang"],"abstract":"Pre-trained vision-language models~(VLMs) are the de-facto foundation models for various downstream tasks. However, scene text recognition methods still prefer backbones pre-trained on a single modality, namely, the visual modality, despite the potential of VLMs to serve as powerful scene text readers. For example, CLIP can robustly identify regular (horizontal) and irregular (rotated, curved, blurred, or occluded) text in images. With such merits, we transform CLIP into a scene text reader and introduce CLIP4STR, a simple yet effective STR method built upon image and text encoders of CLIP. It has two encoder-decoder branches: a visual branch and a cross-modal branch. The visual branch provides an initial prediction based on the visual feature, and the cross-modal branch refines this prediction by addressing the discrepancy between the visual feature and text semantics. To fully leverage the capabilities of both branches, we design a dual predict-and-refine decoding scheme for inference. We scale CLIP4STR in terms of the model size, pre-training data, and training data, achieving state-of-the-art performance on 13 STR benchmarks. Additionally, a comprehensive empirical study is provided to enhance the understanding of the adaptation of CLIP to STR. Our method establishes a simple yet strong baseline for future STR research with VLMs.","url_abs":"https://arxiv.org/abs/2305.14014v4","url_pdf":"https://arxiv.org/pdf/2305.14014v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"clip4str-a-simple-baseline-for-scene-text-1","repo_url":"https://github.com/VamosC/CLIP4STR","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"scene-text-recognition","task_name":"Scene Text Recognition"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/scene-text-recognition-on-coco-text","task":"Scene Text Recognition","dataset":"COCO-Text","model":"CLIP4STR-L","rank_in_archive_order":1,"of":4,"metrics":{"1:1 Accuracy":"81.9"},"uses_additional_data":true},{"leaderboard":"/sota/scene-text-recognition-on-coco-text","task":"Scene Text Recognition","dataset":"COCO-Text","model":"CLIP4STR-B","rank_in_archive_order":3,"of":4,"metrics":{"1:1 Accuracy":"81.1"},"uses_additional_data":true},{"leaderboard":"/sota/scene-text-recognition-on-cute80","task":"Scene Text Recognition","dataset":"CUTE80","model":"CLIP4STR-L (DataComp-1B)","rank_in_archive_order":2,"of":18,"metrics":{"Accuracy":"99.7"},"uses_additional_data":true},{"leaderboard":"/sota/scene-text-recognition-on-cute80","task":"Scene Text Recognition","dataset":"CUTE80","model":"CLIP4STR-B","rank_in_archive_order":5,"of":18,"metrics":{"Accuracy":"99.3"},"uses_additional_data":true},{"leaderboard":"/sota/scene-text-recognition-on-cute80","task":"Scene Text Recognition","dataset":"CUTE80","model":"CLIP4STR-L","rank_in_archive_order":7,"of":18,"metrics":{"Accuracy":"99.0"},"uses_additional_data":true},{"leaderboard":"/sota/scene-text-recognition-on-host","task":"Scene Text Recognition","dataset":"HOST","model":"CLIP4STR-L","rank_in_archive_order":1,"of":3,"metrics":{"1:1 Accuracy":"82.7"},"uses_additional_data":true},{"leaderboard":"/sota/scene-text-recognition-on-host","task":"Scene Text Recognition","dataset":"HOST","model":"CLIP4STR-B","rank_in_archive_order":2,"of":3,"metrics":{"1:1 Accuracy":"79.8"},"uses_additional_data":true},{"leaderboard":"/sota/scene-text-recognition-on-ic19-art","task":"Scene Text Recognition","dataset":"IC19-Art","model":"CLIP4STR-L (DataComp-1B)","rank_in_archive_order":1,"of":5,"metrics":{"Accuracy (%)":"86.4"},"uses_additional_data":false},{"leaderboard":"/sota/scene-text-recognition-on-ic19-art","task":"Scene Text Recognition","dataset":"IC19-Art","model":"CLIP4STR-L","rank_in_archive_order":2,"of":5,"metrics":{"Accuracy (%)":"85.9"},"uses_additional_data":true},{"leaderboard":"/sota/scene-text-recognition-on-ic19-art","task":"Scene Text Recognition","dataset":"IC19-Art","model":"CLIP4STR-B","rank_in_archive_order":3,"of":5,"metrics":{"Accuracy (%)":"85.8"},"uses_additional_data":true},{"leaderboard":"/sota/scene-text-recognition-on-icdar2013","task":"Scene Text Recognition","dataset":"ICDAR2013","model":"CLIP4STR-L (DataComp-1B)","rank_in_archive_order":3,"of":38,"metrics":{"Accuracy":"99.0"},"uses_additional_data":false},{"leaderboard":"/sota/scene-text-recognition-on-icdar2013","task":"Scene Text Recognition","dataset":"ICDAR2013","model":"CLIP4STR-L","rank_in_archive_order":5,"of":38,"metrics":{"Accuracy":"98.5"},"uses_additional_data":true},{"leaderboard":"/sota/scene-text-recognition-on-icdar2013","task":"Scene Text Recognition","dataset":"ICDAR2013","model":"CLIP4STR-B","rank_in_archive_order":9,"of":38,"metrics":{"Accuracy":"98.3"},"uses_additional_data":true},{"leaderboard":"/sota/scene-text-recognition-on-icdar2015","task":"Scene Text Recognition","dataset":"ICDAR2015","model":"CLIP4STR-L (DataComp-1B)","rank_in_archive_order":4,"of":27,"metrics":{"Accuracy":"91.4"},"uses_additional_data":false},{"leaderboard":"/sota/scene-text-recognition-on-icdar2015","task":"Scene Text Recognition","dataset":"ICDAR2015","model":"CLIP4STR-L","rank_in_archive_order":6,"of":27,"metrics":{"Accuracy":"90.8"},"uses_additional_data":true},{"leaderboard":"/sota/scene-text-recognition-on-icdar2015","task":"Scene Text Recognition","dataset":"ICDAR2015","model":"CLIP4STR-B","rank_in_archive_order":7,"of":27,"metrics":{"Accuracy":"90.6"},"uses_additional_data":true},{"leaderboard":"/sota/scene-text-recognition-on-iiit5k","task":"Scene Text Recognition","dataset":"IIIT5k","model":"CLIP4STR-L (DataComp-1B)","rank_in_archive_order":1,"of":17,"metrics":{"Accuracy":"99.6"},"uses_additional_data":true},{"leaderboard":"/sota/scene-text-recognition-on-iiit5k","task":"Scene Text Recognition","dataset":"IIIT5k","model":"CLIP4STR-L","rank_in_archive_order":3,"of":17,"metrics":{"Accuracy":"99.5"},"uses_additional_data":true},{"leaderboard":"/sota/scene-text-recognition-on-iiit5k","task":"Scene Text Recognition","dataset":"IIIT5k","model":"CLIP4STR-B (DataComp-1B)","rank_in_archive_order":4,"of":17,"metrics":{"Accuracy":"99.5"},"uses_additional_data":true},{"leaderboard":"/sota/scene-text-recognition-on-iiit5k","task":"Scene Text Recognition","dataset":"IIIT5k","model":"CLIP4STR-B","rank_in_archive_order":6,"of":17,"metrics":{"Accuracy":"99.2"},"uses_additional_data":true},{"leaderboard":"/sota/scene-text-recognition-on-svt","task":"Scene Text Recognition","dataset":"SVT","model":"CLIP4STR-H (DFN-5B)","rank_in_archive_order":1,"of":37,"metrics":{"Accuracy":"99.1"},"uses_additional_data":true},{"leaderboard":"/sota/scene-text-recognition-on-svt","task":"Scene Text Recognition","dataset":"SVT","model":"CLIP4STR-L (DataComp-1B)","rank_in_archive_order":5,"of":37,"metrics":{"Accuracy":"98.6"},"uses_additional_data":true},{"leaderboard":"/sota/scene-text-recognition-on-svt","task":"Scene Text Recognition","dataset":"SVT","model":"CLIP4STR-L","rank_in_archive_order":7,"of":37,"metrics":{"Accuracy":"98.5"},"uses_additional_data":true},{"leaderboard":"/sota/scene-text-recognition-on-svt","task":"Scene Text Recognition","dataset":"SVT","model":"CLIP4STR-B","rank_in_archive_order":8,"of":37,"metrics":{"Accuracy":"98.3"},"uses_additional_data":true},{"leaderboard":"/sota/scene-text-recognition-on-svtp","task":"Scene Text Recognition","dataset":"SVTP","model":"CLIP4STR-L (DataComp-1B)","rank_in_archive_order":4,"of":17,"metrics":{"Accuracy":"98.1"},"uses_additional_data":false},{"leaderboard":"/sota/scene-text-recognition-on-svtp","task":"Scene Text Recognition","dataset":"SVTP","model":"CLIP4STR-L","rank_in_archive_order":5,"of":17,"metrics":{"Accuracy":"97.4"},"uses_additional_data":false},{"leaderboard":"/sota/scene-text-recognition-on-svtp","task":"Scene Text Recognition","dataset":"SVTP","model":"CLIP4STR-B","rank_in_archive_order":6,"of":17,"metrics":{"Accuracy":"97.2"},"uses_additional_data":true},{"leaderboard":"/sota/scene-text-recognition-on-uber-text","task":"Scene Text Recognition","dataset":"Uber-Text","model":"CLIP4STR-L (DataComp-1B)","rank_in_archive_order":1,"of":3,"metrics":{"Accuracy (%)":"92.2"},"uses_additional_data":false},{"leaderboard":"/sota/scene-text-recognition-on-uber-text","task":"Scene Text Recognition","dataset":"Uber-Text","model":"CLIP4STR-B","rank_in_archive_order":3,"of":3,"metrics":{"Accuracy (%)":"86.8"},"uses_additional_data":true},{"leaderboard":"/sota/scene-text-recognition-on-wost","task":"Scene Text Recognition","dataset":"WOST","model":"CLIP4STR-H (DFN-5B)","rank_in_archive_order":1,"of":5,"metrics":{"1:1 Accuracy":"90.9"},"uses_additional_data":false},{"leaderboard":"/sota/scene-text-recognition-on-wost","task":"Scene Text Recognition","dataset":"WOST","model":"CLIP4STR-L (DataComp-1B)","rank_in_archive_order":2,"of":5,"metrics":{"1:1 Accuracy":"90.6"},"uses_additional_data":true},{"leaderboard":"/sota/scene-text-recognition-on-wost","task":"Scene Text Recognition","dataset":"WOST","model":"CLIP4STR-L","rank_in_archive_order":3,"of":5,"metrics":{"1:1 Accuracy":"88.8"},"uses_additional_data":true},{"leaderboard":"/sota/scene-text-recognition-on-wost","task":"Scene Text Recognition","dataset":"WOST","model":"CLIP4STR-B","rank_in_archive_order":4,"of":5,"metrics":{"1:1 Accuracy":"87.0"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.14014","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.14014"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/VamosC/CLIP4STR","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":3,"unverified":7},"by_repo_kind":{"named_in_paper":{"samples":10,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"98f385d847636a3e","entry":"basic_clean","repo":"VamosC/CLIP4STR","repo_kind":"named_in_paper","path":"strhub/clip/simple_tokenizer.py","file_url":"https://github.com/VamosC/CLIP4STR/blob/HEAD/strhub/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"98f385d847636a3e"}},{"code_sha256_prefix":"d919ae32e5e4e616","entry":"get_pairs","repo":"VamosC/CLIP4STR","repo_kind":"named_in_paper","path":"strhub/clip/simple_tokenizer.py","file_url":"https://github.com/VamosC/CLIP4STR/blob/HEAD/strhub/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d919ae32e5e4e616"}},{"code_sha256_prefix":"9542161e9640b858","entry":"whitespace_clean","repo":"VamosC/CLIP4STR","repo_kind":"named_in_paper","path":"strhub/clip/simple_tokenizer.py","file_url":"https://github.com/VamosC/CLIP4STR/blob/HEAD/strhub/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9542161e9640b858"}},{"code_sha256_prefix":"34938893fe02d1f5","entry":"build_model","repo":"VamosC/CLIP4STR","repo_kind":"named_in_paper","path":"strhub/clip/model.py","file_url":"https://github.com/VamosC/CLIP4STR/blob/HEAD/strhub/clip/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"34938893fe02d1f5"}},{"code_sha256_prefix":"9b2f2fb695d21458","entry":"create_model","repo":"VamosC/CLIP4STR","repo_kind":"named_in_paper","path":"strhub/models/utils.py","file_url":"https://github.com/VamosC/CLIP4STR/blob/HEAD/strhub/models/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9b2f2fb695d21458"}},{"code_sha256_prefix":"406ad6d7f2c448f2","entry":"load","repo":"VamosC/CLIP4STR","repo_kind":"named_in_paper","path":"strhub/clip/clip.py","file_url":"https://github.com/VamosC/CLIP4STR/blob/HEAD/strhub/clip/clip.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"406ad6d7f2c448f2"}},{"code_sha256_prefix":"318b6adc36f8db42","entry":"load_from_checkpoint","repo":"VamosC/CLIP4STR","repo_kind":"named_in_paper","path":"strhub/models/utils.py","file_url":"https://github.com/VamosC/CLIP4STR/blob/HEAD/strhub/models/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"318b6adc36f8db42"}},{"code_sha256_prefix":"50d4d49ecae7edce","entry":"parse_model_args","repo":"VamosC/CLIP4STR","repo_kind":"named_in_paper","path":"strhub/models/utils.py","file_url":"https://github.com/VamosC/CLIP4STR/blob/HEAD/strhub/models/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"50d4d49ecae7edce"}},{"code_sha256_prefix":"784520bd614f333a","entry":"top_k_candidate","repo":"VamosC/CLIP4STR","repo_kind":"named_in_paper","path":"strhub/models/post_process.py","file_url":"https://github.com/VamosC/CLIP4STR/blob/HEAD/strhub/models/post_process.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"784520bd614f333a"}},{"code_sha256_prefix":"ec0f91c34fb1b379","entry":"top_k_candidate_v2","repo":"VamosC/CLIP4STR","repo_kind":"named_in_paper","path":"strhub/models/post_process.py","file_url":"https://github.com/VamosC/CLIP4STR/blob/HEAD/strhub/models/post_process.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ec0f91c34fb1b379"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}