{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/skyscript-a-large-and-semantically-diverse","title":"SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing","arxiv_id":"2312.12856","date":"2023-12-20","proceeding":null,"authors":["Zhecheng Wang","Rajanie Prabha","Tianyuan Huang","Jiajun Wu","Ram Rajagopal"],"abstract":"Remote sensing imagery, despite its broad applications in helping achieve Sustainable Development Goals and tackle climate change, has not yet benefited from the recent advancements of versatile, task-agnostic vision language models (VLMs). A key reason is that the large-scale, semantically diverse image-text dataset required for developing VLMs is still absent for remote sensing images. Unlike natural images, remote sensing images and their associated text descriptions cannot be efficiently collected from the public Internet at scale. In this work, we bridge this gap by using geo-coordinates to automatically connect open, unlabeled remote sensing images with rich semantics covered in OpenStreetMap, and thus construct SkyScript, a comprehensive vision-language dataset for remote sensing images, comprising 2.6 million image-text pairs covering 29K distinct semantic tags. With continual pre-training on this dataset, we obtain a VLM that surpasses baseline models with a 6.2% average accuracy gain in zero-shot scene classification across seven benchmark datasets. It also demonstrates the ability of zero-shot transfer for fine-grained object attribute classification and cross-modal retrieval. We hope this dataset can support the advancement of VLMs for various multi-modal tasks in remote sensing, such as open-vocabulary classification, retrieval, captioning, and text-to-image synthesis.","url_abs":"https://arxiv.org/abs/2312.12856v1","url_pdf":"https://arxiv.org/pdf/2312.12856v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"skyscript-a-large-and-semantically-diverse","repo_url":"https://github.com/wangzhecheng/skyscript","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"attribute","task_name":"Attribute"},{"task_slug":"cross-modal-retrieval","task_name":"Cross-Modal Retrieval"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"scene-classification","task_name":"Scene Classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.12856","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.12856"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wangzhecheng/skyscript","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_violates":1,"ran_draft_wrong":2,"ran":1,"unverified":7},"by_repo_kind":{"official":{"samples":11,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f19962ebb134b3d7","entry":"convert_to_custom_text_state_dict","repo":"wangzhecheng/skyscript","repo_kind":"official","path":"src/open_clip/model.py","file_url":"https://github.com/wangzhecheng/skyscript/blob/HEAD/src/open_clip/model.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f19962ebb134b3d7"}},{"code_sha256_prefix":"dcd422d66b0581d8","entry":"get_cast_dtype","repo":"wangzhecheng/skyscript","repo_kind":"official","path":"src/open_clip/model.py","file_url":"https://github.com/wangzhecheng/skyscript/blob/HEAD/src/open_clip/model.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dcd422d66b0581d8"}},{"code_sha256_prefix":"b476c8cfbf0f1f47","entry":"get_input_dtype","repo":"wangzhecheng/skyscript","repo_kind":"official","path":"src/open_clip/model.py","file_url":"https://github.com/wangzhecheng/skyscript/blob/HEAD/src/open_clip/model.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b476c8cfbf0f1f47"}},{"code_sha256_prefix":"4b3a4d17e1df3140","entry":"load_state_dict","repo":"wangzhecheng/skyscript","repo_kind":"official","path":"src/open_clip/factory.py","file_url":"https://github.com/wangzhecheng/skyscript/blob/HEAD/src/open_clip/factory.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4b3a4d17e1df3140"}},{"code_sha256_prefix":"ddcbd45e940484ee","entry":"gather_features","repo":"wangzhecheng/skyscript","repo_kind":"official","path":"src/open_clip/loss.py","file_url":"https://github.com/wangzhecheng/skyscript/blob/HEAD/src/open_clip/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ddcbd45e940484ee"}},{"code_sha256_prefix":"575ba28efe232215","entry":"get_default_params","repo":"wangzhecheng/skyscript","repo_kind":"official","path":"params.py","file_url":"https://github.com/wangzhecheng/skyscript/blob/HEAD/params.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"575ba28efe232215"}},{"code_sha256_prefix":"5c7cf820e99443ff","entry":"get_model_config","repo":"wangzhecheng/skyscript","repo_kind":"official","path":"src/open_clip/factory.py","file_url":"https://github.com/wangzhecheng/skyscript/blob/HEAD/src/open_clip/factory.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5c7cf820e99443ff"}},{"code_sha256_prefix":"f34d8f34635af394","entry":"get_tokenizer","repo":"wangzhecheng/skyscript","repo_kind":"official","path":"src/open_clip/factory.py","file_url":"https://github.com/wangzhecheng/skyscript/blob/HEAD/src/open_clip/factory.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f34d8f34635af394"}},{"code_sha256_prefix":"95bef696bff75a54","entry":"parse_args","repo":"wangzhecheng/skyscript","repo_kind":"official","path":"params.py","file_url":"https://github.com/wangzhecheng/skyscript/blob/HEAD/params.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"95bef696bff75a54"}},{"code_sha256_prefix":"fb651d0a97fd4d3f","entry":"prepare_inputs_for_generation","repo":"wangzhecheng/skyscript","repo_kind":"official","path":"src/open_clip/coca_model.py","file_url":"https://github.com/wangzhecheng/skyscript/blob/HEAD/src/open_clip/coca_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fb651d0a97fd4d3f"}},{"code_sha256_prefix":"2a377da4a76a2d44","entry":"register_pooler","repo":"wangzhecheng/skyscript","repo_kind":"official","path":"src/open_clip/hf_model.py","file_url":"https://github.com/wangzhecheng/skyscript/blob/HEAD/src/open_clip/hf_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2a377da4a76a2d44"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}