{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/skysensegpt-a-fine-grained-instruction-tuning","title":"SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understanding","arxiv_id":"2406.10100","date":"2024-06-14","proceeding":null,"authors":["Junwei Luo","Zhen Pang","Yongjun Zhang","Tingzhu Wang","LinLin Wang","Bo Dang","Jiangwei Lao","Jian Wang","Jingdong Chen","Yihua Tan","Yansheng Li"],"abstract":"Remote Sensing Large Multi-Modal Models (RSLMMs) are developing rapidly and showcase significant capabilities in remote sensing imagery (RSI) comprehension. However, due to the limitations of existing datasets, RSLMMs have shortcomings in understanding the rich semantic relations among objects in complex remote sensing scenes. To unlock RSLMMs' complex comprehension ability, we propose a large-scale instruction tuning dataset FIT-RS, containing 1,800,851 instruction samples. FIT-RS covers common interpretation tasks and innovatively introduces several complex comprehension tasks of escalating difficulty, ranging from relation reasoning to image-level scene graph generation. Based on FIT-RS, we build the FIT-RSFG benchmark. Furthermore, we establish a new benchmark to evaluate the fine-grained relation comprehension capabilities of LMMs, named FIT-RSRC. Based on combined instruction data, we propose SkySenseGPT, which achieves outstanding performance on both public datasets and FIT-RSFG, surpassing existing RSLMMs. We hope the FIT-RS dataset can enhance the relation comprehension capability of RSLMMs and provide a large-scale fine-grained data source for the remote sensing community. The dataset will be available at https://github.com/Luo-Z13/SkySenseGPT","url_abs":"https://arxiv.org/abs/2406.10100v2","url_pdf":"https://arxiv.org/pdf/2406.10100v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"skysensegpt-a-fine-grained-instruction-tuning","repo_url":"https://github.com/luo-z13/skysensegpt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"graph-generation","task_name":"Graph Generation"},{"task_slug":null,"task_name":"Relation"},{"task_slug":"scene-graph-generation","task_name":"Scene Graph Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.10100","atlas_url":"https://app.syntology.ai/?focus=2406.10100","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.10100"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/luo-z13/skysensegpt","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":4,"ran_draft_wrong":2},"by_repo_kind":{"official":{"samples":6,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"dca3ae95abe2adc0","entry":"calculate_precision_recall","repo":"luo-z13/skysensegpt","repo_kind":"official","path":"Eval_scripts/batch_geochat_sceneclassification.py","file_url":"https://github.com/luo-z13/skysensegpt/blob/HEAD/Eval_scripts/batch_geochat_sceneclassification.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dca3ae95abe2adc0"}},{"code_sha256_prefix":"1b9e414193635510","entry":"extract_choice_label","repo":"luo-z13/skysensegpt","repo_kind":"official","path":"Eval_scripts/batch_fitrsrc_single_choice_qa.py","file_url":"https://github.com/luo-z13/skysensegpt/blob/HEAD/Eval_scripts/batch_fitrsrc_single_choice_qa.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1b9e414193635510"}},{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"luo-z13/skysensegpt","repo_kind":"official","path":"Eval_scripts/batch_fitrsrc_single_choice_qa.py","file_url":"https://github.com/luo-z13/skysensegpt/blob/HEAD/Eval_scripts/batch_fitrsrc_single_choice_qa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"86f3401508de9387","entry":"obb2poly_np_oc","repo":"luo-z13/skysensegpt","repo_kind":"official","path":"Eval_scripts/batch_geochat_complex_compre.py","file_url":"https://github.com/luo-z13/skysensegpt/blob/HEAD/Eval_scripts/batch_geochat_complex_compre.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"86f3401508de9387"}},{"code_sha256_prefix":"719d4e42eca1def2","entry":"parse_model_name","repo":"luo-z13/skysensegpt","repo_kind":"official","path":"Eval_scripts/langperf_imgcaption.py","file_url":"https://github.com/luo-z13/skysensegpt/blob/HEAD/Eval_scripts/langperf_imgcaption.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"719d4e42eca1def2"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"luo-z13/skysensegpt","repo_kind":"official","path":"Eval_scripts/batch_fitrsrc_single_choice_qa.py","file_url":"https://github.com/luo-z13/skysensegpt/blob/HEAD/Eval_scripts/batch_fitrsrc_single_choice_qa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}