{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/instructcv-instruction-tuned-text-to-image","title":"InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists","arxiv_id":"2310.00390","date":"2023-09-30","proceeding":null,"authors":["Yulu Gan","Sungwoo Park","Alexander Schubert","Anthony Philippakis","Ahmed M. Alaa"],"abstract":"Recent advances in generative diffusion models have enabled text-controlled synthesis of realistic and diverse images with impressive quality. Despite these remarkable advances, the application of text-to-image generative models in computer vision for standard visual recognition tasks remains limited. The current de facto approach for these tasks is to design model architectures and loss functions that are tailored to the task at hand. In this paper, we develop a unified language interface for computer vision tasks that abstracts away task-specific design choices and enables task execution by following natural language instructions. Our approach involves casting multiple computer vision tasks as text-to-image generation problems. Here, the text represents an instruction describing the task, and the resulting image is a visually-encoded task output. To train our model, we pool commonly-used computer vision datasets covering a range of tasks, including segmentation, object detection, depth estimation, and classification. We then use a large language model to paraphrase prompt templates that convey the specific tasks to be conducted on each image, and through this process, we create a multi-modal and multi-task training dataset comprising input and output images along with annotated instructions. Following the InstructPix2Pix architecture, we apply instruction-tuning to a text-to-image diffusion model using our constructed dataset, steering its functionality from a generative model to an instruction-guided multi-task vision learner. Experiments demonstrate that our model, dubbed InstructCV, performs competitively compared to other generalist and task-specific vision models. Moreover, it exhibits compelling generalization capabilities to unseen data, categories, and user instructions.","url_abs":"https://arxiv.org/abs/2310.00390v3","url_pdf":"https://arxiv.org/pdf/2310.00390v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"instructcv-instruction-tuned-text-to-image","repo_url":"https://github.com/AlaaLab/InstructCV","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"depth-estimation","task_name":"Depth Estimation"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"text-to-image-generation-1","task_name":"Text to Image Generation"},{"task_slug":"text-to-image-generation","task_name":"Text-to-Image Generation"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.00390","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.00390"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/AlaaLab/InstructCV","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":2,"unverified":10},"by_repo_kind":{"official":{"samples":12,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"1a20155f648328f5","entry":"iou","repo":"AlaaLab/InstructCV","repo_kind":"official","path":"evaluate/evaluate_cls_seg_det.py","file_url":"https://github.com/AlaaLab/InstructCV/blob/HEAD/evaluate/evaluate_cls_seg_det.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1a20155f648328f5"}},{"code_sha256_prefix":"30d263a2a2ae4188","entry":"read_pred_img","repo":"AlaaLab/InstructCV","repo_kind":"official","path":"evaluate/evaluate_depes.py","file_url":"https://github.com/AlaaLab/InstructCV/blob/HEAD/evaluate/evaluate_depes.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"30d263a2a2ae4188"}},{"code_sha256_prefix":"b0f21fc9a1aa3b08","entry":"absolute_mean_relative_error","repo":"AlaaLab/InstructCV","repo_kind":"official","path":"evaluate/evaluate_depes.py","file_url":"https://github.com/AlaaLab/InstructCV/blob/HEAD/evaluate/evaluate_depes.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b0f21fc9a1aa3b08"}},{"code_sha256_prefix":"120433ec9c58b66e","entry":"cal_bboxes_iou","repo":"AlaaLab/InstructCV","repo_kind":"official","path":"evaluate/evaluate_cls_seg_det.py","file_url":"https://github.com/AlaaLab/InstructCV/blob/HEAD/evaluate/evaluate_cls_seg_det.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"120433ec9c58b66e"}},{"code_sha256_prefix":"eeb478400c75c259","entry":"calc_iou","repo":"AlaaLab/InstructCV","repo_kind":"official","path":"evaluate/evaluate_cls_seg_det.py","file_url":"https://github.com/AlaaLab/InstructCV/blob/HEAD/evaluate/evaluate_cls_seg_det.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"eeb478400c75c259"}},{"code_sha256_prefix":"4155a5233179db09","entry":"get_bbox_prompt","repo":"AlaaLab/InstructCV","repo_kind":"official","path":"build_data/format_dataset_fp.py","file_url":"https://github.com/AlaaLab/InstructCV/blob/HEAD/build_data/format_dataset_fp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4155a5233179db09"}},{"code_sha256_prefix":"3647dbec915093fa","entry":"get_bbox_prompt","repo":"AlaaLab/InstructCV","repo_kind":"official","path":"build_data/format_dataset_rp.py","file_url":"https://github.com/AlaaLab/InstructCV/blob/HEAD/build_data/format_dataset_rp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"3647dbec915093fa"}},{"code_sha256_prefix":"4e8ac6d87ea2d135","entry":"get_cls_prompt","repo":"AlaaLab/InstructCV","repo_kind":"official","path":"build_data/format_dataset_fp.py","file_url":"https://github.com/AlaaLab/InstructCV/blob/HEAD/build_data/format_dataset_fp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4e8ac6d87ea2d135"}},{"code_sha256_prefix":"e9016d81762a6bb7","entry":"get_cls_prompt","repo":"AlaaLab/InstructCV","repo_kind":"official","path":"build_data/format_dataset_rp.py","file_url":"https://github.com/AlaaLab/InstructCV/blob/HEAD/build_data/format_dataset_rp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e9016d81762a6bb7"}},{"code_sha256_prefix":"0d7e7a5e4d33be60","entry":"get_seg_prompt","repo":"AlaaLab/InstructCV","repo_kind":"official","path":"build_data/format_dataset_fp.py","file_url":"https://github.com/AlaaLab/InstructCV/blob/HEAD/build_data/format_dataset_fp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0d7e7a5e4d33be60"}},{"code_sha256_prefix":"b544ea6180a53824","entry":"get_seg_prompt","repo":"AlaaLab/InstructCV","repo_kind":"official","path":"build_data/format_dataset_rp.py","file_url":"https://github.com/AlaaLab/InstructCV/blob/HEAD/build_data/format_dataset_rp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b544ea6180a53824"}},{"code_sha256_prefix":"523afd8adffe17ce","entry":"read_gt_img","repo":"AlaaLab/InstructCV","repo_kind":"official","path":"evaluate/evaluate_depes.py","file_url":"https://github.com/AlaaLab/InstructCV/blob/HEAD/evaluate/evaluate_depes.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"523afd8adffe17ce"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}