{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-all-seeing-project-v2-towards-general","title":"The All-Seeing Project V2: Towards General Relation Comprehension of the Open World","arxiv_id":"2402.19474","date":"2024-02-29","proceeding":null,"authors":["Weiyun Wang","Yiming Ren","Haowen Luo","Tiantong Li","Chenxiang Yan","Zhe Chen","Wenhai Wang","Qingyun Li","Lewei Lu","Xizhou Zhu","Yu Qiao","Jifeng Dai"],"abstract":"We present the All-Seeing Project V2: a new model and dataset designed for understanding object relations in images. Specifically, we propose the All-Seeing Model V2 (ASMv2) that integrates the formulation of text generation, object localization, and relation comprehension into a relation conversation (ReC) task. Leveraging this unified task, our model excels not only in perceiving and recognizing all objects within the image but also in grasping the intricate relation graph between them, diminishing the relation hallucination often encountered by Multi-modal Large Language Models (MLLMs). To facilitate training and evaluation of MLLMs in relation understanding, we created the first high-quality ReC dataset ({AS-V2) which is aligned with the format of standard instruction tuning data. In addition, we design a new benchmark, termed Circular-based Relation Probing Evaluation (CRPE) for comprehensively evaluating the relation comprehension capabilities of MLLMs. Notably, our ASMv2 achieves an overall accuracy of 52.04 on this relation-aware benchmark, surpassing the 43.14 of LLaVA-1.5 by a large margin. We hope that our work can inspire more future research and contribute to the evolution towards artificial general intelligence. Our project is released at https://github.com/OpenGVLab/all-seeing.","url_abs":"https://arxiv.org/abs/2402.19474v4","url_pdf":"https://arxiv.org/pdf/2402.19474v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-all-seeing-project-v2-towards-general","repo_url":"https://github.com/opengvlab/all-seeing","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"all","task_name":"All"},{"task_slug":"hallucination","task_name":"Hallucination"},{"task_slug":"object-localization","task_name":"Object Localization"},{"task_slug":null,"task_name":"Relation"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"}],"methods":[],"datasets_introduced":[{"slug":"as-v2","name":"AS-V2","full_name":"The All-Seeing Dataset v2"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-question-answering-on-mm-vet","task":"Visual Question Answering","dataset":"MM-Vet","model":"ASMv2","rank_in_archive_order":103,"of":231,"metrics":{"GPT-4 score":"41.3"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2402.19474","atlas_url":"https://app.syntology.ai/?focus=2402.19474","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.19474"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/opengvlab/all-seeing","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/OpenGVLab/all-seeing","reach":{"status":"ok"}}],"summary":{"ran":3,"ran_honours":2,"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"bbe645c8f76bf19e","entry":"AllSeeingModelConfig","repo":"OpenGVLab/all-seeing","repo_kind":"official","path":"all-seeing/utils/model.py","file_url":"https://github.com/OpenGVLab/all-seeing/blob/HEAD/all-seeing/utils/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bbe645c8f76bf19e"}},{"code_sha256_prefix":"586d698ae8b94c92","entry":"Blip2ForConditionalGenerationModelOutput","repo":"OpenGVLab/all-seeing","repo_kind":"official","path":"all-seeing/utils/model.py","file_url":"https://github.com/OpenGVLab/all-seeing/blob/HEAD/all-seeing/utils/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"586d698ae8b94c92"}},{"code_sha256_prefix":"090af570926107cc","entry":"concat_all_gather","repo":"OpenGVLab/all-seeing","repo_kind":"official","path":"all-seeing/utils/model.py","file_url":"https://github.com/OpenGVLab/all-seeing/blob/HEAD/all-seeing/utils/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"090af570926107cc"}},{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"opengvlab/all-seeing","repo_kind":"official","path":"all-seeing-v2/llava/eval/model_vqa.py","file_url":"https://github.com/opengvlab/all-seeing/blob/HEAD/all-seeing-v2/llava/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"5f205488cb7b9004","entry":"is_dist_avail_and_initialized","repo":"OpenGVLab/all-seeing","repo_kind":"official","path":"all-seeing/utils/model.py","file_url":"https://github.com/OpenGVLab/all-seeing/blob/HEAD/all-seeing/utils/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5f205488cb7b9004"}},{"code_sha256_prefix":"9b3c1cb391672ccb","entry":"load_image","repo":"opengvlab/all-seeing","repo_kind":"official","path":"all-seeing-v2/predict.py","file_url":"https://github.com/opengvlab/all-seeing/blob/HEAD/all-seeing-v2/predict.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9b3c1cb391672ccb"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"opengvlab/all-seeing","repo_kind":"official","path":"all-seeing-v2/llava/eval/model_vqa.py","file_url":"https://github.com/opengvlab/all-seeing/blob/HEAD/all-seeing-v2/llava/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}},{"code_sha256_prefix":"2a96dcf5d12b3da6","entry":"AllSeeingModel","repo":"OpenGVLab/all-seeing","repo_kind":"official","path":"all-seeing/utils/model.py","file_url":"https://github.com/OpenGVLab/all-seeing/blob/HEAD/all-seeing/utils/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2a96dcf5d12b3da6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}