{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/2408-00491","title":"GalleryGPT: Analyzing Paintings with Large Multimodal Models","arxiv_id":"2408.00491","date":"2024-08-01","proceeding":null,"authors":["Yi Bin","Wenhao Shi","Yujuan Ding","Zhiqiang Hu","Zheng Wang","Yang Yang","See-Kiong Ng","Heng Tao Shen"],"abstract":"Artwork analysis is important and fundamental skill for art appreciation, which could enrich personal aesthetic sensibility and facilitate the critical thinking ability. Understanding artworks is challenging due to its subjective nature, diverse interpretations, and complex visual elements, requiring expertise in art history, cultural background, and aesthetic theory. However, limited by the data collection and model ability, previous works for automatically analyzing artworks mainly focus on classification, retrieval, and other simple tasks, which is far from the goal of AI. To facilitate the research progress, in this paper, we step further to compose comprehensive analysis inspired by the remarkable perception and generation ability of large multimodal models. Specifically, we first propose a task of composing paragraph analysis for artworks, i.e., painting in this paper, only focusing on visual characteristics to formulate more comprehensive understanding of artworks. To support the research on formal analysis, we collect a large dataset PaintingForm, with about 19k painting images and 50k analysis paragraphs. We further introduce a superior large multimodal model for painting analysis composing, dubbed GalleryGPT, which is slightly modified and fine-tuned based on LLaVA architecture leveraging our collected data. We conduct formal analysis generation and zero-shot experiments across several datasets to assess the capacity of our model. The results show remarkable performance improvements comparing with powerful baseline LMMs, demonstrating its superb ability of art analysis and generalization. \\textcolor{blue}{The codes and model are available at: https://github.com/steven640pixel/GalleryGPT.","url_abs":"https://arxiv.org/abs/2408.00491v1","url_pdf":"https://arxiv.org/pdf/2408.00491v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"2408-00491","repo_url":"https://github.com/steven640pixel/gallerygpt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"art-analysis","task_name":"Art Analysis"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2408.00491","atlas_url":"https://app.syntology.ai/?focus=2408.00491","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.00491"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/steven640pixel/gallerygpt","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":2,"ran_draft_wrong":1,"ran_honours":2,"ran_fixture":2,"unverified":2},"by_repo_kind":{"official":{"samples":9,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7e03b180fa317c9a","entry":"divide_to_patches","repo":"steven640pixel/gallerygpt","repo_kind":"official","path":"llava/mm_utils.py","file_url":"https://github.com/steven640pixel/gallerygpt/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7e03b180fa317c9a"}},{"code_sha256_prefix":"a7bee88c1c7fd6a3","entry":"image_parser","repo":"steven640pixel/gallerygpt","repo_kind":"official","path":"llava/eval/run_llava.py","file_url":"https://github.com/steven640pixel/gallerygpt/blob/HEAD/llava/eval/run_llava.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a7bee88c1c7fd6a3"}},{"code_sha256_prefix":"9b3c1cb391672ccb","entry":"load_image","repo":"steven640pixel/gallerygpt","repo_kind":"official","path":"llava/eval/run_llava.py","file_url":"https://github.com/steven640pixel/gallerygpt/blob/HEAD/llava/eval/run_llava.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9b3c1cb391672ccb"}},{"code_sha256_prefix":"d5044bdde33c54c2","entry":"load_images","repo":"steven640pixel/gallerygpt","repo_kind":"official","path":"llava/eval/run_llava.py","file_url":"https://github.com/steven640pixel/gallerygpt/blob/HEAD/llava/eval/run_llava.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d5044bdde33c54c2"}},{"code_sha256_prefix":"468eedeba67f1b00","entry":"resize_and_pad_image","repo":"steven640pixel/gallerygpt","repo_kind":"official","path":"llava/mm_utils.py","file_url":"https://github.com/steven640pixel/gallerygpt/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"468eedeba67f1b00"}},{"code_sha256_prefix":"3999ff487573f32c","entry":"select_best_resolution","repo":"steven640pixel/gallerygpt","repo_kind":"official","path":"llava/mm_utils.py","file_url":"https://github.com/steven640pixel/gallerygpt/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3999ff487573f32c"}},{"code_sha256_prefix":"7606525af238fb64","entry":"unpad_image","repo":"steven640pixel/gallerygpt","repo_kind":"official","path":"llava/model/llava_arch.py","file_url":"https://github.com/steven640pixel/gallerygpt/blob/HEAD/llava/model/llava_arch.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7606525af238fb64"}},{"code_sha256_prefix":"37899f22fb191b37","entry":"pretty_print_semaphore","repo":"steven640pixel/gallerygpt","repo_kind":"official","path":"llava/utils.py","file_url":"https://github.com/steven640pixel/gallerygpt/blob/HEAD/llava/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"37899f22fb191b37"}},{"code_sha256_prefix":"f9939a84b9a65279","entry":"violates_moderation","repo":"steven640pixel/gallerygpt","repo_kind":"official","path":"llava/utils.py","file_url":"https://github.com/steven640pixel/gallerygpt/blob/HEAD/llava/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f9939a84b9a65279"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}