{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/vila-learning-image-aesthetics-from-user","title":"VILA: Learning Image Aesthetics from User Comments with Vision-Language Pretraining","arxiv_id":"2303.14302","date":"2023-03-24","proceeding":"CVPR 2023 1","authors":["Junjie Ke","Keren Ye","Jiahui Yu","Yonghui Wu","Peyman Milanfar","Feng Yang"],"abstract":"Assessing the aesthetics of an image is challenging, as it is influenced by multiple factors including composition, color, style, and high-level semantics. Existing image aesthetic assessment (IAA) methods primarily rely on human-labeled rating scores, which oversimplify the visual aesthetic information that humans perceive. Conversely, user comments offer more comprehensive information and are a more natural way to express human opinions and preferences regarding image aesthetics. In light of this, we propose learning image aesthetics from user comments, and exploring vision-language pretraining methods to learn multimodal aesthetic representations. Specifically, we pretrain an image-text encoder-decoder model with image-comment pairs, using contrastive and generative objectives to learn rich and generic aesthetic semantics without human labels. To efficiently adapt the pretrained model for downstream IAA tasks, we further propose a lightweight rank-based adapter that employs text as an anchor to learn the aesthetic ranking concept. Our results show that our pretrained aesthetic vision-language model outperforms prior works on image aesthetic captioning over the AVA-Captions dataset, and it has powerful zero-shot capability for aesthetic tasks such as zero-shot style classification and zero-shot IAA, surpassing many supervised baselines. With only minimal finetuning parameters using the proposed adapter module, our model achieves state-of-the-art IAA performance over the AVA dataset.","url_abs":"https://arxiv.org/abs/2303.14302v2","url_pdf":"https://arxiv.org/pdf/2303.14302v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"vila-learning-image-aesthetics-from-user","repo_url":"https://github.com/google-research/google-research","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":null},{"paper_slug":"vila-learning-image-aesthetics-from-user","repo_url":"https://github.com/fiabdu/Commonly-Interesting-Images","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"video-quality-assessment","task_name":"Video Quality Assessment"}],"methods":[{"method_slug":"adapter","method_name":"Adapter"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-quality-assessment-on-msu-sr-qa-dataset","task":"Video Quality Assessment","dataset":"MSU SR-QA Dataset","model":"VILA","rank_in_archive_order":46,"of":60,"metrics":{"KLCC":"0.26180","PLCC":"0.28846","SROCC":"0.33728","Type":"NR"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2303.14302","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}