{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/paint-transformer-feed-forward-neural","title":"Paint Transformer: Feed Forward Neural Painting with Stroke Prediction","arxiv_id":"2108.03798","date":"2021-08-09","proceeding":"ICCV 2021 10","authors":["Songhua Liu","Tianwei Lin","Dongliang He","Fu Li","Ruifeng Deng","Xin Li","Errui Ding","Hao Wang"],"abstract":"Neural painting refers to the procedure of producing a series of strokes for a given image and non-photo-realistically recreating it using neural networks. While reinforcement learning (RL) based agents can generate a stroke sequence step by step for this task, it is not easy to train a stable RL agent. On the other hand, stroke optimization methods search for a set of stroke parameters iteratively in a large search space; such low efficiency significantly limits their prevalence and practicality. Different from previous methods, in this paper, we formulate the task as a set prediction problem and propose a novel Transformer-based framework, dubbed Paint Transformer, to predict the parameters of a stroke set with a feed forward network. This way, our model can generate a set of strokes in parallel and obtain the final painting of size 512 * 512 in near real time. More importantly, since there is no dataset available for training the Paint Transformer, we devise a self-training pipeline such that it can be trained without any off-the-shelf dataset while still achieving excellent generalization capability. Experiments demonstrate that our method achieves better painting performance than previous ones with cheaper training and inference costs. Codes and models are available.","url_abs":"https://arxiv.org/abs/2108.03798v2","url_pdf":"https://arxiv.org/pdf/2108.03798v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"paint-transformer-feed-forward-neural","repo_url":"https://github.com/huage001/painttransformer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"paint-transformer-feed-forward-neural","repo_url":"https://github.com/wzmsltw/painttransformer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"paddle","reach":null}],"tasks":[{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"prediction","task_name":"Prediction"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"style-transfer","task_name":"Style Transfer"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/object-detection-on-a2d","task":"Object Detection","dataset":"A2D","model":"RL [10] Lpixel","rank_in_archive_order":1,"of":1,"metrics":{"Mean IoU":"5.8"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-on-coco-2017","task":"Object Detection","dataset":"COCO 2017","model":"Lpixel","rank_in_archive_order":24,"of":24,"metrics":{"Mean mAP":"4.2"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-on-sixray","task":"Object Detection","dataset":"SIXray","model":"Optim [39] Lpixel","rank_in_archive_order":2,"of":2,"metrics":{"1 in 10 R@5":"0.073"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2108.03798","atlas_url":"https://app.syntology.ai/?focus=2108.03798","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}