{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-generative-vision-transformer-with-1","title":"Learning Generative Vision Transformer with Energy-Based Latent Space for Saliency Prediction","arxiv_id":"2112.13528","date":"2021-12-27","proceeding":"NeurIPS 2021 12","authors":["Jing Zhang","Jianwen Xie","Nick Barnes","Ping Li"],"abstract":"Vision transformer networks have shown superiority in many computer vision tasks. In this paper, we take a step further by proposing a novel generative vision transformer with latent variables following an informative energy-based prior for salient object detection. Both the vision transformer network and the energy-based prior model are jointly trained via Markov chain Monte Carlo-based maximum likelihood estimation, in which the sampling from the intractable posterior and prior distributions of the latent variables are performed by Langevin dynamics. Further, with the generative vision transformer, we can easily obtain a pixel-wise uncertainty map from an image, which indicates the model confidence in predicting saliency from the image. Different from the existing generative models which define the prior distribution of the latent variables as a simple isotropic Gaussian distribution, our model uses an energy-based informative prior which can be more expressive to capture the latent space of the data. We apply the proposed framework to both RGB and RGB-D salient object detection tasks. Extensive experimental results show that our framework can achieve not only accurate saliency predictions but also meaningful uncertainty maps that are consistent with the human perception.","url_abs":"https://arxiv.org/abs/2112.13528v1","url_pdf":"https://arxiv.org/pdf/2112.13528v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"rgb-d-salient-object-detection","task_name":"RGB-D Salient Object Detection"},{"task_slug":"saliency-prediction","task_name":"Saliency Prediction"},{"task_slug":"salient-object-detection-1","task_name":"Salient Object Detection"},{"task_slug":"thermal-image-segmentation","task_name":"Thermal Image Segmentation"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/thermal-image-segmentation-on-rgb-t-glass","task":"Thermal Image Segmentation","dataset":"RGB-T-Glass-Segmentation","model":"EBS","rank_in_archive_order":4,"of":22,"metrics":{"MAE":"0.040"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2112.13528","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}