{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mars-paying-more-attention-to-visual","title":"MARS: Paying more attention to visual attributes for text-based person search","arxiv_id":"2407.04287","date":"2024-07-05","proceeding":null,"authors":["Alex Ergasti","Tomaso Fontanini","Claudio Ferrari","Massimo Bertozzi","Andrea Prati"],"abstract":"Text-based person search (TBPS) is a problem that gained significant interest within the research community. The task is that of retrieving one or more images of a specific individual based on a textual description. The multi-modal nature of the task requires learning representations that bridge text and image data within a shared latent space. Existing TBPS systems face two major challenges. One is defined as inter-identity noise that is due to the inherent vagueness and imprecision of text descriptions and it indicates how descriptions of visual attributes can be generally associated to different people; the other is the intra-identity variations, which are all those nuisances e.g. pose, illumination, that can alter the visual appearance of the same textual attributes for a given subject. To address these issues, this paper presents a novel TBPS architecture named MARS (Mae-Attribute-Relation-Sensitive), which enhances current state-of-the-art models by introducing two key components: a Visual Reconstruction Loss and an Attribute Loss. The former employs a Masked AutoEncoder trained to reconstruct randomly masked image patches with the aid of the textual description. In doing so the model is encouraged to learn more expressive representations and textual-visual relations in the latent space. The Attribute Loss, instead, balances the contribution of different types of attributes, defined as adjective-noun chunks of text. This loss ensures that every attribute is taken into consideration in the person retrieval process. Extensive experiments on three commonly used datasets, namely CUHK-PEDES, ICFG-PEDES, and RSTPReid, report performance improvements, with significant gains in the mean Average Precision (mAP) metric w.r.t. the current state of the art.","url_abs":"https://arxiv.org/abs/2407.04287v1","url_pdf":"https://arxiv.org/pdf/2407.04287v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mars-paying-more-attention-to-visual","repo_url":"https://github.com/ergastialex/mars","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"attribute","task_name":"Attribute"},{"task_slug":"person-re-identification","task_name":"Person Re-Identification"},{"task_slug":"person-retrieval","task_name":"Person Retrieval"},{"task_slug":"person-search","task_name":"Person Search"},{"task_slug":"nlp-based-person-retrival","task_name":"Text based Person Retrieval"},{"task_slug":"text-based-person-search","task_name":"Text based Person Search"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/nlp-based-person-retrival-on-cuhk-pedes","task":"Text based Person Retrieval","dataset":"CUHK-PEDES","model":"MARS","rank_in_archive_order":1,"of":21,"metrics":{"R@1":"77.62","R@10":"94.27","R@5":"90.63","mAP":"71.41"},"uses_additional_data":false},{"leaderboard":"/sota/text-based-person-retrieval-on-icfg-pedes","task":"Text based Person Retrieval","dataset":"ICFG-PEDES","model":"MARS","rank_in_archive_order":4,"of":12,"metrics":{"R@1":"67.60","R@10":"85.79","R@5":"81.47","mAP":"44.93"},"uses_additional_data":false},{"leaderboard":"/sota/text-based-person-retrieval-on-rstpreid-1","task":"Text based Person Retrieval","dataset":"RSTPReid","model":"MARS","rank_in_archive_order":1,"of":9,"metrics":{"R@1":"67.55","R@10":"91.35","R@5":"86.65","mAP":"52.92"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.04287","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}