{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/amc-attention-guided-multi-modal-correlation","title":"AMC: Attention guided Multi-modal Correlation Learning for Image Search","arxiv_id":"1704.00763","date":"2017-04-03","proceeding":"CVPR 2017 7","authors":["Kan Chen","Trung Bui","Fang Chen","Zhaowen Wang","Ram Nevatia"],"abstract":"Given a user's query, traditional image search systems rank images according\nto its relevance to a single modality (e.g., image content or surrounding\ntext). Nowadays, an increasing number of images on the Internet are available\nwith associated meta data in rich modalities (e.g., titles, keywords, tags,\netc.), which can be exploited for better similarity measure with queries. In\nthis paper, we leverage visual and textual modalities for image search by\nlearning their correlation with input query. According to the intent of query,\nattention mechanism can be introduced to adaptively balance the importance of\ndifferent modalities. We propose a novel Attention guided Multi-modal\nCorrelation (AMC) learning method which consists of a jointly learned hierarchy\nof intra and inter-attention networks. Conditioned on query's intent,\nintra-attention networks (i.e., visual intra-attention network and language\nintra-attention network) attend on informative parts within each modality; a\nmulti-modal inter-attention network promotes the importance of the most\nquery-relevant modalities. In experiments, we evaluate AMC models on the search\nlogs from two real world image search engines and show a significant boost on\nthe ranking of user-clicked images in search results. Additionally, we extend\nAMC models to caption ranking task on COCO dataset and achieve competitive\nresults compared with recent state-of-the-arts.","url_abs":"http://arxiv.org/abs/1704.00763v1","url_pdf":"http://arxiv.org/pdf/1704.00763v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"amc-attention-guided-multi-modal-correlation","repo_url":"https://github.com/kanchen-usc/amc_att","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"amc-attention-guided-multi-modal-correlation","repo_url":"https://github.com/zxy14120448/Summary","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-retrieval","task_name":"Image Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1704.00763","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}