{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multimodal-convolutional-neural-networks-for","title":"Multimodal Convolutional Neural Networks for Matching Image and Sentence","arxiv_id":"1504.06063","date":"2015-04-23","proceeding":"ICCV 2015 12","authors":["Lin Ma","Zhengdong Lu","Lifeng Shang","Hang Li"],"abstract":"In this paper, we propose multimodal convolutional neural networks (m-CNNs)\nfor matching image and sentence. Our m-CNN provides an end-to-end framework\nwith convolutional architectures to exploit image representation, word\ncomposition, and the matching relations between the two modalities. More\nspecifically, it consists of one image CNN encoding the image content, and one\nmatching CNN learning the joint representation of image and sentence. The\nmatching CNN composes words to different semantic fragments and learns the\ninter-modal relations between image and the composed fragments at different\nlevels, thus fully exploit the matching relations between image and sentence.\nExperimental results on benchmark databases of bidirectional image and sentence\nretrieval demonstrate that the proposed m-CNNs can effectively capture the\ninformation necessary for image and sentence matching. Specifically, our\nproposed m-CNNs for bidirectional image and sentence retrieval on Flickr30K and\nMicrosoft COCO databases achieve the state-of-the-art performances.","url_abs":"http://arxiv.org/abs/1504.06063v5","url_pdf":"http://arxiv.org/pdf/1504.06063v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"multimodal-convolutional-neural-networks-for","repo_url":"https://github.com/2281123066/visual-semantic-embedding","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"multimodal-convolutional-neural-networks-for","repo_url":"https://github.com/ryankiros/visual-semantic-embedding","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"multimodal-convolutional-neural-networks-for","repo_url":"https://github.com/2023-MindSpore-1/ms-code-18/tree/main/MCNN","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"sentence-retrieval","task_name":"Sentence Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-retrieval-on-flickr30k-1k-test","task":"Image Retrieval","dataset":"Flickr30K 1K test","model":"mCNN","rank_in_archive_order":16,"of":18,"metrics":{"R@1":"26.2","R@10":"69.6","R@5":"56.3"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/1504.06063","atlas_url":"https://app.syntology.ai/?focus=1504.06063","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}