{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/end-to-end-concept-word-detection-for-video","title":"End-to-end Concept Word Detection for Video Captioning, Retrieval, and Question Answering","arxiv_id":"1610.02947","date":"2016-10-10","proceeding":"CVPR 2017 7","authors":["Youngjae Yu","Hyungjin Ko","Jongwook Choi","Gunhee Kim"],"abstract":"We propose a high-level concept word detector that can be integrated with any\nvideo-to-language models. It takes a video as input and generates a list of\nconcept words as useful semantic priors for language generation models. The\nproposed word detector has two important properties. First, it does not require\nany external knowledge sources for training. Second, the proposed word detector\nis trainable in an end-to-end manner jointly with any video-to-language models.\nTo maximize the values of detected words, we also develop a semantic attention\nmechanism that selectively focuses on the detected concept words and fuse them\nwith the word encoding and decoding in the language model. In order to\ndemonstrate that the proposed approach indeed improves the performance of\nmultiple video-to-language tasks, we participate in four tasks of LSMDC 2016.\nOur approach achieves the best accuracies in three of them, including\nfill-in-the-blank, multiple-choice test, and movie retrieval. We also attain\ncomparable performance for the other task, movie description.","url_abs":"http://arxiv.org/abs/1610.02947v3","url_pdf":"http://arxiv.org/pdf/1610.02947v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"multiple-choice","task_name":"Multiple-choice"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"video-captioning","task_name":"Video Captioning"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-retrieval-on-lsmdc","task":"Video Retrieval","dataset":"LSMDC","model":"CT-SAN","rank_in_archive_order":37,"of":38,"metrics":{"text-to-video Median Rank":"46","text-to-video R@1":"5.1","text-to-video R@10":"25.2","text-to-video R@5":"16.3"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1610.02947","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}