{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/end-to-end-dense-video-captioning-with-masked","title":"End-to-End Dense Video Captioning with Masked Transformer","arxiv_id":"1804.00819","date":"2018-04-03","proceeding":"CVPR 2018 4","authors":["Luowei Zhou","Yingbo Zhou","Jason J. Corso","Richard Socher","Caiming Xiong"],"abstract":"Dense video captioning aims to generate text descriptions for all events in\nan untrimmed video. This involves both detecting and describing events.\nTherefore, all previous methods on dense video captioning tackle this problem\nby building two models, i.e. an event proposal and a captioning model, for\nthese two sub-problems. The models are either trained separately or in\nalternation. This prevents direct influence of the language description to the\nevent proposal, which is important for generating accurate descriptions. To\naddress this problem, we propose an end-to-end transformer model for dense\nvideo captioning. The encoder encodes the video into appropriate\nrepresentations. The proposal decoder decodes from the encoding with different\nanchors to form video event proposals. The captioning decoder employs a masking\nnetwork to restrict its attention to the proposal event over the encoding\nfeature. This masking network converts the event proposal to a differentiable\nmask, which ensures the consistency between the proposal and captioning during\ntraining. In addition, our model employs a self-attention mechanism, which\nenables the use of efficient non-recurrent structure during encoding and leads\nto performance improvements. We demonstrate the effectiveness of this\nend-to-end model on ActivityNet Captions and YouCookII datasets, where we\nachieved 10.12 and 6.58 METEOR score, respectively.","url_abs":"http://arxiv.org/abs/1804.00819v1","url_pdf":"http://arxiv.org/pdf/1804.00819v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"end-to-end-dense-video-captioning-with-masked","repo_url":"https://github.com/salesforce/densecap","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"dense-video-captioning","task_name":"Dense Video Captioning"},{"task_slug":"video-captioning","task_name":"Video Captioning"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-captioning-on-youcook2","task":"Video Captioning","dataset":"YouCook2","model":"Zhou","rank_in_archive_order":12,"of":14,"metrics":{"BLEU-3":"7.53","BLEU-4":"4.38","CIDEr":"0.38","METEOR":"11.55","ROUGE-L":"27.44"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1804.00819","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}