{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/video2commonsense-generating-commonsense","title":"Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioning","arxiv_id":"2003.05162","date":"2020-03-11","proceeding":"EMNLP 2020 11","authors":["Zhiyuan Fang","Tejas Gokhale","Pratyay Banerjee","Chitta Baral","Yezhou Yang"],"abstract":"Captioning is a crucial and challenging task for video understanding. In videos that involve active agents such as humans, the agent's actions can bring about myriad changes in the scene. Observable changes such as movements, manipulations, and transformations of the objects in the scene, are reflected in conventional video captioning. Unlike images, actions in videos are also inherently linked to social aspects such as intentions (why the action is taking place), effects (what changes due to the action), and attributes that describe the agent. Thus for video understanding, such as when captioning videos or when answering questions about videos, one must have an understanding of these commonsense aspects. We present the first work on generating commonsense captions directly from videos, to describe latent aspects such as intentions, effects, and attributes. We present a new dataset \"Video-to-Commonsense (V2C)\" that contains $\\sim9k$ videos of human agents performing various actions, annotated with 3 types of commonsense descriptions. Additionally we explore the use of open-ended video-based commonsense question answering (V2C-QA) as a way to enrich our captions. Both the generation task and the QA task can be used to enrich video captions.","url_abs":"https://arxiv.org/abs/2003.05162v4","url_pdf":"https://arxiv.org/pdf/2003.05162v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"video2commonsense-generating-commonsense","repo_url":"https://github.com/jacobswan1/Video2Commonsense","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"video2commonsense-generating-commonsense","repo_url":"https://github.com/yuweijiang/HybridNet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"video-captioning","task_name":"Video Captioning"},{"task_slug":"video-understanding","task_name":"Video Understanding"}],"methods":[],"datasets_introduced":[{"slug":"v2c","name":"V2C","full_name":"Video-to-Commonsense"}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2003.05162","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2003.05162"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yuweijiang/HybridNet","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jacobswan1/Video2Commonsense","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":1,"ran_fixture":1},"by_repo_kind":{"listed":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"385264598477696b","entry":"list_to_sentence","repo":"yuweijiang/HybridNet","repo_kind":"listed","path":"test_cap2cms.py","file_url":"https://github.com/yuweijiang/HybridNet/blob/HEAD/test_cap2cms.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"385264598477696b"}},{"code_sha256_prefix":"f67b1430631af2f9","entry":"pos_emb_generation","repo":"yuweijiang/HybridNet","repo_kind":"listed","path":"test_cap2cms.py","file_url":"https://github.com/yuweijiang/HybridNet/blob/HEAD/test_cap2cms.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f67b1430631af2f9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}