{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/test-of-time-instilling-video-language-models","title":"Test of Time: Instilling Video-Language Models with a Sense of Time","arxiv_id":"2301.02074","date":"2023-01-05","proceeding":"CVPR 2023 1","authors":["Piyush Bagad","Makarand Tapaswi","Cees G. M. Snoek"],"abstract":"Modelling and understanding time remains a challenge in contemporary video understanding models. With language emerging as a key driver towards powerful generalization, it is imperative for foundational video-language models to have a sense of time. In this paper, we consider a specific aspect of temporal understanding: consistency of time order as elicited by before/after relations. We establish that seven existing video-language models struggle to understand even such simple temporal relations. We then question whether it is feasible to equip these foundational models with temporal awareness without re-training them from scratch. Towards this, we propose a temporal adaptation recipe on top of one such model, VideoCLIP, based on post-pretraining on a small amount of video-text data. We conduct a zero-shot evaluation of the adapted models on six datasets for three downstream tasks which require varying degrees of time awareness. We observe encouraging performance gains especially when the task needs higher time awareness. Our work serves as a first step towards probing and instilling a sense of time in existing video-language models without the need for data and compute-intense training from scratch.","url_abs":"https://arxiv.org/abs/2301.02074v2","url_pdf":"https://arxiv.org/pdf/2301.02074v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"test-of-time-instilling-video-language-models","repo_url":"https://github.com/bpiyush/TestOfTime","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"video-understanding","task_name":"Video Understanding"},{"task_slug":"video-text-retrieval","task_name":"Video-Text Retrieval"}],"methods":[],"datasets_introduced":[{"slug":"test-of-time","name":"Test-of-Time","full_name":"Test of Time Synthetic Video Dataset"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-text-retrieval-on-test-of-time","task":"Video-Text Retrieval","dataset":"Test-of-Time","model":"TACT","rank_in_archive_order":3,"of":4,"metrics":{"2-Class Accuracy":"64.4"},"uses_additional_data":true}],"syntology":{"syntology_url":"https://syntology.ai/paper/2301.02074","atlas_url":"https://app.syntology.ai/?focus=2301.02074","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2301.02074"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bpiyush/TestOfTime","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/jayleicn/singularity","reach":null}],"summary":{"ran":9,"ran_fixture":1},"by_repo_kind":{"official":{"samples":9,"ran":9,"repositories":1},"found_in_text":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d9fa3afc6393fd8c","entry":"active_event_description_combiner","repo":"bpiyush/TestOfTime","repo_kind":"official","path":"package/datasets/synthetic.py","file_url":"https://github.com/bpiyush/TestOfTime/blob/HEAD/package/datasets/synthetic.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d9fa3afc6393fd8c"}},{"code_sha256_prefix":"deacd11293b2686f","entry":"average_precision","repo":"bpiyush/TestOfTime","repo_kind":"official","path":"package/evaluators/videoclip_action_retrieval.py","file_url":"https://github.com/bpiyush/TestOfTime/blob/HEAD/package/evaluators/videoclip_action_retrieval.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"deacd11293b2686f"}},{"code_sha256_prefix":"f2d84c0542d17a04","entry":"check_args","repo":"bpiyush/TestOfTime","repo_kind":"official","path":"postpretrain.py","file_url":"https://github.com/bpiyush/TestOfTime/blob/HEAD/postpretrain.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f2d84c0542d17a04"}},{"code_sha256_prefix":"b717eef2896560b3","entry":"compute_metrics","repo":"bpiyush/TestOfTime","repo_kind":"official","path":"package/metrics/retrieval.py","file_url":"https://github.com/bpiyush/TestOfTime/blob/HEAD/package/metrics/retrieval.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b717eef2896560b3"}},{"code_sha256_prefix":"5c70504364bdc3ac","entry":"freeze_required_layers","repo":"bpiyush/TestOfTime","repo_kind":"official","path":"postpretrain.py","file_url":"https://github.com/bpiyush/TestOfTime/blob/HEAD/postpretrain.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5c70504364bdc3ac"}},{"code_sha256_prefix":"f64c56c357505158","entry":"interpolate_temporal_pos_embed","repo":"jayleicn/singularity","repo_kind":"found_in_text","path":"models/utils.py","file_url":"https://github.com/jayleicn/singularity/blob/HEAD/models/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f64c56c357505158"}},{"code_sha256_prefix":"0196aa2f7fc22400","entry":"mean_average_precision","repo":"bpiyush/TestOfTime","repo_kind":"official","path":"package/evaluators/videoclip_action_retrieval.py","file_url":"https://github.com/bpiyush/TestOfTime/blob/HEAD/package/evaluators/videoclip_action_retrieval.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0196aa2f7fc22400"}},{"code_sha256_prefix":"053b2f9dfa05e107","entry":"passive_event_description_combiner","repo":"bpiyush/TestOfTime","repo_kind":"official","path":"package/datasets/synthetic.py","file_url":"https://github.com/bpiyush/TestOfTime/blob/HEAD/package/datasets/synthetic.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"053b2f9dfa05e107"}},{"code_sha256_prefix":"9fbb5b9290a438a0","entry":"remove_multiple_spaces","repo":"bpiyush/TestOfTime","repo_kind":"official","path":"package/datasets/synthetic.py","file_url":"https://github.com/bpiyush/TestOfTime/blob/HEAD/package/datasets/synthetic.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9fbb5b9290a438a0"}},{"code_sha256_prefix":"149908a60cbc536a","entry":"update_config","repo":"bpiyush/TestOfTime","repo_kind":"official","path":"postpretrain.py","file_url":"https://github.com/bpiyush/TestOfTime/blob/HEAD/postpretrain.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"149908a60cbc536a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}