{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/contrastive-learning-of-global-and-local-1","title":"Contrastive Learning of Global-Local Video Representations","arxiv_id":"2104.05418","date":"2021-04-07","proceeding":null,"authors":["Shuang Ma","Zhaoyang Zeng","Daniel McDuff","Yale Song"],"abstract":"Contrastive learning has delivered impressive results for various tasks in the self-supervised regime. However, existing approaches optimize for learning representations specific to downstream scenarios, i.e., \\textit{global} representations suitable for tasks such as classification or \\textit{local} representations for tasks such as detection and localization. While they produce satisfactory results in the intended downstream scenarios, they often fail to generalize to tasks that they were not originally designed for. In this work, we propose to learn video representations that generalize to both the tasks which require global semantic information (e.g., classification) and the tasks that require local fine-grained spatio-temporal information (e.g., localization). We achieve this by optimizing two contrastive objectives that together encourage our model to learn global-local visual information given audio signals. We show that the two objectives mutually improve the generalizability of the learned global-local representations, significantly outperforming their disjointly learned counterparts. We demonstrate our approach on various tasks including action/sound classification, lip reading, deepfake detection, event and sound localization (https://github.com/yunyikristy/global\\_local).","url_abs":"https://arxiv.org/abs/2104.05418v2","url_pdf":"https://arxiv.org/pdf/2104.05418v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"contrastive-learning-of-global-and-local-1","repo_url":"https://github.com/yunyikristy/global_local","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"deepfake-detection","task_name":"DeepFake Detection"},{"task_slug":"face-swapping","task_name":"Face Swapping"},{"task_slug":"classification","task_name":"General Classification"},{"task_slug":"lip-reading","task_name":"Lip Reading"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"sound-classification","task_name":"Sound Classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2104.05418","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}