{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/space-time-crop-attend-improving-cross-modal","title":"Space-Time Crop & Attend: Improving Cross-modal Video Representation Learning","arxiv_id":"2103.10211","date":"2021-03-18","proceeding":"ICCV 2021 10","authors":["Mandela Patrick","Yuki M. Asano","Bernie Huang","Ishan Misra","Florian Metze","Joao Henriques","Andrea Vedaldi"],"abstract":"The quality of the image representations obtained from self-supervised learning depends strongly on the type of data augmentations used in the learning formulation. Recent papers have ported these methods from still images to videos and found that leveraging both audio and video signals yields strong gains; however, they did not find that spatial augmentations such as cropping, which are very important for still images, work as well for videos. In this paper, we improve these formulations in two ways unique to the spatio-temporal aspect of videos. First, for space, we show that spatial augmentations such as cropping do work well for videos too, but that previous implementations, due to the high processing and memory cost, could not do this at a scale sufficient for it to work well. To address this issue, we first introduce Feature Crop, a method to simulate such augmentations much more efficiently directly in feature space. Second, we show that as opposed to naive average pooling, the use of transformer-based attention improves performance significantly, and is well suited for processing feature crops. Combining both of our discoveries into a new method, Space-Time Crop & Attend (STiCA) we achieve state-of-the-art performance across multiple video-representation learning benchmarks. In particular, we achieve new state-of-the-art accuracies of 67.0% on HMDB-51 and 93.1% on UCF-101 when pre-training on Kinetics-400.","url_abs":"https://arxiv.org/abs/2103.10211v2","url_pdf":"https://arxiv.org/pdf/2103.10211v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"space-time-crop-attend-improving-cross-modal","repo_url":"https://github.com/facebookresearch/GDT","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2103.10211","atlas_url":"https://app.syntology.ai/?focus=2103.10211","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2103.10211"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/GDT","reach":null}],"summary":{"ran":12,"unverified":11},"by_repo_kind":{"official":{"samples":23,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9d7e4b24b5660ac5","entry":"Base","repo":"facebookresearch/GDT","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/GDT/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9d7e4b24b5660ac5"}},{"code_sha256_prefix":"249b5820589b96d3","entry":"BasicBlock","repo":"facebookresearch/GDT","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/GDT/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"249b5820589b96d3"}},{"code_sha256_prefix":"346198401060aa08","entry":"Bottleneck","repo":"facebookresearch/GDT","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/GDT/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"346198401060aa08"}},{"code_sha256_prefix":"4ab7412ae6131e98","entry":"Conv2Plus1D","repo":"facebookresearch/GDT","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/GDT/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4ab7412ae6131e98"}},{"code_sha256_prefix":"ba5523a1a6748c82","entry":"FFN","repo":"facebookresearch/GDT","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/GDT/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ba5523a1a6748c82"}},{"code_sha256_prefix":"66c25c20b741db7d","entry":"MLP","repo":"facebookresearch/GDT","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/GDT/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"66c25c20b741db7d"}},{"code_sha256_prefix":"025f425a305c99a1","entry":"MultiHeadAttention","repo":"facebookresearch/GDT","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/GDT/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"025f425a305c99a1"}},{"code_sha256_prefix":"dfb98feede0d7132","entry":"R2Plus1dStem","repo":"facebookresearch/GDT","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/GDT/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dfb98feede0d7132"}},{"code_sha256_prefix":"a45dddd3d20ddb5b","entry":"Transformer","repo":"facebookresearch/GDT","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/GDT/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a45dddd3d20ddb5b"}},{"code_sha256_prefix":"784b7c6f51d0dc00","entry":"TransformerPooling","repo":"facebookresearch/GDT","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/GDT/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"784b7c6f51d0dc00"}},{"code_sha256_prefix":"bd97e7875c7f6d01","entry":"TransposeSqueeze","repo":"facebookresearch/GDT","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/GDT/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bd97e7875c7f6d01"}},{"code_sha256_prefix":"1c92b909d0cdece1","entry":"VideoResNet","repo":"facebookresearch/GDT","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/GDT/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1c92b909d0cdece1"}},{"code_sha256_prefix":"05373243575c5edd","entry":"AudioBaseNetwork","repo":"facebookresearch/GDT","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/GDT/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"05373243575c5edd"}},{"code_sha256_prefix":"6840fa8301c4edba","entry":"ResNet","repo":"facebookresearch/GDT","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/GDT/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6840fa8301c4edba"}},{"code_sha256_prefix":"85fbc7f944b570fc","entry":"Stica_TransformerFMCrop","repo":"facebookresearch/GDT","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/GDT/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"85fbc7f944b570fc"}},{"code_sha256_prefix":"cf7d2feaca7a7e0e","entry":"VideoBaseNetwork","repo":"facebookresearch/GDT","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/GDT/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cf7d2feaca7a7e0e"}},{"code_sha256_prefix":"f8df10b38e4d9f0d","entry":"_resnet","repo":"facebookresearch/GDT","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/GDT/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f8df10b38e4d9f0d"}},{"code_sha256_prefix":"471f1e36d731e3a0","entry":"_video_resnet","repo":"facebookresearch/GDT","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/GDT/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"471f1e36d731e3a0"}},{"code_sha256_prefix":"cf0d617f6919c7db","entry":"get_audio_feature_extractor","repo":"facebookresearch/GDT","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/GDT/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cf0d617f6919c7db"}},{"code_sha256_prefix":"dfcc255237250347","entry":"get_video_feature_extractor","repo":"facebookresearch/GDT","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/GDT/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dfcc255237250347"}},{"code_sha256_prefix":"9982977c79b6bef4","entry":"r2plus1d_18","repo":"facebookresearch/GDT","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/GDT/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9982977c79b6bef4"}},{"code_sha256_prefix":"dcfa82af3e6afc58","entry":"r2plus1d_34","repo":"facebookresearch/GDT","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/GDT/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dcfa82af3e6afc58"}},{"code_sha256_prefix":"f78b8e83121feab2","entry":"resnet9","repo":"facebookresearch/GDT","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/GDT/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f78b8e83121feab2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}