{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/labelling-unlabelled-videos-from-scratch-with","title":"Labelling unlabelled videos from scratch with multi-modal self-supervision","arxiv_id":"2006.13662","date":"2020-06-24","proceeding":"NeurIPS 2020 12","authors":["Yuki M. Asano","Mandela Patrick","Christian Rupprecht","Andrea Vedaldi"],"abstract":"A large part of the current success of deep learning lies in the effectiveness of data -- more precisely: labelled data. Yet, labelling a dataset with human annotation continues to carry high costs, especially for videos. While in the image domain, recent methods have allowed to generate meaningful (pseudo-) labels for unlabelled datasets without supervision, this development is missing for the video domain where learning feature representations is the current focus. In this work, we a) show that unsupervised labelling of a video dataset does not come for free from strong feature encoders and b) propose a novel clustering method that allows pseudo-labelling of a video dataset without any human annotations, by leveraging the natural correspondence between the audio and visual modalities. An extensive analysis shows that the resulting clusters have high semantic overlap to ground truth human labels. We further introduce the first benchmarking results on unsupervised labelling of common video datasets Kinetics, Kinetics-Sound, VGG-Sound and AVE.","url_abs":"https://arxiv.org/abs/2006.13662v3","url_pdf":"https://arxiv.org/pdf/2006.13662v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"labelling-unlabelled-videos-from-scratch-with","repo_url":"https://github.com/facebookresearch/selavi","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"clustering","task_name":"Clustering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2006.13662","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.13662"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/selavi","reach":null}],"summary":{"ran":7,"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"official":{"samples":10,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"c4627f2f4f396ce7","entry":"AVModel","repo":"facebookresearch/selavi","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/selavi/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c4627f2f4f396ce7"}},{"code_sha256_prefix":"5a359f3350bfb627","entry":"AudioBaseNetwork","repo":"facebookresearch/selavi","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/selavi/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5a359f3350bfb627"}},{"code_sha256_prefix":"a4f923bd5136361f","entry":"Flatten","repo":"facebookresearch/selavi","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/selavi/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a4f923bd5136361f"}},{"code_sha256_prefix":"cd43c253a0e41d2f","entry":"Identity","repo":"facebookresearch/selavi","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/selavi/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"cd43c253a0e41d2f"}},{"code_sha256_prefix":"5daf8b04fc65b8c4","entry":"MLPv2","repo":"facebookresearch/selavi","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/selavi/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5daf8b04fc65b8c4"}},{"code_sha256_prefix":"c7c103d872864926","entry":"Unsqueeze","repo":"facebookresearch/selavi","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/selavi/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c7c103d872864926"}},{"code_sha256_prefix":"d4be6d22f0057805","entry":"VideoBaseNetwork","repo":"facebookresearch/selavi","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/selavi/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d4be6d22f0057805"}},{"code_sha256_prefix":"4e6c0c3673854b32","entry":"get_audio_feature_extractor","repo":"facebookresearch/selavi","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/selavi/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4e6c0c3673854b32"}},{"code_sha256_prefix":"d38b682452622512","entry":"get_video_feature_extractor","repo":"facebookresearch/selavi","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/selavi/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d38b682452622512"}},{"code_sha256_prefix":"2d01475f13deb6ce","entry":"random_weight_init","repo":"facebookresearch/selavi","repo_kind":"official","path":"model.py","file_url":"https://github.com/facebookresearch/selavi/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2d01475f13deb6ce"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}