{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-state-aware-visual-representations","title":"Learning State-Aware Visual Representations from Audible Interactions","arxiv_id":"2209.13583","date":"2022-09-27","proceeding":null,"authors":["Himangi Mittal","Pedro Morgado","Unnat Jain","Abhinav Gupta"],"abstract":"We propose a self-supervised algorithm to learn representations from egocentric video data. Recently, significant efforts have been made to capture humans interacting with their own environments as they go about their daily activities. In result, several large egocentric datasets of interaction-rich multi-modal data have emerged. However, learning representations from videos can be challenging. First, given the uncurated nature of long-form continuous videos, learning effective representations require focusing on moments in time when interactions take place. Second, visual representations of daily activities should be sensitive to changes in the state of the environment. However, current successful multi-modal learning frameworks encourage representation invariance over time. To address these challenges, we leverage audio signals to identify moments of likely interactions which are conducive to better learning. We also propose a novel self-supervised objective that learns from audible state changes caused by interactions. We validate these contributions extensively on two large-scale egocentric datasets, EPIC-Kitchens-100 and the recently released Ego4D, and show improvements on several downstream tasks, including action recognition, long-term action anticipation, and object state change classification.","url_abs":"https://arxiv.org/abs/2209.13583v1","url_pdf":"https://arxiv.org/pdf/2209.13583v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-state-aware-visual-representations","repo_url":"https://github.com/HimangiM/RepLAI","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"action-anticipation","task_name":"Action Anticipation"},{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"long-term-action-anticipation","task_name":"Long Term Action Anticipation"},{"task_slug":"object-state-change-classification","task_name":"Object State Change Classification"},{"task_slug":"point-of-no-return-pnr-temporal-localization","task_name":"Point- of-no-return (PNR) temporal localization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2209.13583","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2209.13583"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/HimangiM/RepLAI","reach":null}],"summary":{"ran":1,"ran_honours":1},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"9561eaaaa7e1b77c","entry":"SSRL_audiopeak_aotv2","repo":"HimangiM/RepLAI","repo_kind":"official","path":"criterions/ssrl_audiopeak_aotv2.py","file_url":"https://github.com/HimangiM/RepLAI/blob/HEAD/criterions/ssrl_audiopeak_aotv2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9561eaaaa7e1b77c"}},{"code_sha256_prefix":"f723ada1d6eea3ab","entry":"concat_all_gather","repo":"HimangiM/RepLAI","repo_kind":"official","path":"criterions/ssrl_audiopeak_aotv2.py","file_url":"https://github.com/HimangiM/RepLAI/blob/HEAD/criterions/ssrl_audiopeak_aotv2.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f723ada1d6eea3ab"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}