{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/adamae-adaptive-masking-for-efficient","title":"AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoders","arxiv_id":"2211.09120","date":"2022-11-16","proceeding":"CVPR 2023 1","authors":["Wele Gedara Chaminda Bandara","Naman Patel","Ali Gholami","Mehdi Nikkhah","Motilal Agrawal","Vishal M. Patel"],"abstract":"Masked Autoencoders (MAEs) learn generalizable representations for image, text, audio, video, etc., by reconstructing masked input data from tokens of the visible data. Current MAE approaches for videos rely on random patch, tube, or frame-based masking strategies to select these tokens. This paper proposes AdaMAE, an adaptive masking strategy for MAEs that is end-to-end trainable. Our adaptive masking strategy samples visible tokens based on the semantic context using an auxiliary sampling network. This network estimates a categorical distribution over spacetime-patch tokens. The tokens that increase the expected reconstruction error are rewarded and selected as visible tokens, motivated by the policy gradient algorithm in reinforcement learning. We show that AdaMAE samples more tokens from the high spatiotemporal information regions, thereby allowing us to mask 95% of tokens, resulting in lower memory requirements and faster pre-training. We conduct ablation studies on the Something-Something v2 (SSv2) dataset to demonstrate the efficacy of our adaptive sampling approach and report state-of-the-art results of 70.0% and 81.7% in top-1 accuracy on SSv2 and Kinetics-400 action classification datasets with a ViT-Base backbone and 800 pre-training epochs.","url_abs":"https://arxiv.org/abs/2211.09120v1","url_pdf":"https://arxiv.org/pdf/2211.09120v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"adamae-adaptive-masking-for-efficient","repo_url":"https://github.com/wgcban/adamae","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"adamae-adaptive-masking-for-efficient","repo_url":"https://github.com/Nithin-GK/UniteandConquer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"action-classification","task_name":"Action Classification"},{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[{"method_slug":"adaptive-masking","method_name":"Adaptive Masking"},{"method_slug":"l1-regularization","method_name":"L1 Regularization"},{"method_slug":"mae","method_name":"MAE"},{"method_slug":"masked-convolution","method_name":"Masked Convolution"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-classification-on-kinetics-400","task":"Action Classification","dataset":"Kinetics-400","model":"AdaMAE","rank_in_archive_order":81,"of":207,"metrics":{"Acc@1":"81.7","Acc@5":"95.2"},"uses_additional_data":false},{"leaderboard":"/sota/action-classification-on-something-something-2","task":"Action Classification","dataset":"Something-Something V2","model":"AdaMAE","rank_in_archive_order":1,"of":1,"metrics":{"Acc@1":"70.04","Acc@5":"92.7"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2211.09120","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2211.09120"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Nithin-GK/UniteandConquer","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wgcban/adamae","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":8,"ran_draft_wrong":1,"unverified":16},"by_repo_kind":{"official":{"samples":15,"ran":7,"repositories":1},"listed":{"samples":10,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4e2ac192b520c457","entry":"args_to_dict","repo":"Nithin-GK/UniteandConquer","repo_kind":"listed","path":"class_utils/script_util.py","file_url":"https://github.com/Nithin-GK/UniteandConquer/blob/HEAD/class_utils/script_util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4e2ac192b520c457"}},{"code_sha256_prefix":"bc39b8ed5fefa7eb","entry":"crop_clip","repo":"wgcban/adamae","repo_kind":"official","path":"functional.py","file_url":"https://github.com/wgcban/adamae/blob/HEAD/functional.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc39b8ed5fefa7eb"}},{"code_sha256_prefix":"a3448702d6025ce6","entry":"get_num_layer_for_vit","repo":"wgcban/adamae","repo_kind":"official","path":"optim_factory.py","file_url":"https://github.com/wgcban/adamae/blob/HEAD/optim_factory.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a3448702d6025ce6"}},{"code_sha256_prefix":"2b53e457d5632b4e","entry":"get_resize_sizes","repo":"wgcban/adamae","repo_kind":"official","path":"functional.py","file_url":"https://github.com/wgcban/adamae/blob/HEAD/functional.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2b53e457d5632b4e"}},{"code_sha256_prefix":"981936c73d3cdd12","entry":"resize_clip","repo":"wgcban/adamae","repo_kind":"official","path":"functional.py","file_url":"https://github.com/wgcban/adamae/blob/HEAD/functional.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"981936c73d3cdd12"}},{"code_sha256_prefix":"e866e1196986f728","entry":"shear_x","repo":"wgcban/adamae","repo_kind":"official","path":"rand_augment.py","file_url":"https://github.com/wgcban/adamae/blob/HEAD/rand_augment.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e866e1196986f728"}},{"code_sha256_prefix":"7b2503c99c9447fb","entry":"shear_y","repo":"wgcban/adamae","repo_kind":"official","path":"rand_augment.py","file_url":"https://github.com/wgcban/adamae/blob/HEAD/rand_augment.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b2503c99c9447fb"}},{"code_sha256_prefix":"323759f906fbb394","entry":"translate_x_rel","repo":"wgcban/adamae","repo_kind":"official","path":"rand_augment.py","file_url":"https://github.com/wgcban/adamae/blob/HEAD/rand_augment.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"323759f906fbb394"}},{"code_sha256_prefix":"129b804760b3115f","entry":"zero_module","repo":"Nithin-GK/UniteandConquer","repo_kind":"listed","path":"class_utils/nn.py","file_url":"https://github.com/Nithin-GK/UniteandConquer/blob/HEAD/class_utils/nn.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"129b804760b3115f"}},{"code_sha256_prefix":"0d86b6995e23b2d4","entry":"CE_loss","repo":"wgcban/adamae","repo_kind":"official","path":"msc/losses.py","file_url":"https://github.com/wgcban/adamae/blob/HEAD/msc/losses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0d86b6995e23b2d4"}},{"code_sha256_prefix":"ecd0fc28815b65ae","entry":"avg_pool_nd","repo":"Nithin-GK/UniteandConquer","repo_kind":"listed","path":"class_utils/nn.py","file_url":"https://github.com/Nithin-GK/UniteandConquer/blob/HEAD/class_utils/nn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ecd0fc28815b65ae"}},{"code_sha256_prefix":"d046b61348049e13","entry":"class_find","repo":"Nithin-GK/UniteandConquer","repo_kind":"listed","path":"gradio_set.py","file_url":"https://github.com/Nithin-GK/UniteandConquer/blob/HEAD/gradio_set.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d046b61348049e13"}},{"code_sha256_prefix":"fe4eb545bbb728e0","entry":"conv_nd","repo":"Nithin-GK/UniteandConquer","repo_kind":"listed","path":"class_utils/nn.py","file_url":"https://github.com/Nithin-GK/UniteandConquer/blob/HEAD/class_utils/nn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fe4eb545bbb728e0"}},{"code_sha256_prefix":"2234d67e4ed38722","entry":"create_model","repo":"Nithin-GK/UniteandConquer","repo_kind":"listed","path":"class_utils/script_util.py","file_url":"https://github.com/Nithin-GK/UniteandConquer/blob/HEAD/class_utils/script_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2234d67e4ed38722"}},{"code_sha256_prefix":"65ce936001e1e5a2","entry":"d_clip_loss","repo":"Nithin-GK/UniteandConquer","repo_kind":"listed","path":"face_utils/farl_text.py","file_url":"https://github.com/Nithin-GK/UniteandConquer/blob/HEAD/face_utils/farl_text.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"65ce936001e1e5a2"}},{"code_sha256_prefix":"3081b94d2db18290","entry":"fetch_file_cached","repo":"Nithin-GK/UniteandConquer","repo_kind":"listed","path":"utils/download_models_func.py","file_url":"https://github.com/Nithin-GK/UniteandConquer/blob/HEAD/utils/download_models_func.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3081b94d2db18290"}},{"code_sha256_prefix":"612575eeaa3ae40c","entry":"get_alpha","repo":"wgcban/adamae","repo_kind":"official","path":"msc/losses.py","file_url":"https://github.com/wgcban/adamae/blob/HEAD/msc/losses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"612575eeaa3ae40c"}},{"code_sha256_prefix":"135b3dc835ffe6ad","entry":"get_confirm_token","repo":"Nithin-GK/UniteandConquer","repo_kind":"listed","path":"utils/download_models_func.py","file_url":"https://github.com/Nithin-GK/UniteandConquer/blob/HEAD/utils/download_models_func.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"135b3dc835ffe6ad"}},{"code_sha256_prefix":"29416043c7035e7c","entry":"get_loss_scale_for_deepspeed","repo":"wgcban/adamae","repo_kind":"official","path":"msc/engine_for_finetuning.py","file_url":"https://github.com/wgcban/adamae/blob/HEAD/msc/engine_for_finetuning.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"29416043c7035e7c"}},{"code_sha256_prefix":"cd41abe0ddd75bc6","entry":"get_model","repo":"wgcban/adamae","repo_kind":"official","path":"pretrain_mae_vit.py","file_url":"https://github.com/wgcban/adamae/blob/HEAD/pretrain_mae_vit.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cd41abe0ddd75bc6"}},{"code_sha256_prefix":"0ef0019514eaaa09","entry":"get_parameter_groups","repo":"wgcban/adamae","repo_kind":"official","path":"optim_factory.py","file_url":"https://github.com/wgcban/adamae/blob/HEAD/optim_factory.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0ef0019514eaaa09"}},{"code_sha256_prefix":"95f14a0c91022fc2","entry":"greet","repo":"Nithin-GK/UniteandConquer","repo_kind":"listed","path":"gradio_set.py","file_url":"https://github.com/Nithin-GK/UniteandConquer/blob/HEAD/gradio_set.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"95f14a0c91022fc2"}},{"code_sha256_prefix":"c66339fca70da624","entry":"softmax_helper","repo":"wgcban/adamae","repo_kind":"official","path":"msc/losses.py","file_url":"https://github.com/wgcban/adamae/blob/HEAD/msc/losses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c66339fca70da624"}},{"code_sha256_prefix":"14b3b13b2e7ee844","entry":"tensor_normalize","repo":"wgcban/adamae","repo_kind":"official","path":"kinetics.py","file_url":"https://github.com/wgcban/adamae/blob/HEAD/kinetics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"14b3b13b2e7ee844"}},{"code_sha256_prefix":"c7976ea27edc377a","entry":"train_class_batch","repo":"wgcban/adamae","repo_kind":"official","path":"msc/engine_for_finetuning.py","file_url":"https://github.com/wgcban/adamae/blob/HEAD/msc/engine_for_finetuning.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c7976ea27edc377a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}