{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tuning-pre-trained-model-via-moment-probing","title":"Tuning Pre-trained Model via Moment Probing","arxiv_id":"2307.11342","date":"2023-07-21","proceeding":"ICCV 2023 1","authors":["Mingze Gao","Qilong Wang","Zhenyi Lin","Pengfei Zhu","QinGhua Hu","Jingbo Zhou"],"abstract":"Recently, efficient fine-tuning of large-scale pre-trained models has attracted increasing research interests, where linear probing (LP) as a fundamental module is involved in exploiting the final representations for task-dependent classification. However, most of the existing methods focus on how to effectively introduce a few of learnable parameters, and little work pays attention to the commonly used LP module. In this paper, we propose a novel Moment Probing (MP) method to further explore the potential of LP. Distinguished from LP which builds a linear classification head based on the mean of final features (e.g., word tokens for ViT) or classification tokens, our MP performs a linear classifier on feature distribution, which provides the stronger representation ability by exploiting richer statistical information inherent in features. Specifically, we represent feature distribution by its characteristic function, which is efficiently approximated by using first- and second-order moments of features. Furthermore, we propose a multi-head convolutional cross-covariance (MHC$^3$) to compute second-order moments in an efficient and effective manner. By considering that MP could affect feature learning, we introduce a partially shared module to learn two recalibrating parameters (PSRP) for backbones based on MP, namely MP$_{+}$. Extensive experiments on ten benchmarks using various models show that our MP significantly outperforms LP and is competitive with counterparts at less training cost, while our MP$_{+}$ achieves state-of-the-art performance.","url_abs":"https://arxiv.org/abs/2307.11342v3","url_pdf":"https://arxiv.org/pdf/2307.11342v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tuning-pre-trained-model-via-moment-probing","repo_url":"https://github.com/mingzeg/moment-probing","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"model","task_name":"model"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"convnext","method_name":"ConvNeXt"},{"method_slug":"deit","method_name":"DeiT"},{"method_slug":"focus","method_name":"Focus"},{"method_slug":"mae","method_name":"MAE"},{"method_slug":"swin-transformer","method_name":"Swin Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2307.11342","atlas_url":"https://app.syntology.ai/?focus=2307.11342","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.11342"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mingzeg/moment-probing","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/mingzeG/Moment-Probing","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":4,"ran":2,"ran_honours":1,"ran_fixture":1,"unverified":5},"by_repo_kind":{"official":{"samples":13,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"c3f4c265067c1717","entry":"MyNorm","repo":"mingzeg/moment-probing","repo_kind":"official","path":"models/as_mlp.py","file_url":"https://github.com/mingzeg/moment-probing/blob/HEAD/models/as_mlp.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c3f4c265067c1717"}},{"code_sha256_prefix":"def23f7d7d042968","entry":"get_init_weights_vit","repo":"mingzeg/moment-probing","repo_kind":"official","path":"models/vision_transformer.py","file_url":"https://github.com/mingzeg/moment-probing/blob/HEAD/models/vision_transformer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"def23f7d7d042968"}},{"code_sha256_prefix":"a9e48df75d2e823e","entry":"get_mce_from_accuracy","repo":"mingzeg/moment-probing","repo_kind":"official","path":"utils/mce_utils.py","file_url":"https://github.com/mingzeg/moment-probing/blob/HEAD/utils/mce_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a9e48df75d2e823e"}},{"code_sha256_prefix":"896b5eec245f1bab","entry":"init_ssf_scale_shift","repo":"mingzeg/moment-probing","repo_kind":"official","path":"models/vision_transformer.py","file_url":"https://github.com/mingzeg/moment-probing/blob/HEAD/models/vision_transformer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"896b5eec245f1bab"}},{"code_sha256_prefix":"88f4962784e09b88","entry":"param_groups_weight_decay","repo":"mingzeg/moment-probing","repo_kind":"official","path":"optim_factory.py","file_url":"https://github.com/mingzeg/moment-probing/blob/HEAD/optim_factory.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"88f4962784e09b88"}},{"code_sha256_prefix":"61e8364b7e9bcb70","entry":"ssf_ada","repo":"mingzeg/moment-probing","repo_kind":"official","path":"models/vision_transformer.py","file_url":"https://github.com/mingzeg/moment-probing/blob/HEAD/models/vision_transformer.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"61e8364b7e9bcb70"}},{"code_sha256_prefix":"993ae96666b00cb5","entry":"window_partition","repo":"mingzeg/moment-probing","repo_kind":"official","path":"models/swin_transformer.py","file_url":"https://github.com/mingzeg/moment-probing/blob/HEAD/models/swin_transformer.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"993ae96666b00cb5"}},{"code_sha256_prefix":"609922bd93c75117","entry":"window_reverse","repo":"mingzeg/moment-probing","repo_kind":"official","path":"models/swin_transformer.py","file_url":"https://github.com/mingzeg/moment-probing/blob/HEAD/models/swin_transformer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"609922bd93c75117"}},{"code_sha256_prefix":"80a1d69c480f2031","entry":"Covariance","repo":"mingzeG/Moment-Probing","repo_kind":"official","path":"models/representation/MP.py","file_url":"https://github.com/mingzeG/Moment-Probing/blob/HEAD/models/representation/MP.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"80a1d69c480f2031"}},{"code_sha256_prefix":"28fa2c5680a21935","entry":"Moment_Probing_ViT","repo":"mingzeG/Moment-Probing","repo_kind":"official","path":"models/representation/MP.py","file_url":"https://github.com/mingzeG/Moment-Probing/blob/HEAD/models/representation/MP.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"28fa2c5680a21935"}},{"code_sha256_prefix":"7619bf7f57c5d2b6","entry":"checkpoint_filter_fn","repo":"mingzeg/moment-probing","repo_kind":"official","path":"models/convnext.py","file_url":"https://github.com/mingzeg/moment-probing/blob/HEAD/models/convnext.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7619bf7f57c5d2b6"}},{"code_sha256_prefix":"61a14e104fefeb56","entry":"group_parameters","repo":"mingzeg/moment-probing","repo_kind":"official","path":"optim_factory.py","file_url":"https://github.com/mingzeg/moment-probing/blob/HEAD/optim_factory.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"61a14e104fefeb56"}},{"code_sha256_prefix":"1fd037cb67e5a59e","entry":"group_with_matcher","repo":"mingzeg/moment-probing","repo_kind":"official","path":"optim_factory.py","file_url":"https://github.com/mingzeg/moment-probing/blob/HEAD/optim_factory.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1fd037cb67e5a59e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}