{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/point-m2ae-multi-scale-masked-autoencoders","title":"Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training","arxiv_id":"2205.14401","date":"2022-05-28","proceeding":null,"authors":["Renrui Zhang","Ziyu Guo","Rongyao Fang","Bin Zhao","Dong Wang","Yu Qiao","Hongsheng Li","Peng Gao"],"abstract":"Masked Autoencoders (MAE) have shown great potentials in self-supervised pre-training for language and 2D image transformers. However, it still remains an open question on how to exploit masked autoencoding for learning 3D representations of irregular point clouds. In this paper, we propose Point-M2AE, a strong Multi-scale MAE pre-training framework for hierarchical self-supervised learning of 3D point clouds. Unlike the standard transformer in MAE, we modify the encoder and decoder into pyramid architectures to progressively model spatial geometries and capture both fine-grained and high-level semantics of 3D shapes. For the encoder that downsamples point tokens by stages, we design a multi-scale masking strategy to generate consistent visible regions across scales, and adopt a local spatial self-attention mechanism during fine-tuning to focus on neighboring patterns. By multi-scale token propagation, the lightweight decoder gradually upsamples point tokens with complementary skip connections from the encoder, which further promotes the reconstruction from a global-to-local perspective. Extensive experiments demonstrate the state-of-the-art performance of Point-M2AE for 3D representation learning. With a frozen encoder after pre-training, Point-M2AE achieves 92.9% accuracy for linear SVM on ModelNet40, even surpassing some fully trained methods. By fine-tuning on downstream tasks, Point-M2AE achieves 86.43% accuracy on ScanObjectNN, +3.36% to the second-best, and largely benefits the few-shot classification, part segmentation and 3D object detection with the hierarchical pre-training scheme. Code is available at https://github.com/ZrrSkywalker/Point-M2AE.","url_abs":"https://arxiv.org/abs/2205.14401v2","url_pdf":"https://arxiv.org/pdf/2205.14401v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"point-m2ae-multi-scale-masked-autoencoders","repo_url":"https://github.com/zrrskywalker/point-m2ae","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"point-m2ae-multi-scale-masked-autoencoders","repo_url":"https://github.com/zrrskywalker/i2p-mae","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"point-m2ae-multi-scale-masked-autoencoders","repo_url":"https://github.com/MindSpore-paper-code-2/code400/tree/main/MAE","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"3d-object-detection","task_name":"3D Object Detection"},{"task_slug":"3d-point-cloud-classification","task_name":"3D Point Cloud Classification"},{"task_slug":"3d-point-cloud-linear-classification","task_name":"3D Point Cloud Linear Classification"},{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"few-shot-3d-point-cloud-classification","task_name":"Few-Shot 3D Point Cloud Classification"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"open-question","task_name":"Open-Ended Question Answering"},{"task_slug":"point-cloud-pre-training","task_name":"Point Cloud Pre-training"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"mae","method_name":"MAE"},{"method_slug":"svm","method_name":"SVM"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/3d-point-cloud-classification-on-modelnet40","task":"3D Point Cloud Classification","dataset":"ModelNet40","model":"Point-M2AE","rank_in_archive_order":31,"of":111,"metrics":{"Overall Accuracy":"94.0"},"uses_additional_data":true},{"leaderboard":"/sota/3d-point-cloud-classification-on-modelnet40","task":"3D Point Cloud Classification","dataset":"ModelNet40","model":"Point-M2AE-SVM","rank_in_archive_order":82,"of":111,"metrics":{"Overall Accuracy":"92.9"},"uses_additional_data":true},{"leaderboard":"/sota/3d-point-cloud-classification-on-scanobjectnn","task":"3D Point Cloud Classification","dataset":"ScanObjectNN","model":"Point-M2AE","rank_in_archive_order":49,"of":77,"metrics":{"OBJ-BG (OA)":"91.22","OBJ-ONLY (OA)":"88.81","Overall Accuracy":"86.43"},"uses_additional_data":true},{"leaderboard":"/sota/3d-point-cloud-linear-classification-on","task":"3D Point Cloud Linear Classification","dataset":"ModelNet40","model":"Point-M2AE","rank_in_archive_order":5,"of":20,"metrics":{"Overall Accuracy":"92.9"},"uses_additional_data":true},{"leaderboard":"/sota/few-shot-3d-point-cloud-classification-on-3","task":"Few-Shot 3D Point Cloud Classification","dataset":"ModelNet40 10-way (10-shot)","model":"Point-M2AE","rank_in_archive_order":16,"of":31,"metrics":{"Overall Accuracy":"92.3","Standard Deviation":"4.5"},"uses_additional_data":true},{"leaderboard":"/sota/few-shot-3d-point-cloud-classification-on-4","task":"Few-Shot 3D Point Cloud Classification","dataset":"ModelNet40 10-way (20-shot)","model":"Point-M2AE","rank_in_archive_order":15,"of":31,"metrics":{"Overall Accuracy":"95.0","Standard Deviation":"3.0"},"uses_additional_data":true},{"leaderboard":"/sota/few-shot-3d-point-cloud-classification-on-1","task":"Few-Shot 3D Point Cloud Classification","dataset":"ModelNet40 5-way (10-shot)","model":"Point-M2AE","rank_in_archive_order":13,"of":30,"metrics":{"Overall Accuracy":"96.8","Standard Deviation":"1.8"},"uses_additional_data":true},{"leaderboard":"/sota/few-shot-3d-point-cloud-classification-on-2","task":"Few-Shot 3D Point Cloud Classification","dataset":"ModelNet40 5-way (20-shot)","model":"Point-M2AE","rank_in_archive_order":11,"of":30,"metrics":{"Overall Accuracy":"98.3","Standard Deviation":"1.4"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2205.14401","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}