{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/self-supervised-pre-training-for-transformer","title":"Self-Supervised Pre-Training for Transformer-Based Person Re-Identification","arxiv_id":"2111.12084","date":"2021-11-23","proceeding":null,"authors":["Hao Luo","Pichao Wang","Yi Xu","Feng Ding","Yanxin Zhou","Fan Wang","Hao Li","Rong Jin"],"abstract":"Transformer-based supervised pre-training achieves great performance in person re-identification (ReID). However, due to the domain gap between ImageNet and ReID datasets, it usually needs a larger pre-training dataset (e.g. ImageNet-21K) to boost the performance because of the strong data fitting ability of the transformer. To address this challenge, this work targets to mitigate the gap between the pre-training and ReID datasets from the perspective of data and model structure, respectively. We first investigate self-supervised learning (SSL) methods with Vision Transformer (ViT) pretrained on unlabelled person images (the LUPerson dataset), and empirically find it significantly surpasses ImageNet supervised pre-training models on ReID tasks. To further reduce the domain gap and accelerate the pre-training, the Catastrophic Forgetting Score (CFS) is proposed to evaluate the gap between pre-training and fine-tuning data. Based on CFS, a subset is selected via sampling relevant data close to the down-stream ReID data and filtering irrelevant data from the pre-training dataset. For the model structure, a ReID-specific module named IBN-based convolution stem (ICS) is proposed to bridge the domain gap by learning more invariant features. Extensive experiments have been conducted to fine-tune the pre-training models under supervised learning, unsupervised domain adaptation (UDA), and unsupervised learning (USL) settings. We successfully downscale the LUPerson dataset to 50% with no performance degradation. Finally, we achieve state-of-the-art performance on Market-1501 and MSMT17. For example, our ViT-S/16 achieves 91.3%/89.9%/89.6% mAP accuracy on Market1501 for supervised/UDA/USL ReID. Codes and models will be released to https://github.com/michuanhaohao/TransReID-SSL.","url_abs":"https://arxiv.org/abs/2111.12084v1","url_pdf":"https://arxiv.org/pdf/2111.12084v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"self-supervised-pre-training-for-transformer","repo_url":"https://github.com/michuanhaohao/transreid-ssl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"self-supervised-pre-training-for-transformer","repo_url":"https://github.com/DengpanFu/LUPerson","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"self-supervised-pre-training-for-transformer","repo_url":"https://github.com/damo-cv/TransReID-SSL","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"domain-adaptation","task_name":"Domain Adaptation"},{"task_slug":"person-re-identification","task_name":"Person Re-Identification"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"unsupervised-domain-adaptation","task_name":"Unsupervised Domain Adaptation"},{"task_slug":"unsupervised-person-re-identification","task_name":"Unsupervised Person Re-Identification"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/person-re-identification-on-msmt17","task":"Person Re-Identification","dataset":"MSMT17","model":"TransReID-SSL (ViT-B without RK)","rank_in_archive_order":10,"of":43,"metrics":{"Rank-1":"89.5","mAP":"75.0"},"uses_additional_data":false},{"leaderboard":"/sota/person-re-identification-on-msmt17","task":"Person Re-Identification","dataset":"MSMT17","model":"TransReID-SSL (without RK)","rank_in_archive_order":43,"of":43,"metrics":{"Rank-1":"89.6"},"uses_additional_data":true},{"leaderboard":"/sota/person-re-identification-on-market-1501","task":"Person Re-Identification","dataset":"Market-1501","model":"TransReID-SSL (ViT-B w/o RK)","rank_in_archive_order":11,"of":135,"metrics":{"Rank-1":"96.7","mAP":"93.2"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-person-re-identification-on-12","task":"Unsupervised Person Re-Identification","dataset":"MSMT17","model":"TransReID-SSL (ViTi-S)","rank_in_archive_order":6,"of":12,"metrics":{"Rank-1":"75","mAP":"50.6"},"uses_additional_data":true},{"leaderboard":"/sota/unsupervised-person-re-identification-on-12","task":"Unsupervised Person Re-Identification","dataset":"MSMT17","model":"TransReID-SSL (ViT-S)","rank_in_archive_order":9,"of":12,"metrics":{"Rank-1":"66.4","mAP":"40.9"},"uses_additional_data":true},{"leaderboard":"/sota/unsupervised-person-re-identification-on-4","task":"Unsupervised Person Re-Identification","dataset":"Market-1501","model":"TransReID-SSL (ViTi-S)","rank_in_archive_order":1,"of":23,"metrics":{"MAP":"89.6","Rank-1":"95.3"},"uses_additional_data":true},{"leaderboard":"/sota/unsupervised-person-re-identification-on-4","task":"Unsupervised Person Re-Identification","dataset":"Market-1501","model":"TransReID-SSL (ViT-S)","rank_in_archive_order":4,"of":23,"metrics":{"MAP":"88.2","Rank-1":"94.2"},"uses_additional_data":true},{"leaderboard":"/sota/unsupervised-person-re-identification-on-4","task":"Unsupervised Person Re-Identification","dataset":"Market-1501","model":"TransReID-SSL (ViT-S w/o RK)","rank_in_archive_order":23,"of":23,"metrics":{"Rank-1":"95.3"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2111.12084","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2111.12084"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/michuanhaohao/transreid-ssl","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/DengpanFu/LUPerson","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/damo-cv/TransReID-SSL","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":3,"unverified":5},"by_repo_kind":{"listed":{"samples":8,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"e3b2a83f52101f2d","entry":"normalize","repo":"damo-cv/TransReID-SSL","repo_kind":"listed","path":"transreid_pytorch/loss/triplet_loss.py","file_url":"https://github.com/damo-cv/TransReID-SSL/blob/HEAD/transreid_pytorch/loss/triplet_loss.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"e3b2a83f52101f2d"}},{"code_sha256_prefix":"a9297ce0fe9cac98","entry":"train_collate_fn","repo":"damo-cv/TransReID-SSL","repo_kind":"listed","path":"transreid_pytorch/datasets/make_dataloader.py","file_url":"https://github.com/damo-cv/TransReID-SSL/blob/HEAD/transreid_pytorch/datasets/make_dataloader.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"a9297ce0fe9cac98"}},{"code_sha256_prefix":"c4ae4a3fc4c4703e","entry":"val_collate_fn","repo":"damo-cv/TransReID-SSL","repo_kind":"listed","path":"transreid_pytorch/datasets/make_dataloader.py","file_url":"https://github.com/damo-cv/TransReID-SSL/blob/HEAD/transreid_pytorch/datasets/make_dataloader.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"c4ae4a3fc4c4703e"}},{"code_sha256_prefix":"71b4e1d375d8fd78","entry":"cosine_dist","repo":"damo-cv/TransReID-SSL","repo_kind":"listed","path":"transreid_pytorch/loss/triplet_loss.py","file_url":"https://github.com/damo-cv/TransReID-SSL/blob/HEAD/transreid_pytorch/loss/triplet_loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"71b4e1d375d8fd78"}},{"code_sha256_prefix":"5315f75bf5367f0a","entry":"euclidean_dist","repo":"damo-cv/TransReID-SSL","repo_kind":"listed","path":"transreid_pytorch/loss/triplet_loss.py","file_url":"https://github.com/damo-cv/TransReID-SSL/blob/HEAD/transreid_pytorch/loss/triplet_loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"5315f75bf5367f0a"}},{"code_sha256_prefix":"b26b436199409345","entry":"make_loss","repo":"damo-cv/TransReID-SSL","repo_kind":"listed","path":"transreid_pytorch/loss/make_loss.py","file_url":"https://github.com/damo-cv/TransReID-SSL/blob/HEAD/transreid_pytorch/loss/make_loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"b26b436199409345"}},{"code_sha256_prefix":"5f1101145f896208","entry":"read_image","repo":"damo-cv/TransReID-SSL","repo_kind":"listed","path":"transreid_pytorch/datasets/bases.py","file_url":"https://github.com/damo-cv/TransReID-SSL/blob/HEAD/transreid_pytorch/datasets/bases.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"5f1101145f896208"}},{"code_sha256_prefix":"fa0d6dd791ce2927","entry":"shuffle_unit","repo":"damo-cv/TransReID-SSL","repo_kind":"listed","path":"transreid_pytorch/model/make_model.py","file_url":"https://github.com/damo-cv/TransReID-SSL/blob/HEAD/transreid_pytorch/model/make_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"fa0d6dd791ce2927"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}