{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multi-label-cluster-discrimination-for-visual","title":"Multi-label Cluster Discrimination for Visual Representation Learning","arxiv_id":"2407.17331","date":"2024-07-24","proceeding":null,"authors":["Xiang An","Kaicheng Yang","Xiangzi Dai","Ziyong Feng","Jiankang Deng"],"abstract":"Contrastive Language Image Pre-training (CLIP) has recently demonstrated success across various tasks due to superior feature representation empowered by image-text contrastive learning. However, the instance discrimination method used by CLIP can hardly encode the semantic structure of training data. To handle this limitation, cluster discrimination has been proposed through iterative cluster assignment and classification. Nevertheless, most cluster discrimination approaches only define a single pseudo-label for each image, neglecting multi-label signals in the image. In this paper, we propose a novel Multi-Label Cluster Discrimination method named MLCD to enhance representation learning. In the clustering step, we first cluster the large-scale LAION-400M dataset into one million centers based on off-the-shelf embedding features. Considering that natural images frequently contain multiple visual objects or attributes, we select the multiple closest centers as auxiliary class labels. In the discrimination step, we design a novel multi-label classification loss, which elegantly separates losses from positive classes and negative classes, and alleviates ambiguity on decision boundary. We validate the proposed multi-label cluster discrimination method with experiments on different scales of models and pre-training datasets. Experimental results show that our method achieves state-of-the-art performance on multiple downstream tasks including linear probe, zero-shot classification, and image-text retrieval. Code and models have been released at https://github.com/deepglint/unicom .","url_abs":"https://arxiv.org/abs/2407.17331v2","url_pdf":"https://arxiv.org/pdf/2407.17331v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"multi-label-cluster-discrimination-for-visual","repo_url":"https://github.com/deepglint/unicom","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"image-text-retrieval","task_name":"Image-text Retrieval"},{"task_slug":"multi-label-classification-2","task_name":"MUlTI-LABEL-ClASSIFICATION"},{"task_slug":"multi-label-classification","task_name":"Multi-Label Classification"},{"task_slug":"pseudo-label","task_name":"Pseudo Label"},{"task_slug":"referring-expression-segmentation","task_name":"Referring Expression Segmentation"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"self-supervised-image-classification","task_name":"Self-Supervised Image Classification"},{"task_slug":"text-retrieval","task_name":"Text Retrieval"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"},{"task_slug":null,"task_name":"zero-shot-classification"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/referring-expression-segmentation-on-refcoco-8","task":"Referring Expression Segmentation","dataset":"RefCOCO testA","model":"MLCD-Seg-7B","rank_in_archive_order":3,"of":13,"metrics":{"Overall IoU":"85.3"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on-refcoco-9","task":"Referring Expression Segmentation","dataset":"RefCOCO testB","model":"MLCD-Seg-7B","rank_in_archive_order":3,"of":13,"metrics":{"Overall IoU":"81.5"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on-refcoco-5","task":"Referring Expression Segmentation","dataset":"RefCOCO+ test B","model":"MLCD-Seg-7B","rank_in_archive_order":1,"of":30,"metrics":{"Overall IoU":"75.6"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on-refcoco-4","task":"Referring Expression Segmentation","dataset":"RefCOCO+ testA","model":"MLCD-Seg-7B","rank_in_archive_order":2,"of":30,"metrics":{"Overall IoU":"82.9"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on-refcoco-3","task":"Referring Expression Segmentation","dataset":"RefCOCO+ val","model":"MLCD-Seg-7B","rank_in_archive_order":1,"of":33,"metrics":{"Overall IoU":"79.4"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on-refcocog-1","task":"Referring Expression Segmentation","dataset":"RefCOCOg-test","model":"MLCD-Seg-7B","rank_in_archive_order":2,"of":18,"metrics":{"Overall IoU":"80.5"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on-refcocog","task":"Referring Expression Segmentation","dataset":"RefCOCOg-val","model":"MLCD-Seg-7B","rank_in_archive_order":1,"of":23,"metrics":{"Overall IoU":"79.9"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on-refcoco","task":"Referring Expression Segmentation","dataset":"RefCoCo val","model":"MLCD-Seg-7B","rank_in_archive_order":4,"of":37,"metrics":{"Overall IoU":"83.6"},"uses_additional_data":true},{"leaderboard":"/sota/visual-question-answering-on-docvqa-test","task":"Visual Question Answering (VQA)","dataset":"DocVQA test","model":"MLCD-Embodied-7B","rank_in_archive_order":2,"of":33,"metrics":{"ANLS":"0.916"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.17331","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.17331"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/deepglint/unicom","reach":null}],"summary":{"ran":7,"unverified":4},"by_repo_kind":{"official":{"samples":11,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7d1b1fa84c7948b6","entry":"MLCDEncoder","repo":"deepglint/unicom","repo_kind":"official","path":"mlcd/vit_rope2d_hf.py","file_url":"https://github.com/deepglint/unicom/blob/HEAD/mlcd/vit_rope2d_hf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7d1b1fa84c7948b6"}},{"code_sha256_prefix":"1147651f76ba5063","entry":"MLCDEncoderLayer","repo":"deepglint/unicom","repo_kind":"official","path":"mlcd/vit_rope2d_hf.py","file_url":"https://github.com/deepglint/unicom/blob/HEAD/mlcd/vit_rope2d_hf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1147651f76ba5063"}},{"code_sha256_prefix":"0744bf53c4214b05","entry":"MLCDMLP","repo":"deepglint/unicom","repo_kind":"official","path":"mlcd/vit_rope2d_hf.py","file_url":"https://github.com/deepglint/unicom/blob/HEAD/mlcd/vit_rope2d_hf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0744bf53c4214b05"}},{"code_sha256_prefix":"27eecf02ec652597","entry":"MLCDSdpaAttention","repo":"deepglint/unicom","repo_kind":"official","path":"mlcd/vit_rope2d_hf.py","file_url":"https://github.com/deepglint/unicom/blob/HEAD/mlcd/vit_rope2d_hf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"27eecf02ec652597"}},{"code_sha256_prefix":"ff45de9afb241b5f","entry":"MLCDVisionConfig","repo":"deepglint/unicom","repo_kind":"official","path":"mlcd/vit_rope2d_hf.py","file_url":"https://github.com/deepglint/unicom/blob/HEAD/mlcd/vit_rope2d_hf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ff45de9afb241b5f"}},{"code_sha256_prefix":"47857c277feef274","entry":"MLCDVisionEmbeddings","repo":"deepglint/unicom","repo_kind":"official","path":"mlcd/vit_rope2d_hf.py","file_url":"https://github.com/deepglint/unicom/blob/HEAD/mlcd/vit_rope2d_hf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"47857c277feef274"}},{"code_sha256_prefix":"a12357d48c1c9ffa","entry":"VisionRotaryEmbedding","repo":"deepglint/unicom","repo_kind":"official","path":"mlcd/vit_rope2d_hf.py","file_url":"https://github.com/deepglint/unicom/blob/HEAD/mlcd/vit_rope2d_hf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a12357d48c1c9ffa"}},{"code_sha256_prefix":"47b28dd1d00a8546","entry":"MLCDPreTrainedModel","repo":"deepglint/unicom","repo_kind":"official","path":"mlcd/vit_rope2d_hf.py","file_url":"https://github.com/deepglint/unicom/blob/HEAD/mlcd/vit_rope2d_hf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"47b28dd1d00a8546"}},{"code_sha256_prefix":"b17ea9baa7a3c22c","entry":"MLCDVisionModel","repo":"deepglint/unicom","repo_kind":"official","path":"mlcd/vit_rope2d_hf.py","file_url":"https://github.com/deepglint/unicom/blob/HEAD/mlcd/vit_rope2d_hf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b17ea9baa7a3c22c"}},{"code_sha256_prefix":"453f2f4154c6fa96","entry":"MLCDVisionTransformer","repo":"deepglint/unicom","repo_kind":"official","path":"mlcd/vit_rope2d_hf.py","file_url":"https://github.com/deepglint/unicom/blob/HEAD/mlcd/vit_rope2d_hf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"453f2f4154c6fa96"}},{"code_sha256_prefix":"9cbc76fa9f2e80cb","entry":"apply_rotary_pos_emb_vision","repo":"deepglint/unicom","repo_kind":"official","path":"mlcd/vit_rope2d_hf.py","file_url":"https://github.com/deepglint/unicom/blob/HEAD/mlcd/vit_rope2d_hf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9cbc76fa9f2e80cb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}