{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/astroclip-cross-modal-pre-training-for","title":"AstroCLIP: A Cross-Modal Foundation Model for Galaxies","arxiv_id":"2310.03024","date":"2023-10-04","proceeding":null,"authors":["Liam Parker","Francois Lanusse","Siavash Golkar","Leopoldo Sarra","Miles Cranmer","Alberto Bietti","Michael Eickenberg","Geraud Krawezik","Michael McCabe","Ruben Ohana","Mariel Pettee","Bruno Regaldo-Saint Blancard","Tiberiu Tesileanu","Kyunghyun Cho","Shirley Ho"],"abstract":"We present AstroCLIP, a single, versatile model that can embed both galaxy images and spectra into a shared, physically meaningful latent space. These embeddings can then be used - without any model fine-tuning - for a variety of downstream tasks including (1) accurate in-modality and cross-modality semantic similarity search, (2) photometric redshift estimation, (3) galaxy property estimation from both images and spectra, and (4) morphology classification. Our approach to implementing AstroCLIP consists of two parts. First, we embed galaxy images and spectra separately by pretraining separate transformer-based image and spectrum encoders in self-supervised settings. We then align the encoders using a contrastive loss. We apply our method to spectra from the Dark Energy Spectroscopic Instrument and images from its corresponding Legacy Imaging Survey. Overall, we find remarkable performance on all downstream tasks, even relative to supervised baselines. For example, for a task like photometric redshift prediction, we find similar performance to a specifically-trained ResNet18, and for additional tasks like physical property estimation (stellar mass, age, metallicity, and sSFR), we beat this supervised baseline by 19\\% in terms of $R^2$. We also compare our results to a state-of-the-art self-supervised single-modal model for galaxy images, and find that our approach outperforms this benchmark by roughly a factor of two on photometric redshift estimation and physical property prediction in terms of $R^2$, while remaining roughly in-line in terms of morphology classification. Ultimately, our approach represents the first cross-modal self-supervised model for galaxies, and the first self-supervised transformer-based architectures for galaxy images and spectra.","url_abs":"https://arxiv.org/abs/2310.03024v2","url_pdf":"https://arxiv.org/pdf/2310.03024v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"astroclip-cross-modal-pre-training-for","repo_url":"https://github.com/PolymathicAI/AstroCLIP","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"morphology-classification","task_name":"Morphology classification"},{"task_slug":"photometric-redshift-estimation","task_name":"Photometric Redshift Estimation"},{"task_slug":"property-prediction","task_name":"Property Prediction"},{"task_slug":"semantic-similarity","task_name":"Semantic Similarity"},{"task_slug":"semantic-textual-similarity","task_name":"Semantic Textual Similarity"},{"task_slug":"model","task_name":"model"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2310.03024","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.03024"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/PolymathicAI/AstroCLIP","reach":null}],"summary":{"ran":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"788c03b4f5ddff93","entry":"AstroClipModel","repo":"PolymathicAI/AstroCLIP","repo_kind":"official","path":"astroclip/models/astroclip.py","file_url":"https://github.com/PolymathicAI/AstroCLIP/blob/HEAD/astroclip/models/astroclip.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"788c03b4f5ddff93"}},{"code_sha256_prefix":"36c97be9f076b733","entry":"CLIPLoss","repo":"PolymathicAI/AstroCLIP","repo_kind":"official","path":"astroclip/models/astroclip.py","file_url":"https://github.com/PolymathicAI/AstroCLIP/blob/HEAD/astroclip/models/astroclip.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"36c97be9f076b733"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}