{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cellvit-vision-transformers-for-precise-cell","title":"CellViT: Vision Transformers for Precise Cell Segmentation and Classification","arxiv_id":"2306.15350","date":"2023-06-27","proceeding":null,"authors":["Fabian Hörst","Moritz Rempe","Lukas Heine","Constantin Seibold","Julius Keyl","Giulia Baldini","Selma Ugurel","Jens Siveke","Barbara Grünwald","Jan Egger","Jens Kleesiek"],"abstract":"Nuclei detection and segmentation in hematoxylin and eosin-stained (H&E) tissue images are important clinical tasks and crucial for a wide range of applications. However, it is a challenging task due to nuclei variances in staining and size, overlapping boundaries, and nuclei clustering. While convolutional neural networks have been extensively used for this task, we explore the potential of Transformer-based networks in this domain. Therefore, we introduce a new method for automated instance segmentation of cell nuclei in digitized tissue samples using a deep learning architecture based on Vision Transformer called CellViT. CellViT is trained and evaluated on the PanNuke dataset, which is one of the most challenging nuclei instance segmentation datasets, consisting of nearly 200,000 annotated Nuclei into 5 clinically important classes in 19 tissue types. We demonstrate the superiority of large-scale in-domain and out-of-domain pre-trained Vision Transformers by leveraging the recently published Segment Anything Model and a ViT-encoder pre-trained on 104 million histological image patches - achieving state-of-the-art nuclei detection and instance segmentation performance on the PanNuke dataset with a mean panoptic quality of 0.50 and an F1-detection score of 0.83. The code is publicly available at https://github.com/TIO-IKIM/CellViT","url_abs":"https://arxiv.org/abs/2306.15350v2","url_pdf":"https://arxiv.org/pdf/2306.15350v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cellvit-vision-transformers-for-precise-cell","repo_url":"https://github.com/tio-ikim/cellvit","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"cellvit-vision-transformers-for-precise-cell","repo_url":"https://github.com/junlinguo/cellvit-kidney","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"cellvit-vision-transformers-for-precise-cell","repo_url":"https://github.com/philippendres/CellPilot","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"cell-detection","task_name":"Cell Detection"},{"task_slug":"cell-segmentation","task_name":"Cell Segmentation"},{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"instance-segmentation","task_name":"Instance Segmentation"},{"task_slug":"panoptic-segmentation","task_name":"Panoptic Segmentation"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/cell-detection-on-pannuke","task":"Cell Detection","dataset":"PanNuke","model":"CellViT-SAM-H","rank_in_archive_order":1,"of":1,"metrics":{"Average F1":"0.83","Average Precision":"0.84","Average Recall":"0.81"},"uses_additional_data":false},{"leaderboard":"/sota/panoptic-segmentation-on-pannuke","task":"Panoptic Segmentation","dataset":"PanNuke","model":"CellViT-SAM-H","rank_in_archive_order":2,"of":4,"metrics":{"PQ":"50.62"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2306.15350","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.15350"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tio-ikim/cellvit","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/junlinguo/cellvit-kidney","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/philippendres/CellPilot","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":1},"by_repo_kind":{"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"e8c211a8a925525e","entry":"find_files","repo":"junlinguo/cellvit-kidney","repo_kind":"listed","path":"cell_segmentation/inference/inference_cellvit_experiment_kidney.py","file_url":"https://github.com/junlinguo/cellvit-kidney/blob/HEAD/cell_segmentation/inference/inference_cellvit_experiment_kidney.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e8c211a8a925525e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}