{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/histgen-histopathology-report-generation-via","title":"HistGen: Histopathology Report Generation via Local-Global Feature Encoding and Cross-modal Context Interaction","arxiv_id":"2403.05396","date":"2024-03-08","proceeding":null,"authors":["Zhengrui Guo","Jiabo Ma","Yingxue Xu","Yihui Wang","Liansheng Wang","Hao Chen"],"abstract":"Histopathology serves as the gold standard in cancer diagnosis, with clinical reports being vital in interpreting and understanding this process, guiding cancer treatment and patient care. The automation of histopathology report generation with deep learning stands to significantly enhance clinical efficiency and lessen the labor-intensive, time-consuming burden on pathologists in report writing. In pursuit of this advancement, we introduce HistGen, a multiple instance learning-empowered framework for histopathology report generation together with the first benchmark dataset for evaluation. Inspired by diagnostic and report-writing workflows, HistGen features two delicately designed modules, aiming to boost report generation by aligning whole slide images (WSIs) and diagnostic reports from local and global granularity. To achieve this, a local-global hierarchical encoder is developed for efficient visual feature aggregation from a region-to-slide perspective. Meanwhile, a cross-modal context module is proposed to explicitly facilitate alignment and interaction between distinct modalities, effectively bridging the gap between the extensive visual sequences of WSIs and corresponding highly summarized reports. Experimental results on WSI report generation show the proposed model outperforms state-of-the-art (SOTA) models by a large margin. Moreover, the results of fine-tuning our model on cancer subtyping and survival analysis tasks further demonstrate superior performance compared to SOTA methods, showcasing strong transfer learning capability. Dataset, model weights, and source code are available in https://github.com/dddavid4real/HistGen.","url_abs":"https://arxiv.org/abs/2403.05396v2","url_pdf":"https://arxiv.org/pdf/2403.05396v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"histgen-histopathology-report-generation-via","repo_url":"https://github.com/dddavid4real/HistGen","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"diagnostic","task_name":"Diagnostic"},{"task_slug":"medical-report-generation","task_name":"Medical Report Generation"},{"task_slug":"multiple-instance-learning","task_name":"Multiple Instance Learning"},{"task_slug":"survival-analysis","task_name":"Survival Analysis"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"},{"task_slug":"whole-slide-images","task_name":"whole slide images"}],"methods":[],"datasets_introduced":[{"slug":"histgen-wsi-report-dataset","name":"HistGen WSI-Report Dataset","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/medical-report-generation-on-histgen-wsi","task":"Medical Report Generation","dataset":"HistGen WSI-Report Dataset","model":"HistGen","rank_in_archive_order":1,"of":1,"metrics":{"BLEU-4":"0.184"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2403.05396","atlas_url":"https://app.syntology.ai/?focus=2403.05396","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.05396"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dddavid4real/HistGen","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":4,"ran_draft_wrong":1,"ran_violates":1},"by_repo_kind":{"official":{"samples":6,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"269003e7c21d565d","entry":"attention","repo":"dddavid4real/HistGen","repo_kind":"official","path":"models/PlainTransformer_Modules.py","file_url":"https://github.com/dddavid4real/HistGen/blob/HEAD/models/PlainTransformer_Modules.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"269003e7c21d565d"}},{"code_sha256_prefix":"a3722169bbc81569","entry":"clones","repo":"dddavid4real/HistGen","repo_kind":"official","path":"models/PlainTransformer_Modules.py","file_url":"https://github.com/dddavid4real/HistGen/blob/HEAD/models/PlainTransformer_Modules.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a3722169bbc81569"}},{"code_sha256_prefix":"dd03765c7da9ca51","entry":"subsequent_mask","repo":"dddavid4real/HistGen","repo_kind":"official","path":"models/PlainTransformer_Modules.py","file_url":"https://github.com/dddavid4real/HistGen/blob/HEAD/models/PlainTransformer_Modules.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dd03765c7da9ca51"}},{"code_sha256_prefix":"ccfa98711a392912","entry":"transform_masks2regions","repo":"dddavid4real/HistGen","repo_kind":"official","path":"modules/histgen_module.py","file_url":"https://github.com/dddavid4real/HistGen/blob/HEAD/modules/histgen_module.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ccfa98711a392912"}},{"code_sha256_prefix":"4707a8b847159f66","entry":"transform_sentences2tokens","repo":"dddavid4real/HistGen","repo_kind":"official","path":"modules/histgen_module.py","file_url":"https://github.com/dddavid4real/HistGen/blob/HEAD/modules/histgen_module.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4707a8b847159f66"}},{"code_sha256_prefix":"7f66a357e52cacea","entry":"transform_tokens2regions","repo":"dddavid4real/HistGen","repo_kind":"official","path":"modules/histgen_module.py","file_url":"https://github.com/dddavid4real/HistGen/blob/HEAD/modules/histgen_module.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7f66a357e52cacea"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}