</> Code with Papers by Syntology

Papers › Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Study

Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Study

21 Apr 2023arXiv:2304.10909archive 2025-07-28

Joakim Edin, Alexander Junge, Jakob D. Havtorn, Lasse Borgholt, Maria Maistro, Tuukka Ruotsalo, Lars Maaløe

Medical coding is the task of assigning medical codes to clinical free-text documentation. Healthcare professionals manually assign such codes to track patient diagnoses and treatments. Automated medical coding can considerably alleviate this administrative burden. In this paper, we reproduce, compare, and analyze state-of-the-art automated medical coding machine learning models. We show that several models underperform due to weak configurations, poorly sampled train-test splits, and insufficient evaluation. In previous work, the macro F1 score has been calculated sub-optimally, and our correction doubles it. We contribute a revised model comparison using stratified sampling and identical experimental setups, including hyperparameters and decision boundary tuning. We analyze prediction errors to validate and falsify assumptions of previous works. The analysis confirms that all models struggle with rare codes, while long documents only have a negligible impact. Finally, we present the first comprehensive results on the newly released MIMIC-IV dataset using the reproduced models. We release our code, model parameters, and new MIMIC-III and MIMIC-IV training and evaluation pipelines to accommodate fair future comparisons.

PaperPDFCode

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

joakimedin/medical-coding-reproducibility officialmentioned in papermentioned on GitHubpytorch report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Medical Code Prediction

Datasets

Introduced by this paper, per the archive.

MIMIC-IV ICD-10MIMIC-IV ICD-9

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Medical Code Prediction MIMIC-IV ICD-10 PLM-ICD AUC Macro 96.6 #1 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 PLM-ICD AUC Micro 99.2 #1 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 PLM-ICD Exact Match Ratio 0.4 #1 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 PLM-ICD F1 Macro 21.1 #1 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 PLM-ICD F1 Micro 58.5 #1 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 PLM-ICD Precision@15 55.0 #1 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 PLM-ICD Precision@8 69.9 #1 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 PLM-ICD R-Prec 57.9 #1 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 PLM-ICD mAP 61.9 #1 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 LAAT AUC Macro 95.4 #2 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 LAAT AUC Micro 99.0 #2 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 LAAT Exact Match Ratio 0.4 #2 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 LAAT F1 Macro 20.3 #2 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 LAAT F1 Micro 57.9 #2 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 LAAT Precision@15 54.3 #2 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 LAAT Precision@8 68.9 #2 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 LAAT R-Prec 57.2 #2 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 LAAT mAP 60.6 #2 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 MultiResCNN AUC Macro 94.5 #3 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 MultiResCNN AUC Micro 99.0 #3 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 MultiResCNN Exact Match Ratio 0.4 #3 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 MultiResCNN F1 Macro 21.1 #3 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 MultiResCNN F1 Micro 56.9 #3 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 MultiResCNN Precision@15 53.5 #3 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 MultiResCNN Precision@8 67.8 #3 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 MultiResCNN R-Prec 56.1 #3 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 MultiResCNN mAP 59.3 #3 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 CAML AUC Macro 91.1 #4 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 CAML AUC Micro 98.5 #4 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 CAML Exact Match Ratio 0.3 #4 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 CAML F1 Macro 16.0 #4 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 CAML F1 Micro 55.4 #4 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 CAML Precision@15 52.2 #4 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 CAML Precision@8 66.8 #4 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 CAML R-Prec 54.5 #4 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 CAML mAP 57.4 #4 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 Bi-GRU AUC Macro 92.4 #5 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 Bi-GRU AUC Micro 98.3 #5 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 Bi-GRU Exact Match Ratio 0.3 #5 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 Bi-GRU F1 Macro 10.6 #5 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 Bi-GRU F1 Micro 50.1 #5 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 Bi-GRU Precision@15 47.7 #5 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 Bi-GRU Precision@8 62.6 #5 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 Bi-GRU R-Prec 49.6 #5 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 Bi-GRU mAP 51.1 #5 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 CNN AUC Macro 87.9 #6 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 CNN AUC Micro 97.5 #6 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 CNN Exact Match Ratio 0.3 #6 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 CNN F1 Macro 8.0 #6 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 CNN F1 Micro 47.2 #6 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 CNN Precision@15 45.7 #6 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 CNN Precision@8 60.3 #6 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 CNN R-Prec 47.3 #6 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-10 CNN mAP 48.2 #6 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 PLM-ICD AUC Macro 97.2 #1 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 PLM-ICD AUC Micro 99.4 #1 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 PLM-ICD Exact Match Ratio 1.0 #1 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 PLM-ICD F1 Macro 29.8 #1 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 PLM-ICD F1 Micro 62.6 #1 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 PLM-ICD Precision@15 53.5 #1 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 PLM-ICD Precision@8 70.0 #1 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 PLM-ICD R-Prec 62.7 #1 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 PLM-ICD mAP 68.0 #1 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 LAAT AUC Macro 96.0 #2 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 LAAT AUC Micro 99.3 #2 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 LAAT Exact Match Ratio 0.9 #2 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 LAAT F1 Macro 26.4 #2 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 LAAT F1 Micro 61.7 #2 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 LAAT Precision@15 52.7 #2 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 LAAT Precision@8 68.9 #2 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 LAAT R-Prec 61.7 #2 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 LAAT mAP 66.3 #2 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 MultiResCNN AUC Macro 95.1 #3 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 MultiResCNN AUC Micro 99.2 #3 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 MultiResCNN Exact Match Ratio 0.8 #3 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 MultiResCNN F1 Macro 27.7 #3 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 MultiResCNN F1 Micro 60.4 #3 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 MultiResCNN Precision@15 51.8 #3 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 MultiResCNN Precision@8 67.6 #3 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 MultiResCNN R-Prec 60.4 #3 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 MultiResCNN mAP 64.7 #3 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 Bi-GRU AUC Macro 93.8 #4 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 Bi-GRU AUC Micro 98.8 #4 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 Bi-GRU Exact Match Ratio 0.7 #4 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 Bi-GRU F1 Macro 16.6 #4 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 Bi-GRU F1 Micro 55.5 #4 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 Bi-GRU Precision@15 47.8 #4 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 Bi-GRU Precision@8 64.1 #4 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 Bi-GRU R-Prec 55.8 #4 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 Bi-GRU mAP 58.9 #4 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 CAML AUC Macro 90.7 #5 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 CAML AUC Micro 98.8 #5 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 CAML Exact Match Ratio 0.6 #5 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 CAML F1 Macro 19.3 #5 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 CAML F1 Micro 58.6 #5 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 CAML Precision@15 50.3 #5 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 CAML Precision@8 66.3 #5 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 CAML R-Prec 58.5 #5 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 CAML mAP 62.4 #5 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 CNN AUC Macro 89.4 #6 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 CNN AUC Micro 98.1 #6 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 CNN Exact Match Ratio 0.6 #6 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 CNN F1 Macro 12.6 #6 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 CNN F1 Micro 52.4 #6 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 CNN Precision@15 45.6 #6 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 CNN Precision@8 61.3 #6 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 CNN R-Prec 52.9 #6 of 6 Archive leaderboard report
Medical Code Prediction MIMIC-IV ICD-9 CNN mAP 55.2 #6 of 6 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections