Papers › Unsupervised Pre-Training on Patient Population Graphs for Patient-Level Predictions

Unsupervised Pre-Training on Patient Population Graphs for Patient-Level Predictions

23 Mar 2022arXiv:2203.12616archive 2025-07-28

Chantal Pellegrini, Anees Kazi, Nassir Navab

Pre-training has shown success in different areas of machine learning, such as Computer Vision (CV), Natural Language Processing (NLP) and medical imaging. However, it has not been fully explored for clinical data analysis. Even though an immense amount of Electronic Health Record (EHR) data is recorded, data and labels can be scarce if the data is collected in small hospitals or deals with rare diseases. In such scenarios, pre-training on a larger set of EHR data could improve the model performance. In this paper, we apply unsupervised pre-training to heterogeneous, multi-modal EHR data for patient outcome prediction. To model this data, we leverage graph deep learning over population graphs. We first design a network architecture based on graph transformer designed to handle various input feature types occurring in EHR data, like continuous, discrete, and time-series features, allowing better multi-modal data fusion. Further, we design pre-training methods based on masked imputation to pre-train our network before fine-tuning on different end tasks. Pre-training is done in a fully unsupervised fashion, which lays the groundwork for pre-training on large public datasets with different tasks and similar modalities in the future. We test our method on two medical datasets of patient records, TADPOLE and MIMIC-III, including imaging and non-imaging features and different prediction tasks. We find that our proposed graph based pre-training method helps in modeling the data at a population level and further improves performance on the fine tuning tasks in terms of AUC on average by 4.15% for MIMIC and 7.64% for TADPOLE.

Paper PDF Code

Code

chantalmp/unsupervised-pre-training-on-patient-population-graphs-for-patient-level-predictions officialmentioned in papermentioned on GitHubpytorch report

chantalmp/unsupervised_pre-training_of_graph_transformers_on_patient_population_graphs mentioned on GitHubpytorch report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Disease Prediction Imputation Length-of-Stay prediction Unsupervised Pre-training

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Length-of-Stay prediction	MIMIC-III	EHR-Graph Transformer (pre-trained)	Accuracy (LOS>3 Days)	71.4%	#1 of 5	Archive leaderboard	report
Length-of-Stay prediction	MIMIC-III	EHR-Graph Transformer	Accuracy (LOS>3 Days)	70.3%	#2 of 5	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Absolute Position Encodings Adam Attention BPE Dense Connections Dropout Graph Transformer Label Smoothing LapEigen Laplacian PE Layer Normalization Linear Layer Multi-Head Attention Position-Wise Feed-Forward Layer Residual Connection Softmax Transformer

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections