{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-11764","title":"Decomposing the Generalization Gap in PROTAC Activity Prediction: Variance Attribution and the Inter-Laboratory Ceiling","arxiv_id":"2605.11764","date":"2026-05-12","proceeding":null,"authors":["Thor Klamt","Wolfgang Nejdl","Ming Tang"],"abstract":"Machine-learning predictors of biochemical activity often exhibit large random-split-to-leave-one-target-out generalisation gaps that have been documented but not decomposed. We frame this as an evaluation-science question and use targeted protein degradation as the empirical test bed. PROTACs (proteolysis-targeting chimeras) are heterobifunctional small molecules that induce targeted protein degradation, with more than forty candidates currently in clinical trials; published predictors report AUROC of 0.85 to 0.91 under random-split cross-validation, while the leave-one-target-out (LOTO) protocol of Ribes et al. reduces performance to approximately 0.67. Random splits reward within-target interpolation, whereas LOTO measures the novel-target prediction that de-novo design depends on. We decompose this gap and identify inter-laboratory measurement variance as the dominant component, anchored by a within-target cross-laboratory cascade bounding the inter-laboratory contribution at 0.124 AUROC, well above the 0.05 contribution from binarisation-threshold choice. Across eight published architectures and ESM-2 protein language models up to 3B parameters, LOTO AUROC plateaus near 0.67, with a comparable plateau under SMILES-level deduplication; a 21-dimensional 2000-trial hyperparameter optimisation cannot break this ceiling, and the rank-1 single-seed configuration regresses by 0.161 AUROC under multi-seed validation, matching a closed-form selection-bias prediction (Bailey and Lopez de Prado, 2014). Few-shot k=5 stratified per-target retraining combined with ADMET features lifts 65-target LOTO AUROC from 0.668 to 0.7050, and post-hoc Platt scaling recovers raw output to within the 0.05 well-calibrated threshold. We release PROTAC-Bench (10,748 measurements, 173 targets, 65 LOTO folds), the variance-decomposition framework, the per-target calibration protocol, and the evaluation code.","url_abs":"https://arxiv.org/abs/2605.11764","url_pdf":"https://arxiv.org/pdf/2605.11764","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.11764"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/ThorKlm/PROTAC-Bench","reach":null}],"summary":{"ran":19,"unverified":5},"by_repo_kind":{"found_in_text":{"samples":24,"ran":19,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":24,"samples":[{"code_sha256_prefix":"ae7c2260be8ee0e4","entry":"analyze_pocket_diversity","repo":"ThorKlm/PROTAC-Bench","repo_kind":"found_in_text","path":"baselines/deepprotacs_eval.py","file_url":"https://github.com/ThorKlm/PROTAC-Bench/blob/HEAD/baselines/deepprotacs_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ae7c2260be8ee0e4"}},{"code_sha256_prefix":"8b3541f249fe4e8f","entry":"bulk_fps","repo":"ThorKlm/PROTAC-Bench","repo_kind":"found_in_text","path":"src/fingerprints.py","file_url":"https://github.com/ThorKlm/PROTAC-Bench/blob/HEAD/src/fingerprints.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8b3541f249fe4e8f"}},{"code_sha256_prefix":"29cb7b75b3e65672","entry":"confidence_interval","repo":"ThorKlm/PROTAC-Bench","repo_kind":"found_in_text","path":"src/stats.py","file_url":"https://github.com/ThorKlm/PROTAC-Bench/blob/HEAD/src/stats.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"29cb7b75b3e65672"}},{"code_sha256_prefix":"fed1bb6371e9597b","entry":"format_result","repo":"ThorKlm/PROTAC-Bench","repo_kind":"found_in_text","path":"src/stats.py","file_url":"https://github.com/ThorKlm/PROTAC-Bench/blob/HEAD/src/stats.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fed1bb6371e9597b"}},{"code_sha256_prefix":"a3587119a8f8ef97","entry":"get_loto_folds","repo":"ThorKlm/PROTAC-Bench","repo_kind":"found_in_text","path":"src/data_utils.py","file_url":"https://github.com/ThorKlm/PROTAC-Bench/blob/HEAD/src/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a3587119a8f8ef97"}},{"code_sha256_prefix":"855de0a93ab56a7c","entry":"get_representative_targets","repo":"ThorKlm/PROTAC-Bench","repo_kind":"found_in_text","path":"baselines/chemprop_hpo.py","file_url":"https://github.com/ThorKlm/PROTAC-Bench/blob/HEAD/baselines/chemprop_hpo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"855de0a93ab56a7c"}},{"code_sha256_prefix":"a4604ed65bf0bec2","entry":"load_cached_results","repo":"ThorKlm/PROTAC-Bench","repo_kind":"found_in_text","path":"baselines/deepprotacs_eval.py","file_url":"https://github.com/ThorKlm/PROTAC-Bench/blob/HEAD/baselines/deepprotacs_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a4604ed65bf0bec2"}},{"code_sha256_prefix":"a66482b2d39cfeaf","entry":"load_dataset","repo":"ThorKlm/PROTAC-Bench","repo_kind":"found_in_text","path":"src/data_utils.py","file_url":"https://github.com/ThorKlm/PROTAC-Bench/blob/HEAD/src/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a66482b2d39cfeaf"}},{"code_sha256_prefix":"6b121ca3ff6d1dc2","entry":"load_dm_data","repo":"ThorKlm/PROTAC-Bench","repo_kind":"found_in_text","path":"baselines/dm_loto.py","file_url":"https://github.com/ThorKlm/PROTAC-Bench/blob/HEAD/baselines/dm_loto.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6b121ca3ff6d1dc2"}},{"code_sha256_prefix":"91458d823b67614d","entry":"load_plm_embeddings","repo":"ThorKlm/PROTAC-Bench","repo_kind":"found_in_text","path":"baselines/plm_scaling.py","file_url":"https://github.com/ThorKlm/PROTAC-Bench/blob/HEAD/baselines/plm_scaling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"91458d823b67614d"}},{"code_sha256_prefix":"382bc9e04e66faa1","entry":"loto_evaluate","repo":"ThorKlm/PROTAC-Bench","repo_kind":"found_in_text","path":"src/evaluation.py","file_url":"https://github.com/ThorKlm/PROTAC-Bench/blob/HEAD/src/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"382bc9e04e66faa1"}},{"code_sha256_prefix":"8141e1d925395b8d","entry":"one_hot","repo":"ThorKlm/PROTAC-Bench","repo_kind":"found_in_text","path":"baselines/gnn_baselines.py","file_url":"https://github.com/ThorKlm/PROTAC-Bench/blob/HEAD/baselines/gnn_baselines.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8141e1d925395b8d"}},{"code_sha256_prefix":"866a835b5642ea90","entry":"paired_wilcoxon","repo":"ThorKlm/PROTAC-Bench","repo_kind":"found_in_text","path":"src/stats.py","file_url":"https://github.com/ThorKlm/PROTAC-Bench/blob/HEAD/src/stats.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"866a835b5642ea90"}},{"code_sha256_prefix":"50668a29b064f865","entry":"predict","repo":"ThorKlm/PROTAC-Bench","repo_kind":"found_in_text","path":"baselines/rs_per_target_gnn.py","file_url":"https://github.com/ThorKlm/PROTAC-Bench/blob/HEAD/baselines/rs_per_target_gnn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"50668a29b064f865"}},{"code_sha256_prefix":"269ce5e588ebec42","entry":"random_cv","repo":"ThorKlm/PROTAC-Bench","repo_kind":"found_in_text","path":"src/evaluation.py","file_url":"https://github.com/ThorKlm/PROTAC-Bench/blob/HEAD/src/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"269ce5e588ebec42"}},{"code_sha256_prefix":"ac6b59df44704eb9","entry":"reduce_for_debug","repo":"ThorKlm/PROTAC-Bench","repo_kind":"found_in_text","path":"src/debug.py","file_url":"https://github.com/ThorKlm/PROTAC-Bench/blob/HEAD/src/debug.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ac6b59df44704eb9"}},{"code_sha256_prefix":"20bdd910d097f4a9","entry":"rf_model_fn","repo":"ThorKlm/PROTAC-Bench","repo_kind":"found_in_text","path":"baselines/plm_scaling.py","file_url":"https://github.com/ThorKlm/PROTAC-Bench/blob/HEAD/baselines/plm_scaling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"20bdd910d097f4a9"}},{"code_sha256_prefix":"5759036494bff808","entry":"rf_model_fn","repo":"ThorKlm/PROTAC-Bench","repo_kind":"found_in_text","path":"baselines/rf_morgan.py","file_url":"https://github.com/ThorKlm/PROTAC-Bench/blob/HEAD/baselines/rf_morgan.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5759036494bff808"}},{"code_sha256_prefix":"d8dd6ba0960d8ec2","entry":"train_one","repo":"ThorKlm/PROTAC-Bench","repo_kind":"found_in_text","path":"baselines/rs_per_target_gnn.py","file_url":"https://github.com/ThorKlm/PROTAC-Bench/blob/HEAD/baselines/rs_per_target_gnn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d8dd6ba0960d8ec2"}},{"code_sha256_prefix":"8f61115c606e45f0","entry":"attempt_evaluation","repo":"ThorKlm/PROTAC-Bench","repo_kind":"found_in_text","path":"baselines/deepprotacs_eval.py","file_url":"https://github.com/ThorKlm/PROTAC-Bench/blob/HEAD/baselines/deepprotacs_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8f61115c606e45f0"}},{"code_sha256_prefix":"1084da9f2753e177","entry":"dm_loto_evaluation","repo":"ThorKlm/PROTAC-Bench","repo_kind":"found_in_text","path":"baselines/dm_loto.py","file_url":"https://github.com/ThorKlm/PROTAC-Bench/blob/HEAD/baselines/dm_loto.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1084da9f2753e177"}},{"code_sha256_prefix":"e4b1e57ac71cbf56","entry":"evaluate_config","repo":"ThorKlm/PROTAC-Bench","repo_kind":"found_in_text","path":"baselines/chemprop_hpo.py","file_url":"https://github.com/ThorKlm/PROTAC-Bench/blob/HEAD/baselines/chemprop_hpo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e4b1e57ac71cbf56"}},{"code_sha256_prefix":"0e8bbdb8786a909e","entry":"make_dmpnn_cls","repo":"ThorKlm/PROTAC-Bench","repo_kind":"found_in_text","path":"baselines/chemprop_hpo.py","file_url":"https://github.com/ThorKlm/PROTAC-Bench/blob/HEAD/baselines/chemprop_hpo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0e8bbdb8786a909e"}},{"code_sha256_prefix":"9d1423753770cbe5","entry":"run_method","repo":"ThorKlm/PROTAC-Bench","repo_kind":"found_in_text","path":"baselines/rs_per_target_gnn.py","file_url":"https://github.com/ThorKlm/PROTAC-Bench/blob/HEAD/baselines/rs_per_target_gnn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9d1423753770cbe5"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}