{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/evaluating-representation-learning-on-the","title":"Evaluating representation learning on the protein structure universe","arxiv_id":"2406.13864","date":"2024-06-19","proceeding":null,"authors":["Arian R. Jamasb","Alex Morehead","Chaitanya K. Joshi","Zuobai Zhang","Kieran Didi","Simon V. Mathis","Charles Harris","Jian Tang","Jianlin Cheng","Pietro Lio","Tom L. Blundell"],"abstract":"We introduce ProteinWorkshop, a comprehensive benchmark suite for representation learning on protein structures with Geometric Graph Neural Networks. We consider large-scale pre-training and downstream tasks on both experimental and predicted structures to enable the systematic evaluation of the quality of the learned structural representation and their usefulness in capturing functional relationships for downstream tasks. We find that: (1) large-scale pretraining on AlphaFold structures and auxiliary tasks consistently improve the performance of both rotation-invariant and equivariant GNNs, and (2) more expressive equivariant GNNs benefit from pretraining to a greater extent compared to invariant models. We aim to establish a common ground for the machine learning and computational biology communities to rigorously compare and advance protein structure representation learning. Our open-source codebase reduces the barrier to entry for working with large protein structure datasets by providing: (1) storage-efficient dataloaders for large-scale structural databases including AlphaFoldDB and ESM Atlas, as well as (2) utilities for constructing new tasks from the entire PDB. ProteinWorkshop is available at: github.com/a-r-j/ProteinWorkshop.","url_abs":"https://arxiv.org/abs/2406.13864v1","url_pdf":"https://arxiv.org/pdf/2406.13864v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"evaluating-representation-learning-on-the","repo_url":"https://github.com/a-r-j/proteinworkshop","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[{"method_slug":"alphafold","method_name":"AlphaFold"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.13864","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.13864"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/a-r-j/ProteinWorkshop","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/LPDI-EPFL/masif","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/a-r-j/proteinworkshop","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":13,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":8,"ran":8,"repositories":1},"found_in_text":{"samples":8,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c038d262db53bffe","entry":"compute_dists","repo":"LPDI-EPFL/masif","repo_kind":"found_in_text","path":"source/masif_modules/train_ppi_search.py","file_url":"https://github.com/LPDI-EPFL/masif/blob/HEAD/source/masif_modules/train_ppi_search.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c038d262db53bffe"}},{"code_sha256_prefix":"7462f95ee754ba4c","entry":"construct_batch","repo":"LPDI-EPFL/masif","repo_kind":"found_in_text","path":"source/masif_modules/train_ppi_search.py","file_url":"https://github.com/LPDI-EPFL/masif/blob/HEAD/source/masif_modules/train_ppi_search.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7462f95ee754ba4c"}},{"code_sha256_prefix":"c921767debc219a9","entry":"construct_batch_val_test","repo":"LPDI-EPFL/masif","repo_kind":"found_in_text","path":"source/masif_modules/train_ppi_search.py","file_url":"https://github.com/LPDI-EPFL/masif/blob/HEAD/source/masif_modules/train_ppi_search.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c921767debc219a9"}},{"code_sha256_prefix":"7b8f9a99ba265533","entry":"default","repo":"a-r-j/ProteinWorkshop","repo_kind":"official","path":"proteinworkshop/models/utils.py","file_url":"https://github.com/a-r-j/ProteinWorkshop/blob/HEAD/proteinworkshop/models/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b8f9a99ba265533"}},{"code_sha256_prefix":"77f79a4a1c1acb84","entry":"exists","repo":"a-r-j/ProteinWorkshop","repo_kind":"official","path":"proteinworkshop/models/utils.py","file_url":"https://github.com/a-r-j/ProteinWorkshop/blob/HEAD/proteinworkshop/models/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"77f79a4a1c1acb84"}},{"code_sha256_prefix":"09ec7a346cca7bcd","entry":"get_data_path","repo":"a-r-j/ProteinWorkshop","repo_kind":"official","path":"proteinworkshop/datasets/atom3d_datamodule.py","file_url":"https://github.com/a-r-j/ProteinWorkshop/blob/HEAD/proteinworkshop/datasets/atom3d_datamodule.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"09ec7a346cca7bcd"}},{"code_sha256_prefix":"6f28864098d23c94","entry":"get_task_split","repo":"a-r-j/ProteinWorkshop","repo_kind":"official","path":"proteinworkshop/datasets/atom3d_datamodule.py","file_url":"https://github.com/a-r-j/ProteinWorkshop/blob/HEAD/proteinworkshop/datasets/atom3d_datamodule.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6f28864098d23c94"}},{"code_sha256_prefix":"69b7c9afed7ab8de","entry":"get_test_data_path","repo":"a-r-j/ProteinWorkshop","repo_kind":"official","path":"proteinworkshop/datasets/atom3d_datamodule.py","file_url":"https://github.com/a-r-j/ProteinWorkshop/blob/HEAD/proteinworkshop/datasets/atom3d_datamodule.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"69b7c9afed7ab8de"}},{"code_sha256_prefix":"5b87cb0fe092cb16","entry":"load_config_from_checkpoint","repo":"a-r-j/ProteinWorkshop","repo_kind":"official","path":"proteinworkshop/model_io.py","file_url":"https://github.com/a-r-j/ProteinWorkshop/blob/HEAD/proteinworkshop/model_io.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5b87cb0fe092cb16"}},{"code_sha256_prefix":"de681cc738ca5b43","entry":"mask_input_feat","repo":"LPDI-EPFL/masif","repo_kind":"found_in_text","path":"source/masif_modules/train_masif_site.py","file_url":"https://github.com/LPDI-EPFL/masif/blob/HEAD/source/masif_modules/train_masif_site.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"de681cc738ca5b43"}},{"code_sha256_prefix":"f2fe51a4a5646c6f","entry":"max_neg_value","repo":"a-r-j/ProteinWorkshop","repo_kind":"official","path":"proteinworkshop/models/utils.py","file_url":"https://github.com/a-r-j/ProteinWorkshop/blob/HEAD/proteinworkshop/models/utils.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f2fe51a4a5646c6f"}},{"code_sha256_prefix":"fa57bb24283f3605","entry":"pad_indices","repo":"LPDI-EPFL/masif","repo_kind":"found_in_text","path":"source/masif_modules/train_masif_site.py","file_url":"https://github.com/LPDI-EPFL/masif/blob/HEAD/source/masif_modules/train_masif_site.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fa57bb24283f3605"}},{"code_sha256_prefix":"851def46ba261ee9","entry":"run_masif_site","repo":"LPDI-EPFL/masif","repo_kind":"found_in_text","path":"source/masif_modules/train_masif_site.py","file_url":"https://github.com/LPDI-EPFL/masif/blob/HEAD/source/masif_modules/train_masif_site.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"851def46ba261ee9"}},{"code_sha256_prefix":"c7130aa8aaddfd58","entry":"str2bool","repo":"a-r-j/ProteinWorkshop","repo_kind":"official","path":"proteinworkshop/datasets/flip_datamodule.py","file_url":"https://github.com/a-r-j/ProteinWorkshop/blob/HEAD/proteinworkshop/datasets/flip_datamodule.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c7130aa8aaddfd58"}},{"code_sha256_prefix":"914a02a1cfbb46e1","entry":"extract_patch_and_coord","repo":"LPDI-EPFL/masif","repo_kind":"found_in_text","path":"source/masif_modules/read_data_from_surface.py","file_url":"https://github.com/LPDI-EPFL/masif/blob/HEAD/source/masif_modules/read_data_from_surface.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"914a02a1cfbb46e1"}},{"code_sha256_prefix":"1f11ce42bb37e65d","entry":"get_masif_ligand_prediction","repo":"LPDI-EPFL/masif","repo_kind":"found_in_text","path":"comparison/masif_ligand/parse_json_tmscore_rocauc.py","file_url":"https://github.com/LPDI-EPFL/masif/blob/HEAD/comparison/masif_ligand/parse_json_tmscore_rocauc.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1f11ce42bb37e65d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}