{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/prioritized-training-on-points-that-are-1","title":"Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt","arxiv_id":"2206.07137","date":"2022-06-14","proceeding":null,"authors":["Sören Mindermann","Jan Brauner","Muhammed Razzak","Mrinank Sharma","Andreas Kirsch","Winnie Xu","Benedikt Höltgen","Aidan N. Gomez","Adrien Morisot","Sebastian Farquhar","Yarin Gal"],"abstract":"Training on web-scale data can take months. But most computation and time is wasted on redundant and noisy points that are already learnt or not learnable. To accelerate training, we introduce Reducible Holdout Loss Selection (RHO-LOSS), a simple but principled technique which selects approximately those points for training that most reduce the model's generalization loss. As a result, RHO-LOSS mitigates the weaknesses of existing data selection methods: techniques from the optimization literature typically select 'hard' (e.g. high loss) points, but such points are often noisy (not learnable) or less task-relevant. Conversely, curriculum learning prioritizes 'easy' points, but such points need not be trained on once learned. In contrast, RHO-LOSS selects points that are learnable, worth learning, and not yet learnt. RHO-LOSS trains in far fewer steps than prior art, improves accuracy, and speeds up training on a wide range of datasets, hyperparameters, and architectures (MLPs, CNNs, and BERT). On the large web-scraped image dataset Clothing-1M, RHO-LOSS trains in 18x fewer steps and reaches 2% higher final accuracy than uniform data shuffling.","url_abs":"https://arxiv.org/abs/2206.07137v3","url_pdf":"https://arxiv.org/pdf/2206.07137v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"prioritized-training-on-points-that-are-1","repo_url":"https://github.com/oatml/rho-loss","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"prioritized-training-on-points-that-are-1","repo_url":"https://github.com/williambankes/REDUCR","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2206.07137","atlas_url":"https://app.syntology.ai/?focus=2206.07137","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.07137"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/williambankes/REDUCR","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/oatml/rho-loss","reach":null}],"summary":{"ran_fixture":1,"ran_draft_wrong":3,"ran":1,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1},"listed":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"efaf909cebf5b06e","entry":"_compute_irreducible_loss","repo":"oatml/rho-loss","repo_kind":"official","path":"src/curricula/selection_methods.py","file_url":"https://github.com/oatml/rho-loss/blob/HEAD/src/curricula/selection_methods.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"efaf909cebf5b06e"}},{"code_sha256_prefix":"370b75cc1c56ab2d","entry":"_compute_irreducible_loss","repo":"williambankes/REDUCR","repo_kind":"listed","path":"src/curricula/selection_methods.py","file_url":"https://github.com/williambankes/REDUCR/blob/HEAD/src/curricula/selection_methods.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"370b75cc1c56ab2d"}},{"code_sha256_prefix":"139838ce03447871","entry":"create_logging_dict","repo":"oatml/rho-loss","repo_kind":"official","path":"src/curricula/selection_methods.py","file_url":"https://github.com/oatml/rho-loss/blob/HEAD/src/curricula/selection_methods.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"139838ce03447871"}},{"code_sha256_prefix":"fc555220bc6d965c","entry":"reducible_loss_selection","repo":"oatml/rho-loss","repo_kind":"official","path":"src/curricula/selection_methods.py","file_url":"https://github.com/oatml/rho-loss/blob/HEAD/src/curricula/selection_methods.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fc555220bc6d965c"}},{"code_sha256_prefix":"b5032e9d522ad22f","entry":"top_x_indices","repo":"oatml/rho-loss","repo_kind":"official","path":"src/curricula/selection_methods.py","file_url":"https://github.com/oatml/rho-loss/blob/HEAD/src/curricula/selection_methods.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b5032e9d522ad22f"}},{"code_sha256_prefix":"c41ea49af5734870","entry":"reducible_loss_selection","repo":"williambankes/REDUCR","repo_kind":"listed","path":"src/curricula/selection_methods.py","file_url":"https://github.com/williambankes/REDUCR/blob/HEAD/src/curricula/selection_methods.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c41ea49af5734870"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}