{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ratt-leveraging-unlabeled-data-to-guarantee","title":"RATT: Leveraging Unlabeled Data to Guarantee Generalization","arxiv_id":"2105.00303","date":"2021-05-01","proceeding":null,"authors":["Saurabh Garg","Sivaraman Balakrishnan","J. Zico Kolter","Zachary C. Lipton"],"abstract":"To assess generalization, machine learning scientists typically either (i) bound the generalization gap and then (after training) plug in the empirical risk to obtain a bound on the true risk; or (ii) validate empirically on holdout data. However, (i) typically yields vacuous guarantees for overparameterized models. Furthermore, (ii) shrinks the training set and its guarantee erodes with each re-use of the holdout set. In this paper, we introduce a method that leverages unlabeled data to produce generalization bounds. After augmenting our (labeled) training set with randomly labeled fresh examples, we train in the standard fashion. Whenever classifiers achieve low error on clean data and high error on noisy data, our bound provides a tight upper bound on the true risk. We prove that our bound is valid for 0-1 empirical risk minimization and with linear classifiers trained by gradient descent. Our approach is especially useful in conjunction with deep learning due to the early learning phenomenon whereby networks fit true labels before noisy labels but requires one intuitive assumption. Empirically, on canonical computer vision and NLP tasks, our bound provides non-vacuous generalization guarantees that track actual performance closely. This work provides practitioners with an option for certifying the generalization of deep nets even when unseen labeled data is unavailable and provides theoretical insights into the relationship between random label noise and generalization.","url_abs":"https://arxiv.org/abs/2105.00303v2","url_pdf":"https://arxiv.org/pdf/2105.00303v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ratt-leveraging-unlabeled-data-to-guarantee","repo_url":"https://github.com/acmi-lab/ratt_generalization_bound","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"generalization-bounds","task_name":"Generalization Bounds"},{"task_slug":"holdout-set","task_name":"Holdout Set"},{"task_slug":null,"task_name":"valid"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2105.00303","atlas_url":"https://app.syntology.ai/?focus=2105.00303","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2105.00303"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/acmilab/RATT","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/acmi-lab/ratt_generalization_bound","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":1,"unverified":15},"by_repo_kind":{"official":{"samples":16,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8d7ec010c29e813b","entry":"format_time","repo":"acmi-lab/ratt_generalization_bound","repo_kind":"official","path":"utils.py","file_url":"https://github.com/acmi-lab/ratt_generalization_bound/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8d7ec010c29e813b"}},{"code_sha256_prefix":"ee3a23d9284921b0","entry":"ResNet18","repo":"acmi-lab/ratt_generalization_bound","repo_kind":"official","path":"models/resnet.py","file_url":"https://github.com/acmi-lab/ratt_generalization_bound/blob/HEAD/models/resnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ee3a23d9284921b0"}},{"code_sha256_prefix":"33c9f87823514bad","entry":"custom_dataset","repo":"acmi-lab/ratt_generalization_bound","repo_kind":"official","path":"data_helper.py","file_url":"https://github.com/acmi-lab/ratt_generalization_bound/blob/HEAD/data_helper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"33c9f87823514bad"}},{"code_sha256_prefix":"5bbd31590c16d155","entry":"download_and_load_datasets","repo":"acmi-lab/ratt_generalization_bound","repo_kind":"official","path":"train_imdb_elmo.py","file_url":"https://github.com/acmi-lab/ratt_generalization_bound/blob/HEAD/train_imdb_elmo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5bbd31590c16d155"}},{"code_sha256_prefix":"08da3d75f4a68eb5","entry":"getBertTokenizer","repo":"acmi-lab/ratt_generalization_bound","repo_kind":"official","path":"datasets/IMDb.py","file_url":"https://github.com/acmi-lab/ratt_generalization_bound/blob/HEAD/datasets/IMDb.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"08da3d75f4a68eb5"}},{"code_sha256_prefix":"7bd3a77007a65eec","entry":"get_mean_and_std","repo":"acmi-lab/ratt_generalization_bound","repo_kind":"official","path":"utils.py","file_url":"https://github.com/acmi-lab/ratt_generalization_bound/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7bd3a77007a65eec"}},{"code_sha256_prefix":"4e4f63cb91e4b0e0","entry":"get_model","repo":"acmi-lab/ratt_generalization_bound","repo_kind":"official","path":"model_helper.py","file_url":"https://github.com/acmi-lab/ratt_generalization_bound/blob/HEAD/model_helper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4e4f63cb91e4b0e0"}},{"code_sha256_prefix":"18d06a46390badba","entry":"get_train_data","repo":"acmi-lab/ratt_generalization_bound","repo_kind":"official","path":"data_helper.py","file_url":"https://github.com/acmi-lab/ratt_generalization_bound/blob/HEAD/data_helper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"18d06a46390badba"}},{"code_sha256_prefix":"b24bb0be9085efab","entry":"initialize_bert_based_model","repo":"acmi-lab/ratt_generalization_bound","repo_kind":"official","path":"models/bert.py","file_url":"https://github.com/acmi-lab/ratt_generalization_bound/blob/HEAD/models/bert.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b24bb0be9085efab"}},{"code_sha256_prefix":"bb921ba6a0704620","entry":"initialize_bert_transform","repo":"acmi-lab/ratt_generalization_bound","repo_kind":"official","path":"datasets/IMDb.py","file_url":"https://github.com/acmi-lab/ratt_generalization_bound/blob/HEAD/datasets/IMDb.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bb921ba6a0704620"}},{"code_sha256_prefix":"c7a16597f295a1ba","entry":"load_dataset","repo":"acmi-lab/ratt_generalization_bound","repo_kind":"official","path":"train_imdb_elmo.py","file_url":"https://github.com/acmi-lab/ratt_generalization_bound/blob/HEAD/train_imdb_elmo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c7a16597f295a1ba"}},{"code_sha256_prefix":"7b0fc787f8d444d1","entry":"load_directory_data","repo":"acmi-lab/ratt_generalization_bound","repo_kind":"official","path":"train_imdb_elmo.py","file_url":"https://github.com/acmi-lab/ratt_generalization_bound/blob/HEAD/train_imdb_elmo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7b0fc787f8d444d1"}},{"code_sha256_prefix":"7e5acdc16eb78bae","entry":"pred_accuracy","repo":"acmi-lab/ratt_generalization_bound","repo_kind":"official","path":"model_helper.py","file_url":"https://github.com/acmi-lab/ratt_generalization_bound/blob/HEAD/model_helper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7e5acdc16eb78bae"}},{"code_sha256_prefix":"5de1b3eb7678aa7c","entry":"random_label_noise","repo":"acmi-lab/ratt_generalization_bound","repo_kind":"official","path":"data_helper.py","file_url":"https://github.com/acmi-lab/ratt_generalization_bound/blob/HEAD/data_helper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5de1b3eb7678aa7c"}},{"code_sha256_prefix":"eeb3d9b0ff6d3654","entry":"read_file","repo":"acmi-lab/ratt_generalization_bound","repo_kind":"official","path":"plots_helper/plot_acc.py","file_url":"https://github.com/acmi-lab/ratt_generalization_bound/blob/HEAD/plots_helper/plot_acc.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eeb3d9b0ff6d3654"}},{"code_sha256_prefix":"e1e8c58b1243b4e0","entry":"read_imdb_split","repo":"acmi-lab/ratt_generalization_bound","repo_kind":"official","path":"datasets/IMDb.py","file_url":"https://github.com/acmi-lab/ratt_generalization_bound/blob/HEAD/datasets/IMDb.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e1e8c58b1243b4e0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}