{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/simulated-annealing-in-early-layers-leads-to","title":"Simulated Annealing in Early Layers Leads to Better Generalization","arxiv_id":"2304.04858","date":"2023-04-10","proceeding":"CVPR 2023 1","authors":["AmirMohammad Sarfi","Zahra Karimpour","Muawiz Chaudhary","Nasir M. Khalid","Mirco Ravanelli","Sudhir Mudur","Eugene Belilovsky"],"abstract":"Recently, a number of iterative learning methods have been introduced to improve generalization. These typically rely on training for longer periods of time in exchange for improved generalization. LLF (later-layer-forgetting) is a state-of-the-art method in this category. It strengthens learning in early layers by periodically re-initializing the last few layers of the network. Our principal innovation in this work is to use Simulated annealing in EArly Layers (SEAL) of the network in place of re-initialization of later layers. Essentially, later layers go through the normal gradient descent process, while the early layers go through short stints of gradient ascent followed by gradient descent. Extensive experiments on the popular Tiny-ImageNet dataset benchmark and a series of transfer learning and few-shot learning tasks show that we outperform LLF by a significant margin. We further show that, compared to normal training, LLF features, although improving on the target task, degrade the transfer learning performance across all datasets we explored. In comparison, our method outperforms LLF across the same target datasets by a large margin. We also show that the prediction depth of our method is significantly lower than that of LLF and normal training, indicating on average better prediction performance.","url_abs":"https://arxiv.org/abs/2304.04858v1","url_pdf":"https://arxiv.org/pdf/2304.04858v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"simulated-annealing-in-early-layers-leads-to","repo_url":"https://github.com/amiiir-sarfi/seal","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"few-shot-learning","task_name":"Few-Shot Learning"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2304.04858","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2304.04858"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/amiiir-sarfi/seal","reach":null}],"summary":{"ran":6,"ran_honours":2,"ran_draft_wrong":12,"ran_fixture":1,"unverified":16},"by_repo_kind":{"official":{"samples":35,"ran":20,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":37,"samples":[{"code_sha256_prefix":"402a4933d75f91b2","entry":"AverageMeter","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"402a4933d75f91b2"}},{"code_sha256_prefix":"77b9d81bc5e4dd19","entry":"BasicBlock","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"77b9d81bc5e4dd19"}},{"code_sha256_prefix":"966080dd9d9fa737","entry":"Bottleneck","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"966080dd9d9fa737"}},{"code_sha256_prefix":"ae4f25439b3acbb3","entry":"KDLoss","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ae4f25439b3acbb3"}},{"code_sha256_prefix":"afc0fd280b0b83c5","entry":"LabelSmoothing","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"afc0fd280b0b83c5"}},{"code_sha256_prefix":"ae4960bdf070be2e","entry":"PairBatchSampler","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ae4960bdf070be2e"}},{"code_sha256_prefix":"adb312560b8eca5d","entry":"_get_mean_std","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"adb312560b8eca5d"}},{"code_sha256_prefix":"60087ec002f63a64","entry":"_warmup_lr","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"60087ec002f63a64"}},{"code_sha256_prefix":"2d6895c1a9b20b6a","entry":"cifar_lr","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2d6895c1a9b20b6a"}},{"code_sha256_prefix":"c0f4ef67992f303a","entry":"constant_lr","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c0f4ef67992f303a"}},{"code_sha256_prefix":"e18ed73aa89f051a","entry":"conv3x3","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"e18ed73aa89f051a"}},{"code_sha256_prefix":"2e32661e8d23fbbb","entry":"cosine_lr","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2e32661e8d23fbbb"}},{"code_sha256_prefix":"4f6369d4ba085a0b","entry":"get_ascenders_descenders","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4f6369d4ba085a0b"}},{"code_sha256_prefix":"4e2cad7e754e6954","entry":"get_optimizer","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4e2cad7e754e6954"}},{"code_sha256_prefix":"84121d5fa2d05bf2","entry":"get_policy","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"84121d5fa2d05bf2"}},{"code_sha256_prefix":"ae403e09fc2a944f","entry":"get_transforms","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ae403e09fc2a944f"}},{"code_sha256_prefix":"1a15317faad06688","entry":"long_cosine_lr","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1a15317faad06688"}},{"code_sha256_prefix":"701988efefe74cd8","entry":"multistep_lr","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"701988efefe74cd8"}},{"code_sha256_prefix":"7a31c2db2967e1ca","entry":"reinitalize_forgetting","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7a31c2db2967e1ca"}},{"code_sha256_prefix":"ba2b2c184b211eff","entry":"step_lr","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ba2b2c184b211eff"}},{"code_sha256_prefix":"beefa7eb58aba900","entry":"val_dependent_lr","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"beefa7eb58aba900"}},{"code_sha256_prefix":"d7dfb645689ad0c6","entry":"IterativeTrainer","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d7dfb645689ad0c6"}},{"code_sha256_prefix":"de2f613a8c141264","entry":"Meter","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"de2f613a8c141264"}},{"code_sha256_prefix":"4da1826351727b34","entry":"ResNet","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4da1826351727b34"}},{"code_sha256_prefix":"0a92f9d39e77ee23","entry":"Stage","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0a92f9d39e77ee23"}},{"code_sha256_prefix":"fed6cc06ad4c821e","entry":"_reinit","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fed6cc06ad4c821e"}},{"code_sha256_prefix":"047bd41c9984a7fb","entry":"assign_learning_rate","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"047bd41c9984a7fb"}},{"code_sha256_prefix":"492c4d6d39fb421a","entry":"get_model","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"492c4d6d39fb421a"}},{"code_sha256_prefix":"ac6f2ab677109bb5","entry":"load_ckpt","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ac6f2ab677109bb5"}},{"code_sha256_prefix":"80e07bb784b914ea","entry":"load_dataset","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"80e07bb784b914ea"}},{"code_sha256_prefix":"6433f0d1078d502e","entry":"resnet18","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6433f0d1078d502e"}},{"code_sha256_prefix":"0cb0dca347d472c9","entry":"resnet50","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0cb0dca347d472c9"}},{"code_sha256_prefix":"44925cf3831bfe79","entry":"rgetattr","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"44925cf3831bfe79"}},{"code_sha256_prefix":"6fcb2d5ec5bc611b","entry":"save_checkpoint","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6fcb2d5ec5bc611b"}},{"code_sha256_prefix":"9b97b707b5baccde","entry":"summary_logging","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9b97b707b5baccde"}},{"code_sha256_prefix":"c5d4b37a50c67113","entry":"touch_dir","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c5d4b37a50c67113"}},{"code_sha256_prefix":"e62d3be00dcdc132","entry":"wandb_logging","repo":"amiiir-sarfi/seal","repo_kind":"official","path":"core.py","file_url":"https://github.com/amiiir-sarfi/seal/blob/HEAD/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e62d3be00dcdc132"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}