{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/implicit-regularization-of-multi-task","title":"Inductive biases of multi-task learning and finetuning: multiple regimes of feature reuse","arxiv_id":"2310.02396","date":"2023-10-03","proceeding":null,"authors":["Samuel Lippl","Jack W. Lindsey"],"abstract":"Neural networks are often trained on multiple tasks, either simultaneously (multi-task learning, MTL) or sequentially (pretraining and subsequent finetuning, PT+FT). In particular, it is common practice to pretrain neural networks on a large auxiliary task before finetuning on a downstream task with fewer samples. Despite the prevalence of this approach, the inductive biases that arise from learning multiple tasks are poorly characterized. In this work, we address this gap. We describe novel implicit regularization penalties associated with MTL and PT+FT in diagonal linear networks and single-hidden-layer ReLU networks. These penalties indicate that MTL and PT+FT induce the network to reuse features in different ways. 1) Both MTL and PT+FT exhibit biases towards feature reuse between tasks, and towards sparsity in the set of learned features. We show a \"conservation law\" that implies a direct tradeoff between these two biases. 2) PT+FT exhibits a novel \"nested feature selection\" regime, not described by either the \"lazy\" or \"rich\" regimes identified in prior work, which biases it to rely on a sparse subset of the features learned during pretraining. This regime is much narrower for MTL. 3) PT+FT (but not MTL) in ReLU networks benefits from features that are correlated between the auxiliary and main task. We confirm these findings empirically with teacher-student models, and introduce a technique -- weight rescaling following pretraining -- that can elicit the nested feature selection regime. Finally, we validate our theory in deep neural networks trained on image classification. We find that weight rescaling improves performance when it causes models to display signatures of nested feature selection. Our results suggest that nested feature selection may be an important inductive bias for finetuning neural networks.","url_abs":"https://arxiv.org/abs/2310.02396v4","url_pdf":"https://arxiv.org/pdf/2310.02396v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"implicit-regularization-of-multi-task","repo_url":"https://github.com/sflippl/multi-task","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"inductive-bias","task_name":"Inductive Bias"},{"task_slug":"multi-task-learning","task_name":"Multi-Task Learning"},{"task_slug":"feature-selection","task_name":"feature selection"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"feature-selection","method_name":"Feature Selection"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.02396","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.02396"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sflippl/multi-task","reach":{"status":"ok"}}],"summary":{"ran":7,"ran_draft_wrong":2},"by_repo_kind":{"official":{"samples":9,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"6154ef06ab70f9c2","entry":"B","repo":"sflippl/multi-task","repo_kind":"official","path":"functions/networks.py","file_url":"https://github.com/sflippl/multi-task/blob/HEAD/functions/networks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6154ef06ab70f9c2"}},{"code_sha256_prefix":"906dcd37eb8e06f4","entry":"add_argparse_arguments","repo":"sflippl/multi-task","repo_kind":"official","path":"functions/toy_networks_2.py","file_url":"https://github.com/sflippl/multi-task/blob/HEAD/functions/toy_networks_2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"906dcd37eb8e06f4"}},{"code_sha256_prefix":"2a80220dabcb742a","entry":"conv1x1","repo":"sflippl/multi-task","repo_kind":"official","path":"functions/custom_resnet.py","file_url":"https://github.com/sflippl/multi-task/blob/HEAD/functions/custom_resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2a80220dabcb742a"}},{"code_sha256_prefix":"600ff2c45e0de056","entry":"conv3x3","repo":"sflippl/multi-task","repo_kind":"official","path":"functions/custom_resnet.py","file_url":"https://github.com/sflippl/multi-task/blob/HEAD/functions/custom_resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"600ff2c45e0de056"}},{"code_sha256_prefix":"787ca95fd9f74cd6","entry":"get_network","repo":"sflippl/multi-task","repo_kind":"official","path":"functions/toy_networks_2.py","file_url":"https://github.com/sflippl/multi-task/blob/HEAD/functions/toy_networks_2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"787ca95fd9f74cd6"}},{"code_sha256_prefix":"47cd3b12f7a09bdf","entry":"l1_norm","repo":"sflippl/multi-task","repo_kind":"official","path":"experiments/diagonal/diagonal_network_finetune.py","file_url":"https://github.com/sflippl/multi-task/blob/HEAD/experiments/diagonal/diagonal_network_finetune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"47cd3b12f7a09bdf"}},{"code_sha256_prefix":"003e039860a4ae99","entry":"l2_norm","repo":"sflippl/multi-task","repo_kind":"official","path":"experiments/diagonal/diagonal_network_finetune.py","file_url":"https://github.com/sflippl/multi-task/blob/HEAD/experiments/diagonal/diagonal_network_finetune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"003e039860a4ae99"}},{"code_sha256_prefix":"b924dc7c735abcc9","entry":"mt_norm","repo":"sflippl/multi-task","repo_kind":"official","path":"experiments/diagonal/diagonal_network_finetune.py","file_url":"https://github.com/sflippl/multi-task/blob/HEAD/experiments/diagonal/diagonal_network_finetune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b924dc7c735abcc9"}},{"code_sha256_prefix":"6ee72f8c332d57be","entry":"train","repo":"sflippl/multi-task","repo_kind":"official","path":"experiments/diagonal/diagonal_network_pretrain.py","file_url":"https://github.com/sflippl/multi-task/blob/HEAD/experiments/diagonal/diagonal_network_pretrain.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6ee72f8c332d57be"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}