{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-no-free-lunch-theorem-kolmogorov","title":"The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learning","arxiv_id":"2304.05366","date":"2023-04-11","proceeding":null,"authors":["Micah Goldblum","Marc Finzi","Keefer Rowan","Andrew Gordon Wilson"],"abstract":"No free lunch theorems for supervised learning state that no learner can solve all problems or that all learners achieve exactly the same accuracy on average over a uniform distribution on learning problems. Accordingly, these theorems are often referenced in support of the notion that individual problems require specially tailored inductive biases. While virtually all uniformly sampled datasets have high complexity, real-world problems disproportionately generate low-complexity data, and we argue that neural network models share this same preference, formalized using Kolmogorov complexity. Notably, we show that architectures designed for a particular domain, such as computer vision, can compress datasets on a variety of seemingly unrelated domains. Our experiments show that pre-trained and even randomly initialized language models prefer to generate low-complexity sequences. Whereas no free lunch theorems seemingly indicate that individual problems require specialized learners, we explain how tasks that often require human intervention such as picking an appropriately sized model when labeled data is scarce or plentiful can be automated into a single learning algorithm. These observations justify the trend in deep learning of unifying seemingly disparate problems with an increasingly small set of machine learning models.","url_abs":"https://arxiv.org/abs/2304.05366v3","url_pdf":"https://arxiv.org/pdf/2304.05366v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-no-free-lunch-theorem-kolmogorov","repo_url":"https://github.com/goldblum/free-lunch","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2304.05366","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2304.05366"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/goldblum/free-lunch","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":11},"by_repo_kind":{"official":{"samples":11,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1a896a21f3fedb88","entry":"get_directory_size","repo":"goldblum/free-lunch","repo_kind":"official","path":"compress_data/utils.py","file_url":"https://github.com/goldblum/free-lunch/blob/HEAD/compress_data/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1a896a21f3fedb88"}},{"code_sha256_prefix":"91c718d12195c871","entry":"get_mdl","repo":"goldblum/free-lunch","repo_kind":"official","path":"llm_sequences/utils.py","file_url":"https://github.com/goldblum/free-lunch/blob/HEAD/llm_sequences/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"91c718d12195c871"}},{"code_sha256_prefix":"7c1a8ddca993d7e5","entry":"kl_div","repo":"goldblum/free-lunch","repo_kind":"official","path":"llm_sequences/utils.py","file_url":"https://github.com/goldblum/free-lunch/blob/HEAD/llm_sequences/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7c1a8ddca993d7e5"}},{"code_sha256_prefix":"5bc69fc51a4c86fb","entry":"logit_bias_dict","repo":"goldblum/free-lunch","repo_kind":"official","path":"llm_sequences/gpt3.py","file_url":"https://github.com/goldblum/free-lunch/blob/HEAD/llm_sequences/gpt3.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5bc69fc51a4c86fb"}},{"code_sha256_prefix":"8cbc4075a5dda86a","entry":"model_id","repo":"goldblum/free-lunch","repo_kind":"official","path":"llm_sequences/gpt3.py","file_url":"https://github.com/goldblum/free-lunch/blob/HEAD/llm_sequences/gpt3.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8cbc4075a5dda86a"}},{"code_sha256_prefix":"01ecf28711af5855","entry":"numpy_dataset_size_asarray","repo":"goldblum/free-lunch","repo_kind":"official","path":"compress_data/utils.py","file_url":"https://github.com/goldblum/free-lunch/blob/HEAD/compress_data/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"01ecf28711af5855"}},{"code_sha256_prefix":"b97eb4f54158059a","entry":"numpy_dataset_size_asimagefolder","repo":"goldblum/free-lunch","repo_kind":"official","path":"compress_data/utils.py","file_url":"https://github.com/goldblum/free-lunch/blob/HEAD/compress_data/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b97eb4f54158059a"}},{"code_sha256_prefix":"23df837f39aca985","entry":"random_operator_list","repo":"goldblum/free-lunch","repo_kind":"official","path":"llm_sequences/trees.py","file_url":"https://github.com/goldblum/free-lunch/blob/HEAD/llm_sequences/trees.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"23df837f39aca985"}},{"code_sha256_prefix":"e355b840ee660f21","entry":"random_term_list","repo":"goldblum/free-lunch","repo_kind":"official","path":"llm_sequences/trees.py","file_url":"https://github.com/goldblum/free-lunch/blob/HEAD/llm_sequences/trees.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e355b840ee660f21"}},{"code_sha256_prefix":"c82ba51785878be8","entry":"random_tree_list","repo":"goldblum/free-lunch","repo_kind":"official","path":"llm_sequences/trees.py","file_url":"https://github.com/goldblum/free-lunch/blob/HEAD/llm_sequences/trees.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c82ba51785878be8"}},{"code_sha256_prefix":"8cbcc6d1d211a4cd","entry":"sequences_by_mdl","repo":"goldblum/free-lunch","repo_kind":"official","path":"llm_sequences/utils.py","file_url":"https://github.com/goldblum/free-lunch/blob/HEAD/llm_sequences/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8cbcc6d1d211a4cd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}