{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/elf-gym-evaluating-large-language-models","title":"ELF-Gym: Evaluating Large Language Models Generated Features for Tabular Prediction","arxiv_id":"2410.12865","date":"2024-10-13","proceeding":null,"authors":["Yanlin Zhang","Ning li","Quan Gan","Weinan Zhang","David Wipf","Minjie Wang"],"abstract":"Crafting effective features is a crucial yet labor-intensive and domain-specific task within machine learning pipelines. Fortunately, recent advancements in Large Language Models (LLMs) have shown promise in automating various data science tasks, including feature engineering. But despite this potential, evaluations thus far are primarily based on the end performance of a complete ML pipeline, providing limited insight into precisely how LLMs behave relative to human experts in feature engineering. To address this gap, we propose ELF-Gym, a framework for Evaluating LLM-generated Features. We curated a new dataset from historical Kaggle competitions, including 251 \"golden\" features used by top-performing teams. ELF-Gym then quantitatively evaluates LLM-generated features by measuring their impact on downstream model performance as well as their alignment with expert-crafted features through semantic and functional similarity assessments. This approach provides a more comprehensive evaluation of disparities between LLMs and human experts, while offering valuable insights into specific areas where LLMs may have room for improvement. For example, using ELF-Gym we empirically demonstrate that, in the best-case scenario, LLMs can semantically capture approximately 56% of the golden features, but at the more demanding implementation level this overlap drops to 13%. Moreover, in other cases LLMs may fail completely, particularly on datasets that require complex features, indicating broad potential pathways for improvement.","url_abs":"https://arxiv.org/abs/2410.12865v1","url_pdf":"https://arxiv.org/pdf/2410.12865v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"elf-gym-evaluating-large-language-models","repo_url":"https://github.com/Lilyzhangyanlin/ELF-Gym","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"feature-engineering","task_name":"Feature Engineering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.12865","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.12865"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Lilyzhangyanlin/ELF-Gym","reach":{"status":"ok"}}],"summary":{"ran":9,"unverified":2},"by_repo_kind":{"official":{"samples":11,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"aa330865f9a44d97","entry":"check_correlation_equality","repo":"Lilyzhangyanlin/ELF-Gym","repo_kind":"official","path":"evaluator/correlation_evaluator.py","file_url":"https://github.com/Lilyzhangyanlin/ELF-Gym/blob/HEAD/evaluator/correlation_evaluator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa330865f9a44d97"}},{"code_sha256_prefix":"d4ea5de54a44a9bb","entry":"check_functional_equality","repo":"Lilyzhangyanlin/ELF-Gym","repo_kind":"official","path":"evaluator/functional_evaluator.py","file_url":"https://github.com/Lilyzhangyanlin/ELF-Gym/blob/HEAD/evaluator/functional_evaluator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d4ea5de54a44a9bb"}},{"code_sha256_prefix":"e646a1e4fb2a7ff1","entry":"clean_string","repo":"Lilyzhangyanlin/ELF-Gym","repo_kind":"official","path":"task/task.py","file_url":"https://github.com/Lilyzhangyanlin/ELF-Gym/blob/HEAD/task/task.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e646a1e4fb2a7ff1"}},{"code_sha256_prefix":"a11a3c364906cfe8","entry":"downsample","repo":"Lilyzhangyanlin/ELF-Gym","repo_kind":"official","path":"evaluator/utils.py","file_url":"https://github.com/Lilyzhangyanlin/ELF-Gym/blob/HEAD/evaluator/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a11a3c364906cfe8"}},{"code_sha256_prefix":"14b796c4dd681adb","entry":"downsample_avito","repo":"Lilyzhangyanlin/ELF-Gym","repo_kind":"official","path":"evaluator/utils.py","file_url":"https://github.com/Lilyzhangyanlin/ELF-Gym/blob/HEAD/evaluator/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"14b796c4dd681adb"}},{"code_sha256_prefix":"1a1abd813219a672","entry":"downsample_avito_for_generate","repo":"Lilyzhangyanlin/ELF-Gym","repo_kind":"official","path":"evaluator/utils.py","file_url":"https://github.com/Lilyzhangyanlin/ELF-Gym/blob/HEAD/evaluator/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1a1abd813219a672"}},{"code_sha256_prefix":"0754af0fa72eaf46","entry":"evaluator","repo":"Lilyzhangyanlin/ELF-Gym","repo_kind":"official","path":"evaluator/base.py","file_url":"https://github.com/Lilyzhangyanlin/ELF-Gym/blob/HEAD/evaluator/base.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0754af0fa72eaf46"}},{"code_sha256_prefix":"e28d3485501b3eb1","entry":"is_numeric_dtype","repo":"Lilyzhangyanlin/ELF-Gym","repo_kind":"official","path":"evaluator/functional_evaluator.py","file_url":"https://github.com/Lilyzhangyanlin/ELF-Gym/blob/HEAD/evaluator/functional_evaluator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e28d3485501b3eb1"}},{"code_sha256_prefix":"1e493d91abd19387","entry":"load_pyd","repo":"Lilyzhangyanlin/ELF-Gym","repo_kind":"official","path":"evaluator/yaml_utils.py","file_url":"https://github.com/Lilyzhangyanlin/ELF-Gym/blob/HEAD/evaluator/yaml_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1e493d91abd19387"}},{"code_sha256_prefix":"84a966c60cf51938","entry":"generate_new_task_df","repo":"Lilyzhangyanlin/ELF-Gym","repo_kind":"official","path":"evaluator/downstream_evaluator.py","file_url":"https://github.com/Lilyzhangyanlin/ELF-Gym/blob/HEAD/evaluator/downstream_evaluator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"84a966c60cf51938"}},{"code_sha256_prefix":"b70dfb0d42625eb1","entry":"get_evaluator_class","repo":"Lilyzhangyanlin/ELF-Gym","repo_kind":"official","path":"evaluator/base.py","file_url":"https://github.com/Lilyzhangyanlin/ELF-Gym/blob/HEAD/evaluator/base.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b70dfb0d42625eb1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}