{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/beyond-pip-install-evaluating-llm-agents-for","title":"Beyond pip install: Evaluating LLM Agents for the Automated Installation of Python Projects","arxiv_id":"2412.06294","date":"2024-12-09","proceeding":null,"authors":["Louis Milliken","Sungmin Kang","Shin Yoo"],"abstract":"Many works have recently proposed the use of Large Language Model (LLM) based agents for performing `repository level' tasks, loosely defined as a set of tasks whose scopes are greater than a single file. This has led to speculation that the orchestration of these repository-level tasks could lead to software engineering agents capable of performing almost independently of human intervention. However, of the suite of tasks that would need to be performed by this autonomous software engineering agent, we argue that one important task is missing, which is to fulfil project level dependency by installing other repositories. To investigate the feasibility of this repository level installation task, we introduce a benchmark of of repository installation tasks curated from 40 open source Python projects, which includes a ground truth installation process for each target repository. Further, we propose Installamatic, an agent which aims to perform and verify the installation of a given repository by searching for relevant instructions from documentation in the repository. Empirical experiments reveal that that 55% of the studied repositories can be automatically installed by our agent at least one out of ten times. Through further analysis, we identify the common causes for our agent's inability to install a repository, discuss the challenges faced in the design and implementation of such an agent and consider the implications that such an agent could have for developers.","url_abs":"https://arxiv.org/abs/2412.06294v1","url_pdf":"https://arxiv.org/pdf/2412.06294v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"links_only","authors_date_abstract":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license), from the Kaggle arXiv metadata snapshot of 2026-09-12"},"code_links":[{"paper_slug":"beyond-pip-install-evaluating-llm-agents-for","repo_url":"https://github.com/coinse/installamatic","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.06294","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.06294"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/coinse/installamatic","reach":{"status":"ok"}}],"summary":{"unverified":7},"by_repo_kind":{"official":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"29922f074be40b6e","entry":"classify_output","repo":"coinse/installamatic","repo_kind":"official","path":"install_test/utils.py","file_url":"https://github.com/coinse/installamatic/blob/HEAD/install_test/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"29922f074be40b6e"}},{"code_sha256_prefix":"104c8a833aed2c49","entry":"get_api_url","repo":"coinse/installamatic","repo_kind":"official","path":"install_test/agent/functions.py","file_url":"https://github.com/coinse/installamatic/blob/HEAD/install_test/agent/functions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"104c8a833aed2c49"}},{"code_sha256_prefix":"a66633389f22086d","entry":"get_repository_language","repo":"coinse/installamatic","repo_kind":"official","path":"git_scraping.py","file_url":"https://github.com/coinse/installamatic/blob/HEAD/git_scraping.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a66633389f22086d"}},{"code_sha256_prefix":"c60fd3fe99446474","entry":"objectify","repo":"coinse/installamatic","repo_kind":"official","path":"install_test/utils.py","file_url":"https://github.com/coinse/installamatic/blob/HEAD/install_test/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c60fd3fe99446474"}},{"code_sha256_prefix":"1e060d43df09d10b","entry":"print_page","repo":"coinse/installamatic","repo_kind":"official","path":"git_scraping.py","file_url":"https://github.com/coinse/installamatic/blob/HEAD/git_scraping.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1e060d43df09d10b"}},{"code_sha256_prefix":"b3fd9264cd4ae634","entry":"send_request","repo":"coinse/installamatic","repo_kind":"official","path":"install_test/agent/functions.py","file_url":"https://github.com/coinse/installamatic/blob/HEAD/install_test/agent/functions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b3fd9264cd4ae634"}},{"code_sha256_prefix":"f63287ed2ce0f383","entry":"wrap_message","repo":"coinse/installamatic","repo_kind":"official","path":"install_test/utils.py","file_url":"https://github.com/coinse/installamatic/blob/HEAD/install_test/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f63287ed2ce0f383"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}