{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/archon-an-architecture-search-framework-for","title":"Archon: An Architecture Search Framework for Inference-Time Techniques","arxiv_id":"2409.15254","date":"2024-09-23","proceeding":null,"authors":["Jon Saad-Falcon","Adrian Gamarra Lafuente","Shlok Natarajan","Nahum Maru","Hristo Todorov","Etash Guha","E. Kelly Buchanan","Mayee Chen","Neel Guha","Christopher Ré","Azalia Mirhoseini"],"abstract":"Inference-time techniques are emerging as highly effective tools to enhance large language model (LLM) capabilities. However, best practices for developing systems that combine these techniques remain underdeveloped due to our limited understanding of the utility of individual inference-time techniques and the interactions between them. Additionally, efficiently and automatically searching the space of model choices, inference-time techniques, and their compositions is challenging due to the large design space. To address these challenges, we introduce Archon, a modular framework for selecting, combining, and stacking layers of inference-time techniques to construct optimized LLM systems for target benchmarks. Rather than relying on a single LLM called once, we leverage a diverse set of LLMs and inference-time techniques, creating LLM systems greater than the sum of their parts. Archon defines an extensible design space, encompassing techniques such as generation ensembling, repeated sampling, ranking, fusion, critiquing, verification, and unit testing. It transforms the problem of building LLM systems into a hyperparameter optimization objective. Given the available LLMs, inference-time techniques, and compute budget, Archon utilizes hyperparameter search techniques to discover optimized architectures for target benchmark(s). We evaluate Archon architectures across a range of instruction-following, reasoning, and coding benchmarks, including MT-Bench, Arena-Hard-Auto, AlpacaEval 2.0, MixEval, MixEval Hard, MATH, and CodeContests. Archon architectures outperform frontier models, such as GPT-4o and Claude 3.5 Sonnet, on these benchmarks, achieving an average accuracy increase of 15.1 percentage points by using all available LLMs. We make our code and datasets available publicly on Github: https://github.com/ScalingIntelligence/Archon.","url_abs":"https://arxiv.org/abs/2409.15254v5","url_pdf":"https://arxiv.org/pdf/2409.15254v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"archon-an-architecture-search-framework-for","repo_url":"https://github.com/scalingintelligence/archon","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"hyperparameter-optimization","task_name":"Hyperparameter Optimization"},{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"math","task_name":"Math"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.15254","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.15254"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/scalingintelligence/archon","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":14,"unverified":2},"by_repo_kind":{"official":{"samples":16,"ran":14,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0ff4a203cd88039c","entry":"append_json","repo":"scalingintelligence/archon","repo_kind":"official","path":"src/archon/itas_algorithms/itas_algorithm.py","file_url":"https://github.com/scalingintelligence/archon/blob/HEAD/src/archon/itas_algorithms/itas_algorithm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0ff4a203cd88039c"}},{"code_sha256_prefix":"107600cb9ca038b0","entry":"clean_messages","repo":"scalingintelligence/archon","repo_kind":"official","path":"src/archon/completions/utils.py","file_url":"https://github.com/scalingintelligence/archon/blob/HEAD/src/archon/completions/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"107600cb9ca038b0"}},{"code_sha256_prefix":"e699241c4fab3815","entry":"compute_mle_elo","repo":"scalingintelligence/archon","repo_kind":"official","path":"src/archon/benchmarks/arena_hard_auto/show_arena_hard_auto_result.py","file_url":"https://github.com/scalingintelligence/archon/blob/HEAD/src/archon/benchmarks/arena_hard_auto/show_arena_hard_auto_result.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e699241c4fab3815"}},{"code_sha256_prefix":"e4f095487017d03d","entry":"count_correct_yaml_files","repo":"scalingintelligence/archon","repo_kind":"official","path":"src/archon/benchmarks/code_contests/show_code_results.py","file_url":"https://github.com/scalingintelligence/archon/blob/HEAD/src/archon/benchmarks/code_contests/show_code_results.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e4f095487017d03d"}},{"code_sha256_prefix":"a3621beb3eb4d82a","entry":"format_prompt","repo":"scalingintelligence/archon","repo_kind":"official","path":"src/archon/completions/utils.py","file_url":"https://github.com/scalingintelligence/archon/blob/HEAD/src/archon/completions/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a3621beb3eb4d82a"}},{"code_sha256_prefix":"96970be6c2e04513","entry":"generate_pairwise_config","repo":"scalingintelligence/archon","repo_kind":"official","path":"src/archon/benchmarks/arena_hard_auto/gen_judgment.py","file_url":"https://github.com/scalingintelligence/archon/blob/HEAD/src/archon/benchmarks/arena_hard_auto/gen_judgment.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"96970be6c2e04513"}},{"code_sha256_prefix":"4a901a353cc86998","entry":"get_bootstrap_result","repo":"scalingintelligence/archon","repo_kind":"official","path":"src/archon/benchmarks/arena_hard_auto/show_arena_hard_auto_result.py","file_url":"https://github.com/scalingintelligence/archon/blob/HEAD/src/archon/benchmarks/arena_hard_auto/show_arena_hard_auto_result.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4a901a353cc86998"}},{"code_sha256_prefix":"c8136157d71c093b","entry":"get_endpoint","repo":"scalingintelligence/archon","repo_kind":"official","path":"src/archon/benchmarks/arena_hard_auto/arena_hard_auto_utils.py","file_url":"https://github.com/scalingintelligence/archon/blob/HEAD/src/archon/benchmarks/arena_hard_auto/arena_hard_auto_utils.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c8136157d71c093b"}},{"code_sha256_prefix":"a92705181ff634e3","entry":"load_config","repo":"scalingintelligence/archon","repo_kind":"official","path":"src/archon/completions/utils.py","file_url":"https://github.com/scalingintelligence/archon/blob/HEAD/src/archon/completions/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a92705181ff634e3"}},{"code_sha256_prefix":"28bbb1e46eb5f17c","entry":"load_model_answers","repo":"scalingintelligence/archon","repo_kind":"official","path":"src/archon/benchmarks/arena_hard_auto/arena_hard_auto_utils.py","file_url":"https://github.com/scalingintelligence/archon/blob/HEAD/src/archon/benchmarks/arena_hard_auto/arena_hard_auto_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"28bbb1e46eb5f17c"}},{"code_sha256_prefix":"47892c5631a199d4","entry":"load_questions","repo":"scalingintelligence/archon","repo_kind":"official","path":"src/archon/benchmarks/arena_hard_auto/arena_hard_auto_utils.py","file_url":"https://github.com/scalingintelligence/archon/blob/HEAD/src/archon/benchmarks/arena_hard_auto/arena_hard_auto_utils.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"47892c5631a199d4"}},{"code_sha256_prefix":"047d950b82412f4f","entry":"load_search_config","repo":"scalingintelligence/archon","repo_kind":"official","path":"src/archon/itas_algorithms/itas_algorithm.py","file_url":"https://github.com/scalingintelligence/archon/blob/HEAD/src/archon/itas_algorithms/itas_algorithm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"047d950b82412f4f"}},{"code_sha256_prefix":"cec182ddf2df6b1e","entry":"parse_model_name","repo":"scalingintelligence/archon","repo_kind":"official","path":"src/archon/itas_algorithms/power_ranker.py","file_url":"https://github.com/scalingintelligence/archon/blob/HEAD/src/archon/itas_algorithms/power_ranker.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cec182ddf2df6b1e"}},{"code_sha256_prefix":"2a7dcca7dcf80307","entry":"preety_print_two_ratings","repo":"scalingintelligence/archon","repo_kind":"official","path":"src/archon/benchmarks/arena_hard_auto/show_arena_hard_auto_result.py","file_url":"https://github.com/scalingintelligence/archon/blob/HEAD/src/archon/benchmarks/arena_hard_auto/show_arena_hard_auto_result.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2a7dcca7dcf80307"}},{"code_sha256_prefix":"b2d83f0b138fe334","entry":"get_score","repo":"scalingintelligence/archon","repo_kind":"official","path":"src/archon/benchmarks/arena_hard_auto/gen_judgment.py","file_url":"https://github.com/scalingintelligence/archon/blob/HEAD/src/archon/benchmarks/arena_hard_auto/gen_judgment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b2d83f0b138fe334"}},{"code_sha256_prefix":"4a737f68f1c25eef","entry":"load_benchmark","repo":"scalingintelligence/archon","repo_kind":"official","path":"src/archon/benchmarks/benchmarks.py","file_url":"https://github.com/scalingintelligence/archon/blob/HEAD/src/archon/benchmarks/benchmarks.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4a737f68f1c25eef"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}