{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fullstack-bench-evaluating-llms-as-full-stack","title":"FullStack Bench: Evaluating LLMs as Full Stack Coders","arxiv_id":"2412.00535","date":"2024-11-30","proceeding":null,"authors":["Bytedance-Seed-Foundation-Code-Team",":","Yao Cheng","Jianfeng Chen","Jie Chen","Li Chen","Liyu Chen","Wentao Chen","Zhengyu Chen","Shijie Geng","Aoyan Li","Bo Li","Bowen Li","Linyi Li","Boyi Liu","Jiaheng Liu","Kaibo Liu","Qi Liu","Shukai Liu","Siyao Liu","Tianyi Liu","Tingkai Liu","Yongfei Liu","Rui Long","Jing Mai","Guanghan Ning","Z. Y. Peng","Kai Shen","Jiahao Su","Jing Su","Tao Sun","Yifan Sun","Yunzhe Tao","Guoyin Wang","Siwei Wang","Xuwu Wang","Yite Wang","Zihan Wang","Jinxiang Xia","Liang Xiang","Xia Xiao","Yongsheng Xiao","Chenguang Xi","Shulin Xin","Jingjing Xu","Shikun Xu","Hongxia Yang","Jack Yang","Yingxiang Yang","Jianbo Yuan","Jun Zhang","Yufeng Zhang","Yuyu Zhang","Shen Zheng","He Zhu","Ming Zhu"],"abstract":"As the capabilities of code large language models (LLMs) continue to expand, their applications across diverse code intelligence domains are rapidly increasing. However, most existing datasets only evaluate limited application domains. To address this gap, we have developed a comprehensive code evaluation dataset FullStack Bench focusing on full-stack programming, which encompasses a wide range of application domains (e.g., basic programming, data analysis, software engineering, mathematics, and machine learning). Besides, to assess multilingual programming capabilities, in FullStack Bench, we design real-world instructions and corresponding unit test cases from 16 widely-used programming languages to reflect real-world usage scenarios rather than simple translations. Moreover, we also release an effective code sandbox execution tool (i.e., SandboxFusion) supporting various programming languages and packages to evaluate the performance of our FullStack Bench efficiently. Comprehensive experimental results on our FullStack Bench demonstrate the necessity and effectiveness of our FullStack Bench and SandboxFusion.","url_abs":"https://arxiv.org/abs/2412.00535v6","url_pdf":"https://arxiv.org/pdf/2412.00535v6.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"fullstack-bench-evaluating-llms-as-full-stack","repo_url":"https://github.com/bytedance/fullstackbench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"fullstack-bench-evaluating-llms-as-full-stack","repo_url":"https://github.com/bytedance/sandboxfusion","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.00535","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.00535"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bytedance/sandboxfusion","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bytedance/fullstackbench","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":14},"by_repo_kind":{"official":{"samples":14,"ran":0,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0c8ec0958e7449cb","entry":"append_test","repo":"bytedance/sandboxfusion","repo_kind":"official","path":"sandbox/datasets/autoeval.py","file_url":"https://github.com/bytedance/sandboxfusion/blob/HEAD/sandbox/datasets/autoeval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0c8ec0958e7449cb"}},{"code_sha256_prefix":"1f97a0c8318ed187","entry":"configurable_retry","repo":"bytedance/fullstackbench","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/bytedance/fullstackbench/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1f97a0c8318ed187"}},{"code_sha256_prefix":"cb2a2f189c014acf","entry":"extract_question","repo":"bytedance/sandboxfusion","repo_kind":"official","path":"sandbox/datasets/live_code_bench.py","file_url":"https://github.com/bytedance/sandboxfusion/blob/HEAD/sandbox/datasets/live_code_bench.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cb2a2f189c014acf"}},{"code_sha256_prefix":"88b93ad818bda02f","entry":"generate_fewshot_prompt","repo":"bytedance/sandboxfusion","repo_kind":"official","path":"sandbox/datasets/live_code_bench.py","file_url":"https://github.com/bytedance/sandboxfusion/blob/HEAD/sandbox/datasets/live_code_bench.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"88b93ad818bda02f"}},{"code_sha256_prefix":"a31e27f6de0a1ad8","entry":"generate_input_prompt","repo":"bytedance/sandboxfusion","repo_kind":"official","path":"sandbox/datasets/cruxeval.py","file_url":"https://github.com/bytedance/sandboxfusion/blob/HEAD/sandbox/datasets/cruxeval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a31e27f6de0a1ad8"}},{"code_sha256_prefix":"310d913d34ff3a84","entry":"generate_output_prompt","repo":"bytedance/sandboxfusion","repo_kind":"official","path":"sandbox/datasets/cruxeval.py","file_url":"https://github.com/bytedance/sandboxfusion/blob/HEAD/sandbox/datasets/cruxeval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"310d913d34ff3a84"}},{"code_sha256_prefix":"c1ac2d1e1f9e2e6d","entry":"get_categories","repo":"bytedance/sandboxfusion","repo_kind":"official","path":"sandbox/datasets/humanevoeval.py","file_url":"https://github.com/bytedance/sandboxfusion/blob/HEAD/sandbox/datasets/humanevoeval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c1ac2d1e1f9e2e6d"}},{"code_sha256_prefix":"42dedaf330d28e13","entry":"get_table_name","repo":"bytedance/sandboxfusion","repo_kind":"official","path":"sandbox/database.py","file_url":"https://github.com/bytedance/sandboxfusion/blob/HEAD/sandbox/database.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42dedaf330d28e13"}},{"code_sha256_prefix":"5f94c3a946bcfdf6","entry":"max_concurrency","repo":"bytedance/fullstackbench","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/bytedance/fullstackbench/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5f94c3a946bcfdf6"}},{"code_sha256_prefix":"cb97c663edcbd66d","entry":"postprocess_full_code","repo":"bytedance/sandboxfusion","repo_kind":"official","path":"sandbox/datasets/autoeval.py","file_url":"https://github.com/bytedance/sandboxfusion/blob/HEAD/sandbox/datasets/autoeval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cb97c663edcbd66d"}},{"code_sha256_prefix":"588673a8052153a0","entry":"postprocess_full_code","repo":"bytedance/sandboxfusion","repo_kind":"official","path":"sandbox/datasets/humaneval.py","file_url":"https://github.com/bytedance/sandboxfusion/blob/HEAD/sandbox/datasets/humaneval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"588673a8052153a0"}},{"code_sha256_prefix":"ff3dacdd2cf428ba","entry":"read_jsonl","repo":"bytedance/fullstackbench","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/bytedance/fullstackbench/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ff3dacdd2cf428ba"}},{"code_sha256_prefix":"52d455f13017fb44","entry":"stop_token_trim","repo":"bytedance/sandboxfusion","repo_kind":"official","path":"sandbox/datasets/humaneval.py","file_url":"https://github.com/bytedance/sandboxfusion/blob/HEAD/sandbox/datasets/humaneval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"52d455f13017fb44"}},{"code_sha256_prefix":"dcd915834ca5db10","entry":"stop_token_trim","repo":"bytedance/sandboxfusion","repo_kind":"official","path":"sandbox/datasets/humaneval_deprecated.py","file_url":"https://github.com/bytedance/sandboxfusion/blob/HEAD/sandbox/datasets/humaneval_deprecated.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dcd915834ca5db10"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}