{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/compl-ai-framework-a-technical-interpretation","title":"COMPL-AI Framework: A Technical Interpretation and LLM Benchmarking Suite for the EU Artificial Intelligence Act","arxiv_id":"2410.07959","date":"2024-10-10","proceeding":null,"authors":["Philipp Guldimann","Alexander Spiridonov","Robin Staab","Nikola Jovanović","Mark Vero","Velko Vechev","Anna-Maria Gueorguieva","Mislav Balunović","Nikola Konstantinov","Pavol Bielik","Petar Tsankov","Martin Vechev"],"abstract":"The EU's Artificial Intelligence Act (AI Act) is a significant step towards responsible AI development, but lacks clear technical interpretation, making it difficult to assess models' compliance. This work presents COMPL-AI, a comprehensive framework consisting of (i) the first technical interpretation of the EU AI Act, translating its broad regulatory requirements into measurable technical requirements, with the focus on large language models (LLMs), and (ii) an open-source Act-centered benchmarking suite, based on thorough surveying and implementation of state-of-the-art LLM benchmarks. By evaluating 12 prominent LLMs in the context of COMPL-AI, we reveal shortcomings in existing models and benchmarks, particularly in areas like robustness, safety, diversity, and fairness. This work highlights the need for a shift in focus towards these aspects, encouraging balanced development of LLMs and more comprehensive regulation-aligned benchmarks. Simultaneously, COMPL-AI for the first time demonstrates the possibilities and difficulties of bringing the Act's obligations to a more concrete, technical level. As such, our work can serve as a useful first step towards having actionable recommendations for model providers, and contributes to ongoing efforts of the EU to enable application of the Act, such as the drafting of the GPAI Code of Practice.","url_abs":"https://arxiv.org/abs/2410.07959v2","url_pdf":"https://arxiv.org/pdf/2410.07959v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"compl-ai-framework-a-technical-interpretation","repo_url":"https://github.com/compl-ai/compl-ai","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"fairness","task_name":"Fairness"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.07959","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.07959"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/compl-ai/compl-ai","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":7},"by_repo_kind":{"official":{"samples":7,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ae2c71047a6f8690","entry":"aggregate_gender_polarity","repo":"compl-ai/compl-ai","repo_kind":"official","path":"src/complai/tasks/bold/bold.py","file_url":"https://github.com/compl-ai/compl-ai/blob/HEAD/src/complai/tasks/bold/bold.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ae2c71047a6f8690"}},{"code_sha256_prefix":"858f6a4dfb49de6a","entry":"complete_truncated_json","repo":"compl-ai/compl-ai","repo_kind":"official","path":"src/complai/tasks/cab/utils.py","file_url":"https://github.com/compl-ai/compl-ai/blob/HEAD/src/complai/tasks/cab/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"858f6a4dfb49de6a"}},{"code_sha256_prefix":"7832009e0dbb93c7","entry":"format_question_and_choices","repo":"compl-ai/compl-ai","repo_kind":"official","path":"src/complai/tasks/bigbench_calibration/bigbench_calibration.py","file_url":"https://github.com/compl-ai/compl-ai/blob/HEAD/src/complai/tasks/bigbench_calibration/bigbench_calibration.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7832009e0dbb93c7"}},{"code_sha256_prefix":"ef528c44edf8abc1","entry":"get_input_choices_target","repo":"compl-ai/compl-ai","repo_kind":"official","path":"src/complai/tasks/bigbench_calibration/bigbench_calibration.py","file_url":"https://github.com/compl-ai/compl-ai/blob/HEAD/src/complai/tasks/bigbench_calibration/bigbench_calibration.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ef528c44edf8abc1"}},{"code_sha256_prefix":"06cb2e92fc3dcbf1","entry":"parse_bias_evaluation","repo":"compl-ai/compl-ai","repo_kind":"official","path":"src/complai/tasks/cab/utils.py","file_url":"https://github.com/compl-ai/compl-ai/blob/HEAD/src/complai/tasks/cab/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"06cb2e92fc3dcbf1"}},{"code_sha256_prefix":"4639ba6b0d5b3abd","entry":"parse_tasks","repo":"compl-ai/compl-ai","repo_kind":"official","path":"src/complai/_cli/utils.py","file_url":"https://github.com/compl-ai/compl-ai/blob/HEAD/src/complai/_cli/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4639ba6b0d5b3abd"}},{"code_sha256_prefix":"e9847cdadba04ded","entry":"substitute_placeholder","repo":"compl-ai/compl-ai","repo_kind":"official","path":"src/complai/tasks/cab/utils.py","file_url":"https://github.com/compl-ai/compl-ai/blob/HEAD/src/complai/tasks/cab/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e9847cdadba04ded"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}