{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/2506-08762","title":"EDINET-Bench: Evaluating LLMs on Complex Financial Tasks using Japanese Financial Statements","arxiv_id":"2506.08762","date":"2025-06-10","proceeding":null,"authors":["Issa Sugiura","Takashi Ishida","Taro Makino","Chieko Tazuke","Takanori Nakagawa","Kosuke Nakago","David Ha"],"abstract":"Financial analysis presents complex challenges that could leverage large language model (LLM) capabilities. However, the scarcity of challenging financial datasets, particularly for Japanese financial data, impedes academic innovation in financial analytics. As LLMs advance, this lack of accessible research resources increasingly hinders their development and evaluation in this specialized domain. To address this gap, we introduce EDINET-Bench, an open-source Japanese financial benchmark designed to evaluate the performance of LLMs on challenging financial tasks including accounting fraud detection, earnings forecasting, and industry prediction. EDINET-Bench is constructed by downloading annual reports from the past 10 years from Japan's Electronic Disclosure for Investors' NETwork (EDINET) and automatically assigning labels corresponding to each evaluation task. Our experiments reveal that even state-of-the-art LLMs struggle, performing only slightly better than logistic regression in binary classification for fraud detection and earnings forecasting. These results highlight significant challenges in applying LLMs to real-world financial applications and underscore the need for domain-specific adaptation. Our dataset, benchmark construction code, and evaluation code is publicly available to facilitate future research in finance with LLMs.","url_abs":"https://arxiv.org/abs/2506.08762v1","url_pdf":"https://arxiv.org/pdf/2506.08762v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"2506-08762","repo_url":"https://github.com/sakanaai/edinet-bench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"2506-08762","repo_url":"https://github.com/sakanaai/edinet2dataset","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"binary-classification","task_name":"Binary Classification"},{"task_slug":"financial-analysis","task_name":"Financial Analysis"},{"task_slug":"fraud-detection","task_name":"Fraud Detection"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[{"method_slug":"logistic-regression","method_name":"Logistic Regression"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2506.08762","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.08762"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sakanaai/edinet-bench","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sakanaai/edinet2dataset","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":9},"by_repo_kind":{"official":{"samples":9,"ran":0,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e05392e27d07bd24","entry":"collect_industry_metrics","repo":"sakanaai/edinet-bench","repo_kind":"official","path":"src/edinet_bench/industry_prediction/make_leaderboard.py","file_url":"https://github.com/sakanaai/edinet-bench/blob/HEAD/src/edinet_bench/industry_prediction/make_leaderboard.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e05392e27d07bd24"}},{"code_sha256_prefix":"7f47c8b8002eec2e","entry":"create_industry_confusion_matrix","repo":"sakanaai/edinet-bench","repo_kind":"official","path":"src/edinet_bench/industry_prediction/make_leaderboard.py","file_url":"https://github.com/sakanaai/edinet-bench/blob/HEAD/src/edinet_bench/industry_prediction/make_leaderboard.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7f47c8b8002eec2e"}},{"code_sha256_prefix":"c149dea72586c53e","entry":"extract_json_between_markers","repo":"sakanaai/edinet-bench","repo_kind":"official","path":"src/edinet_bench/utils.py","file_url":"https://github.com/sakanaai/edinet-bench/blob/HEAD/src/edinet_bench/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c149dea72586c53e"}},{"code_sha256_prefix":"73a0780970b78559","entry":"extract_leaf_elements","repo":"sakanaai/edinet2dataset","repo_kind":"official","path":"src/edinet2dataset/parser.py","file_url":"https://github.com/sakanaai/edinet2dataset/blob/HEAD/src/edinet2dataset/parser.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"73a0780970b78559"}},{"code_sha256_prefix":"1f2f0bea426febc2","entry":"fill_and_align_data","repo":"sakanaai/edinet-bench","repo_kind":"official","path":"src/edinet_bench/logistic.py","file_url":"https://github.com/sakanaai/edinet-bench/blob/HEAD/src/edinet_bench/logistic.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1f2f0bea426febc2"}},{"code_sha256_prefix":"9673c324bd62abd9","entry":"get_fiscal_start_year","repo":"sakanaai/edinet-bench","repo_kind":"official","path":"src/edinet_bench/contamination/analyze_performance_per_year.py","file_url":"https://github.com/sakanaai/edinet-bench/blob/HEAD/src/edinet_bench/contamination/analyze_performance_per_year.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9673c324bd62abd9"}},{"code_sha256_prefix":"67b49b0983888fd3","entry":"prepare_dataset","repo":"sakanaai/edinet-bench","repo_kind":"official","path":"src/edinet_bench/logistic.py","file_url":"https://github.com/sakanaai/edinet-bench/blob/HEAD/src/edinet_bench/logistic.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"67b49b0983888fd3"}},{"code_sha256_prefix":"f738811c8a5b14ff","entry":"preprocess_data","repo":"sakanaai/edinet-bench","repo_kind":"official","path":"src/edinet_bench/logistic.py","file_url":"https://github.com/sakanaai/edinet-bench/blob/HEAD/src/edinet_bench/logistic.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f738811c8a5b14ff"}},{"code_sha256_prefix":"3b28bcad9699de49","entry":"search_company","repo":"sakanaai/edinet2dataset","repo_kind":"official","path":"src/edinet2dataset/downloader.py","file_url":"https://github.com/sakanaai/edinet2dataset/blob/HEAD/src/edinet2dataset/downloader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3b28bcad9699de49"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}