{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/orkg-leaderboards-a-systematic-workflow-for","title":"ORKG-Leaderboards: A Systematic Workflow for Mining Leaderboards as a Knowledge Graph","arxiv_id":"2305.11068","date":"2023-05-10","proceeding":null,"authors":["Salomon Kabongo","Jennifer D'Souza","Sören Auer"],"abstract":"The purpose of this work is to describe the Orkg-Leaderboard software designed to extract leaderboards defined as Task-Dataset-Metric tuples automatically from large collections of empirical research papers in Artificial Intelligence (AI). The software can support both the main workflows of scholarly publishing, viz. as LaTeX files or as PDF files. Furthermore, the system is integrated with the Open Research Knowledge Graph (ORKG) platform, which fosters the machine-actionable publishing of scholarly findings. Thus the system output, when integrated within the ORKG's supported Semantic Web infrastructure of representing machine-actionable 'resources' on the Web, enables: 1) broadly, the integration of empirical results of researchers across the world, thus enabling transparency in empirical research with the potential to also being complete contingent on the underlying data source(s) of publications; and 2) specifically, enables researchers to track the progress in AI with an overview of the state-of-the-art (SOTA) across the most common AI tasks and their corresponding datasets via dynamic ORKG frontend views leveraging tables and visualization charts over the machine-actionable data. Our best model achieves performances above 90% F1 on the \\textit{leaderboard} extraction task, thus proving Orkg-Leaderboards a practically viable tool for real-world usage. Going forward, in a sense, Orkg-Leaderboards transforms the leaderboard extraction task to an automated digitalization task, which has been, for a long time in the community, a crowdsourced endeavor.","url_abs":"https://arxiv.org/abs/2305.11068v1","url_pdf":"https://arxiv.org/pdf/2305.11068v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"orkg-leaderboards-a-systematic-workflow-for","repo_url":"https://github.com/kabongosalomon/task-dataset-metric-nli-extraction","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2305.11068","atlas_url":"https://app.syntology.ai/?focus=2305.11068","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.11068"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kabongosalomon/task-dataset-metric-nli-extraction","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":1,"unverified":5},"by_repo_kind":{"official":{"samples":6,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3e21e6589663b136","entry":"epoch_time","repo":"kabongosalomon/task-dataset-metric-nli-extraction","repo_kind":"official","path":"utils/helpers.py","file_url":"https://github.com/kabongosalomon/task-dataset-metric-nli-extraction/blob/HEAD/utils/helpers.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e21e6589663b136"}},{"code_sha256_prefix":"1244c7e5c654ea45","entry":"count_parameters","repo":"kabongosalomon/task-dataset-metric-nli-extraction","repo_kind":"official","path":"utils/helpers.py","file_url":"https://github.com/kabongosalomon/task-dataset-metric-nli-extraction/blob/HEAD/utils/helpers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1244c7e5c654ea45"}},{"code_sha256_prefix":"c0afb00b571adb4e","entry":"extract_latex","repo":"kabongosalomon/task-dataset-metric-nli-extraction","repo_kind":"official","path":"pdf_to_txt/doc2json/tex2json/tex_to_xml.py","file_url":"https://github.com/kabongosalomon/task-dataset-metric-nli-extraction/blob/HEAD/pdf_to_txt/doc2json/tex2json/tex_to_xml.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c0afb00b571adb4e"}},{"code_sha256_prefix":"93d2b461b0299ebc","entry":"load_s2orc","repo":"kabongosalomon/task-dataset-metric-nli-extraction","repo_kind":"official","path":"pdf_to_txt/doc2json/s2orc.py","file_url":"https://github.com/kabongosalomon/task-dataset-metric-nli-extraction/blob/HEAD/pdf_to_txt/doc2json/s2orc.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"93d2b461b0299ebc"}},{"code_sha256_prefix":"91e64fe26dcc9c60","entry":"normalize_grobid_id","repo":"kabongosalomon/task-dataset-metric-nli-extraction","repo_kind":"official","path":"pdf_to_txt/doc2json/grobid2json/tei_to_json.py","file_url":"https://github.com/kabongosalomon/task-dataset-metric-nli-extraction/blob/HEAD/pdf_to_txt/doc2json/grobid2json/tei_to_json.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"91e64fe26dcc9c60"}},{"code_sha256_prefix":"96b822af4f39d289","entry":"train","repo":"kabongosalomon/task-dataset-metric-nli-extraction","repo_kind":"official","path":"utils/helpers.py","file_url":"https://github.com/kabongosalomon/task-dataset-metric-nli-extraction/blob/HEAD/utils/helpers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"96b822af4f39d289"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}