{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/open-data-on-github-unlocking-the-potential","title":"Open Data on GitHub: Unlocking the Potential of AI","arxiv_id":"2306.06191","date":"2023-06-09","proceeding":null,"authors":["Anthony Cintron Roman","Kevin Xu","Arfon Smith","Jehu Torres Vega","Caleb Robinson","Juan M Lavista Ferres"],"abstract":"GitHub is the world's largest platform for collaborative software development, with over 100 million users. GitHub is also used extensively for open data collaboration, hosting more than 800 million open data files, totaling 142 terabytes of data. This study highlights the potential of open data on GitHub and demonstrates how it can accelerate AI research. We analyze the existing landscape of open data on GitHub and the patterns of how users share datasets. Our findings show that GitHub is one of the largest hosts of open data in the world and has experienced an accelerated growth of open data assets over the past four years. By examining the open data landscape on GitHub, we aim to empower users and organizations to leverage existing open datasets and improve their discoverability -- ultimately contributing to the ongoing AI revolution to help address complex societal issues. We release the three datasets that we have collected to support this analysis as open datasets at https://github.com/github/open-data-on-github.","url_abs":"https://arxiv.org/abs/2306.06191v1","url_pdf":"https://arxiv.org/pdf/2306.06191v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"open-data-on-github-unlocking-the-potential","repo_url":"https://github.com/github/open-data-on-github","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2306.06191","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.06191"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/github/open-data-on-github","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/rajpurkar/SQuAD-explorer","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":3},"by_repo_kind":{"found_in_text":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"aa6bc32d9c17a2f1","entry":"get_tokens","repo":"rajpurkar/SQuAD-explorer","repo_kind":"found_in_text","path":"evaluate-v2.0.py","file_url":"https://github.com/rajpurkar/SQuAD-explorer/blob/HEAD/evaluate-v2.0.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa6bc32d9c17a2f1"}},{"code_sha256_prefix":"e91c44f5680827e2","entry":"make_qid_to_has_ans","repo":"rajpurkar/SQuAD-explorer","repo_kind":"found_in_text","path":"evaluate-v2.0.py","file_url":"https://github.com/rajpurkar/SQuAD-explorer/blob/HEAD/evaluate-v2.0.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e91c44f5680827e2"}},{"code_sha256_prefix":"6a96435eba311b08","entry":"normalize_answer","repo":"rajpurkar/SQuAD-explorer","repo_kind":"found_in_text","path":"evaluate-v2.0.py","file_url":"https://github.com/rajpurkar/SQuAD-explorer/blob/HEAD/evaluate-v2.0.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6a96435eba311b08"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}