{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/texttt-tasksource-structured-dataset","title":"tasksource: A Dataset Harmonization Framework for Streamlined NLP Multi-Task Learning and Evaluation","arxiv_id":"2301.05948","date":"2023-01-14","proceeding":null,"authors":["Damien Sileo"],"abstract":"The HuggingFace Datasets Hub hosts thousands of datasets, offering exciting opportunities for language model training and evaluation. However, datasets for a specific task type often have different schemas, making harmonization challenging. Multi-task training or evaluation necessitates manual work to fit data into task templates. Several initiatives independently tackle this issue by releasing harmonized datasets or providing harmonization codes to preprocess datasets into a consistent format. We identify patterns across previous preprocessing efforts, such as column name mapping and extracting specific sub-fields from structured data in a column. We then propose a structured annotation framework that ensures our annotations are fully exposed and not hidden within unstructured code. We release a dataset annotation framework and dataset annotations for more than 500 English tasks\\footnote{\\url{https://github.com/sileod/tasksource}}. These annotations include metadata, such as the names of columns to be used as input or labels for all datasets, which can save time for future dataset preprocessing, regardless of whether our framework is utilized. We fine-tune a multi-task text encoder on all tasksource tasks, outperforming every publicly available text encoder of comparable size in an external evaluation.","url_abs":"https://arxiv.org/abs/2301.05948v3","url_pdf":"https://arxiv.org/pdf/2301.05948v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"texttt-tasksource-structured-dataset","repo_url":"https://github.com/sileod/tasksource","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"linguistic-acceptability","task_name":"Linguistic Acceptability"},{"task_slug":"multi-task-learning","task_name":"Multi-Task Learning"}],"methods":[],"datasets_introduced":[{"slug":"tasksource","name":"Tasksource","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/linguistic-acceptability-on-cola","task":"Linguistic Acceptability","dataset":"CoLA","model":"deberta-v3-base+tasksource","rank_in_archive_order":4,"of":43,"metrics":{"Accuracy":"87.15%"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2301.05948","atlas_url":"https://app.syntology.ai/?focus=2301.05948","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2301.05948"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sileod/tasksource","reach":null}],"summary":{"ran_draft_wrong":3,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"8eb178c77b03e6da","entry":"parse_var_name","repo":"sileod/tasksource","repo_kind":"official","path":"src/tasksource/access.py","file_url":"https://github.com/sileod/tasksource/blob/HEAD/src/tasksource/access.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"8eb178c77b03e6da"}},{"code_sha256_prefix":"21059751a26a8512","entry":"pretty_name","repo":"sileod/tasksource","repo_kind":"official","path":"src/tasksource/access.py","file_url":"https://github.com/sileod/tasksource/blob/HEAD/src/tasksource/access.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"21059751a26a8512"}},{"code_sha256_prefix":"4022aa306c228058","entry":"remove_neg_1","repo":"sileod/tasksource","repo_kind":"official","path":"src/tasksource/tasks.py","file_url":"https://github.com/sileod/tasksource/blob/HEAD/src/tasksource/tasks.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"4022aa306c228058"}},{"code_sha256_prefix":"fdad92bf012e1a7d","entry":"stance_kwargs","repo":"sileod/tasksource","repo_kind":"official","path":"src/tasksource/tasks.py","file_url":"https://github.com/sileod/tasksource/blob/HEAD/src/tasksource/tasks.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"fdad92bf012e1a7d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}