{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2604-13055","title":"WorkRB: A Community-Driven Evaluation Framework for AI in the Work Domain","arxiv_id":"2604.13055","date":"2026-03-17","proceeding":null,"authors":["Matthias De Lange","Warre Veys","Federico Retyk","Daniel Deniz","Warren Jouanneau","Mike Zhang","Aleksander Bielinski","Emma Jouffroy","Nicole Clobes","Nina Baranowska","David Graus","Marc Palyart","Rabih Zbib","Dimitra Gkatzia","Thomas Demeester","Tijl De Bie","Toine Bogers","Jens-Joris Decorte","Jeroen Van Hautte"],"abstract":"Today's evolving labor markets rely increasingly on recommender systems for hiring, talent management, and workforce analytics, with natural language processing (NLP) capabilities at the core. Yet, research in this area remains highly fragmented. Studies employ divergent ontologies (ESCO, O*NET, national taxonomies), heterogeneous task formulations, and diverse model families, making cross-study comparison and reproducibility exceedingly difficult. General-purpose benchmarks lack coverage of work-specific tasks, and the inherent sensitivity of employment data further limits open evaluation. We present \\textbf{WorkRB} (Work Research Benchmark), the first open-source, community-driven benchmark tailored to work-domain AI. WorkRB organizes 13 diverse tasks from 7 task groups as unified recommendation and NLP tasks, including job/skill recommendation, candidate recommendation, similar item recommendation, and skill extraction and normalization. WorkRB enables both monolingual and cross-lingual evaluation settings through dynamic loading of multilingual ontologies. Developed within a multi-stakeholder ecosystem of academia, industry, and public institutions, WorkRB has a modular design for seamless contributions and enables integration of proprietary tasks without disclosing sensitive data. WorkRB is available under the Apache 2.0 license at https://github.com/techwolf-ai/WorkRB.","url_abs":"https://arxiv.org/abs/2604.13055","url_pdf":"https://arxiv.org/pdf/2604.13055","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2604.13055","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2604.13055"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/techwolf-ai/WorkRB","reach":null}],"summary":{"unverified":12},"by_repo_kind":{"found_in_text":{"samples":12,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c74b47334acd762e","entry":"calculate_classification_metrics","repo":"techwolf-ai/WorkRB","repo_kind":"found_in_text","path":"src/workrb/metrics/classification.py","file_url":"https://github.com/techwolf-ai/WorkRB/blob/HEAD/src/workrb/metrics/classification.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c74b47334acd762e"}},{"code_sha256_prefix":"4288fe0b43a88064","entry":"calculate_classification_roc_auc","repo":"techwolf-ai/WorkRB","repo_kind":"found_in_text","path":"src/workrb/metrics/classification.py","file_url":"https://github.com/techwolf-ai/WorkRB/blob/HEAD/src/workrb/metrics/classification.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4288fe0b43a88064"}},{"code_sha256_prefix":"0db5be4d53371d28","entry":"calculate_ranking_metrics","repo":"techwolf-ai/WorkRB","repo_kind":"found_in_text","path":"src/workrb/metrics/ranking.py","file_url":"https://github.com/techwolf-ai/WorkRB/blob/HEAD/src/workrb/metrics/ranking.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0db5be4d53371d28"}},{"code_sha256_prefix":"56f5efe7a3fc3f58","entry":"create_task_from_config","repo":"techwolf-ai/WorkRB","repo_kind":"found_in_text","path":"src/workrb/registry.py","file_url":"https://github.com/techwolf-ai/WorkRB/blob/HEAD/src/workrb/registry.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"56f5efe7a3fc3f58"}},{"code_sha256_prefix":"20bba0cdd37c46e8","entry":"get_language_grouping_key","repo":"techwolf-ai/WorkRB","repo_kind":"found_in_text","path":"src/workrb/types.py","file_url":"https://github.com/techwolf-ai/WorkRB/blob/HEAD/src/workrb/types.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"20bba0cdd37c46e8"}},{"code_sha256_prefix":"28fb46361c08cc07","entry":"load_rankings_artifact","repo":"techwolf-ai/WorkRB","repo_kind":"found_in_text","path":"src/workrb/rankings.py","file_url":"https://github.com/techwolf-ai/WorkRB/blob/HEAD/src/workrb/rankings.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"28fb46361c08cc07"}},{"code_sha256_prefix":"d94ab1f03c5dc016","entry":"materialize_prediction_matrix","repo":"techwolf-ai/WorkRB","repo_kind":"found_in_text","path":"src/workrb/rankings.py","file_url":"https://github.com/techwolf-ai/WorkRB/blob/HEAD/src/workrb/rankings.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d94ab1f03c5dc016"}},{"code_sha256_prefix":"d553f0984b9ef465","entry":"optimize_threshold","repo":"techwolf-ai/WorkRB","repo_kind":"found_in_text","path":"src/workrb/metrics/classification.py","file_url":"https://github.com/techwolf-ai/WorkRB/blob/HEAD/src/workrb/metrics/classification.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d553f0984b9ef465"}},{"code_sha256_prefix":"bc0ef7b2764fe0e7","entry":"rankings_filename","repo":"techwolf-ai/WorkRB","repo_kind":"found_in_text","path":"src/workrb/rankings.py","file_url":"https://github.com/techwolf-ai/WorkRB/blob/HEAD/src/workrb/rankings.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bc0ef7b2764fe0e7"}},{"code_sha256_prefix":"ed46f89654a952a1","entry":"register_model","repo":"techwolf-ai/WorkRB","repo_kind":"found_in_text","path":"src/workrb/registry.py","file_url":"https://github.com/techwolf-ai/WorkRB/blob/HEAD/src/workrb/registry.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ed46f89654a952a1"}},{"code_sha256_prefix":"72dd1924b06256ec","entry":"register_task","repo":"techwolf-ai/WorkRB","repo_kind":"found_in_text","path":"src/workrb/registry.py","file_url":"https://github.com/techwolf-ai/WorkRB/blob/HEAD/src/workrb/registry.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"72dd1924b06256ec"}},{"code_sha256_prefix":"fd47088d1cd25c23","entry":"setup_logger","repo":"techwolf-ai/WorkRB","repo_kind":"found_in_text","path":"src/workrb/logging.py","file_url":"https://github.com/techwolf-ai/WorkRB/blob/HEAD/src/workrb/logging.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fd47088d1cd25c23"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}