{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/impact-of-the-query-set-on-the-evaluation-of","title":"Impact of the Query Set on the Evaluation of Expert Finding Systems","arxiv_id":"1806.10813","date":"2018-06-28","proceeding":null,"authors":["Brochier Robin  ERIC","Guille Adrien  ERIC","Rothan Benjamin  ERIC","Velcin Julien  ERIC"],"abstract":"Expertise is a loosely defined concept that is hard to formalize. Much\nresearch has focused on designing efficient algorithms for expert finding in\nlarge databases in various application domains. The evaluation of such\nrecommender systems lies most of the time on human-annotated sets of experts\nassociated with topics. The protocol of evaluation consists in using the\nnamings or short descriptions of these topics as raw queries in order to rank\nthe available set of candidates. Several measures taken from the field of\ninformation retrieval are then applied to rate the rankings of candidates\nagainst the ground truth set of experts. In this paper, we apply this\ntopic-query evaluation methodology with the AMiner data and explore a new\ndocument-query methodology to evaluate experts retrieval from a set of queries\nsampled directly from the experts documents. Specifically, we describe two\ndatasets extracted from AMiner, three baseline algorithms from the literature\nbased on several document representations and provide experiment results to\nshow that using a wide range of more realistic queries provides different\nevaluation results to the usual topic-queries.","url_abs":"http://arxiv.org/abs/1806.10813v1","url_pdf":"http://arxiv.org/pdf/1806.10813v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"impact-of-the-query-set-on-the-evaluation-of","repo_url":"https://github.com/brochier/impact_query_expert_finding","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"information-retrieval","task_name":"Information Retrieval"},{"task_slug":"recommendation-systems","task_name":"Recommendation Systems"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}