{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/stance-detection-benchmark-how-robust-is-your","title":"Stance Detection Benchmark: How Robust Is Your Stance Detection?","arxiv_id":"2001.01565","date":"2020-01-06","proceeding":null,"authors":["Benjamin Schiller","Johannes Daxenberger","Iryna Gurevych"],"abstract":"Stance Detection (StD) aims to detect an author's stance towards a certain topic or claim and has become a key component in applications like fake news detection, claim validation, and argument search. However, while stance is easily detected by humans, machine learning models are clearly falling short of this task. Given the major differences in dataset sizes and framing of StD (e.g. number of classes and inputs), we introduce a StD benchmark that learns from ten StD datasets of various domains in a multi-dataset learning (MDL) setting, as well as from related tasks via transfer learning. Within this benchmark setup, we are able to present new state-of-the-art results on five of the datasets. Yet, the models still perform well below human capabilities and even simple adversarial attacks severely hurt the performance of MDL models. Deeper investigation into this phenomenon suggests the existence of biases inherited from multiple datasets by design. Our analysis emphasizes the need of focus on robustness and de-biasing strategies in multi-task learning approaches. The benchmark dataset and code is made available.","url_abs":"https://arxiv.org/abs/2001.01565v1","url_pdf":"https://arxiv.org/pdf/2001.01565v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"stance-detection-benchmark-how-robust-is-your","repo_url":"https://github.com/UKPLab/mdl-stance-robustness","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"fake-news-detection","task_name":"Fake News Detection"},{"task_slug":"multi-task-learning","task_name":"Multi-Task Learning"},{"task_slug":"stance-detection","task_name":"Stance Detection"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"}],"methods":[{"method_slug":"mdl","method_name":"MDL"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2001.01565","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2001.01565"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/UKPLab/mdl-stance-robustness","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":2,"unverified":8},"by_repo_kind":{"official":{"samples":10,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"691b566282a42d88","entry":"linear","repo":"UKPLab/mdl-stance-robustness","repo_kind":"official","path":"module/common.py","file_url":"https://github.com/UKPLab/mdl-stance-robustness/blob/HEAD/module/common.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"691b566282a42d88"}},{"code_sha256_prefix":"dea985d5f4e928e8","entry":"warmup_linear_xdl","repo":"UKPLab/mdl-stance-robustness","repo_kind":"official","path":"module/bert_optim.py","file_url":"https://github.com/UKPLab/mdl-stance-robustness/blob/HEAD/module/bert_optim.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dea985d5f4e928e8"}},{"code_sha256_prefix":"b9de0da0c0405d07","entry":"create_sample_list_pair","repo":"UKPLab/mdl-stance-robustness","repo_kind":"official","path":"data_utils/glue_utils.py","file_url":"https://github.com/UKPLab/mdl-stance-robustness/blob/HEAD/data_utils/glue_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b9de0da0c0405d07"}},{"code_sha256_prefix":"7e4f856cc2416800","entry":"create_sample_list_single","repo":"UKPLab/mdl-stance-robustness","repo_kind":"official","path":"data_utils/glue_utils.py","file_url":"https://github.com/UKPLab/mdl-stance-robustness/blob/HEAD/data_utils/glue_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e4f856cc2416800"}},{"code_sha256_prefix":"53ebf6e30980ffe9","entry":"gelu","repo":"UKPLab/mdl-stance-robustness","repo_kind":"official","path":"module/common.py","file_url":"https://github.com/UKPLab/mdl-stance-robustness/blob/HEAD/module/common.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"53ebf6e30980ffe9"}},{"code_sha256_prefix":"a45d66c0712a81bd","entry":"generate_decoder_opt","repo":"UKPLab/mdl-stance-robustness","repo_kind":"official","path":"data_utils/label_map.py","file_url":"https://github.com/UKPLab/mdl-stance-robustness/blob/HEAD/data_utils/label_map.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a45d66c0712a81bd"}},{"code_sha256_prefix":"809cfdb94234ea27","entry":"generate_mask","repo":"UKPLab/mdl-stance-robustness","repo_kind":"official","path":"module/san.py","file_url":"https://github.com/UKPLab/mdl-stance-robustness/blob/HEAD/module/san.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"809cfdb94234ea27"}},{"code_sha256_prefix":"8dbe6dcc5c31662f","entry":"get_random_indices","repo":"UKPLab/mdl-stance-robustness","repo_kind":"official","path":"prepro.py","file_url":"https://github.com/UKPLab/mdl-stance-robustness/blob/HEAD/prepro.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8dbe6dcc5c31662f"}},{"code_sha256_prefix":"47cf8803722c3336","entry":"swish","repo":"UKPLab/mdl-stance-robustness","repo_kind":"official","path":"module/common.py","file_url":"https://github.com/UKPLab/mdl-stance-robustness/blob/HEAD/module/common.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"47cf8803722c3336"}},{"code_sha256_prefix":"0f9a0dbc776dbc60","entry":"weight_norm","repo":"UKPLab/mdl-stance-robustness","repo_kind":"official","path":"module/my_optim.py","file_url":"https://github.com/UKPLab/mdl-stance-robustness/blob/HEAD/module/my_optim.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f9a0dbc776dbc60"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}