{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/bringing-replication-and-reproduction","title":"Bringing replication and reproduction together with generalisability in NLP: Three reproduction studies for Target Dependent Sentiment Analysis","arxiv_id":"1806.05219","date":"2018-06-13","proceeding":"COLING 2018 8","authors":["Andrew Moore","Paul Rayson"],"abstract":"Lack of repeatability and generalisability are two significant threats to\ncontinuing scientific development in Natural Language Processing. Language\nmodels and learning methods are so complex that scientific conference papers no\nlonger contain enough space for the technical depth required for replication or\nreproduction. Taking Target Dependent Sentiment Analysis as a case study, we\nshow how recent work in the field has not consistently released code, or\ndescribed settings for learning methods in enough detail, and lacks\ncomparability and generalisability in train, test or validation data. To\ninvestigate generalisability and to enable state of the art comparative\nevaluations, we carry out the first reproduction studies of three groups of\ncomplementary methods and perform the first large-scale mass evaluation on six\ndifferent English datasets. Reflecting on our experiences, we recommend that\nfuture replication or reproduction experiments should always consider a variety\nof datasets alongside documenting and releasing their methods and published\ncode in order to minimise the barriers to both repeatability and\ngeneralisability. We have released our code with a model zoo on GitHub with\nJupyter Notebooks to aid understanding and full documentation, and we recommend\nthat others do the same with their papers at submission time through an\nanonymised GitHub account.","url_abs":"http://arxiv.org/abs/1806.05219v2","url_pdf":"http://arxiv.org/pdf/1806.05219v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"bringing-replication-and-reproduction","repo_url":"https://github.com/apmoore1/Bella","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"sentiment-analysis","task_name":"Sentiment Analysis"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1806.05219","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}