{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/automated-evaluation-of-out-of-context-errors","title":"Automated Evaluation of Out-of-Context Errors","arxiv_id":"1803.08983","date":"2018-03-23","proceeding":"LREC 2018 5","authors":["Patrick Huber","Jan Niehues","Alex Waibel"],"abstract":"We present a new approach to evaluate computational models for the task of\ntext understanding by the means of out-of-context error detection. Through the\nnovel design of our automated modification process, existing large-scale data\nsources can be adopted for a vast number of text understanding tasks. The data\nis thereby altered on a semantic level, allowing models to be tested against a\nchallenging set of modified text passages that require to comprise a broader\nnarrative discourse. Our newly introduced task targets actual real-world\nproblems of transcription and translation systems by inserting authentic\nout-of-context errors. The automated modification process is applied to the\n2016 TEDTalk corpus. Entirely automating the process allows the adoption of\ncomplete datasets at low cost, facilitating supervised learning procedures and\ndeeper networks to be trained and tested. To evaluate the quality of the\nmodification algorithm a language model and a supervised binary classification\nmodel are trained and tested on the altered dataset. A human baseline\nevaluation is examined to compare the results with human performance. The\noutcome of the evaluation task indicates the difficulty to detect semantic\nerrors for machine-learning algorithms and humans, showing that the errors\ncannot be identified when limited to a single sentence.","url_abs":"http://arxiv.org/abs/1803.08983v1","url_pdf":"http://arxiv.org/pdf/1803.08983v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"automated-evaluation-of-out-of-context-errors","repo_url":"https://github.com/isl-mt/SemanticWordReplacement-LREC2018","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"binary-classification","task_name":"Binary Classification"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"translation","task_name":"Translation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}