{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/reading-stackoverflow-encourages-cheating","title":"Reading StackOverflow Encourages Cheating: Adding Question Text Improves Extractive Code Generation","arxiv_id":"2106.04447","date":"2021-06-08","proceeding":"ACL (NLP4Prog) 2021 8","authors":["Gabriel Orlanski","Alex Gittens"],"abstract":"Answering a programming question using only its title is difficult as salient contextual information is omitted. Based on this observation, we present a corpus of over 40,000 StackOverflow question texts to be used in conjunction with their corresponding intents from the CoNaLa dataset (Yin et al., 2018). Using both the intent and question body, we use BART to establish a baseline BLEU score of 34.35 for this new task. We find further improvements of $2.8\\%$ by combining the mined CoNaLa data with the labeled data to achieve a 35.32 BLEU score. We evaluate prior state-of-the-art CoNaLa models with this additional data and find that our proposed method of using the body and mined data beats the BLEU score of the prior state-of-the-art by $71.96\\%$. Finally, we perform ablations to demonstrate that BART is an unsupervised multimodal learner and examine its extractive behavior. The code and data can be found https://github.com/gabeorlanski/stackoverflow-encourages-cheating.","url_abs":"https://arxiv.org/abs/2106.04447v1","url_pdf":"https://arxiv.org/pdf/2106.04447v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"reading-stackoverflow-encourages-cheating","repo_url":"https://github.com/gabeorlanski/stackoverflow-encourages-cheating","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"code-generation","task_name":"Code Generation"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bart","method_name":"BART"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[{"slug":"conala-ext","name":"CoNaLa-Ext","full_name":"CoNaLa Extended With Question Text"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/code-generation-on-conala","task":"Code Generation","dataset":"CoNaLa","model":"BART W/ Mined","rank_in_archive_order":9,"of":14,"metrics":{"BLEU":"30.55"},"uses_additional_data":false},{"leaderboard":"/sota/code-generation-on-conala","task":"Code Generation","dataset":"CoNaLa","model":"BART Base","rank_in_archive_order":13,"of":14,"metrics":{"BLEU":"26.24"},"uses_additional_data":false},{"leaderboard":"/sota/code-generation-on-conala-ext","task":"Code Generation","dataset":"CoNaLa-Ext","model":"BART W/ Mined","rank_in_archive_order":1,"of":6,"metrics":{"BLEU":"35.32"},"uses_additional_data":false},{"leaderboard":"/sota/code-generation-on-conala-ext","task":"Code Generation","dataset":"CoNaLa-Ext","model":"BART Base","rank_in_archive_order":2,"of":6,"metrics":{"BLEU":"34.35"},"uses_additional_data":false}],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}