{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/when-to-show-a-suggestion-integrating-human","title":"When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming","arxiv_id":"2306.04930","date":"2023-06-08","proceeding":null,"authors":["Hussein Mozannar","Gagan Bansal","Adam Fourney","Eric Horvitz"],"abstract":"AI powered code-recommendation systems, such as Copilot and CodeWhisperer, provide code suggestions inside a programmer's environment (e.g., an IDE) with the aim of improving productivity. We pursue mechanisms for leveraging signals about programmers' acceptance and rejection of code suggestions to guide recommendations. We harness data drawn from interactions with GitHub Copilot, a system used by millions of programmers, to develop interventions that can save time for programmers. We introduce a utility-theoretic framework to drive decisions about suggestions to display versus withhold. The approach, conditional suggestion display from human feedback (CDHF), relies on a cascade of models that provide the likelihood that recommended code will be accepted. These likelihoods are used to selectively hide suggestions, reducing both latency and programmer verification time. Using data from 535 programmers, we perform a retrospective evaluation of CDHF and show that we can avoid displaying a significant fraction of suggestions that would have been rejected. We further demonstrate the importance of incorporating the programmer's latent unobserved state in decisions about when to display suggestions through an ablation study. Finally, we showcase how using suggestion acceptance as a reward signal for guiding the display of suggestions can lead to suggestions of reduced quality, indicating an unexpected pitfall.","url_abs":"https://arxiv.org/abs/2306.04930v3","url_pdf":"https://arxiv.org/pdf/2306.04930v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"when-to-show-a-suggestion-integrating-human","repo_url":"https://github.com/microsoft/coderec_programming_states","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"recommendation-systems","task_name":"Recommendation Systems"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2306.04930","atlas_url":"https://app.syntology.ai/?focus=2306.04930","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.04930"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/coderec_programming_states","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":19},"by_repo_kind":{"official":{"samples":19,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ed19bdc4230daed5","entry":"bootstrap_confidence_interval","repo":"microsoft/coderec_programming_states","repo_kind":"official","path":"when_to_show/metrics_get.py","file_url":"https://github.com/microsoft/coderec_programming_states/blob/HEAD/when_to_show/metrics_get.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed19bdc4230daed5"}},{"code_sha256_prefix":"0490eac5bafb4137","entry":"crude_label","repo":"microsoft/coderec_programming_states","repo_kind":"official","path":"when_to_show/get_code_label.py","file_url":"https://github.com/microsoft/coderec_programming_states/blob/HEAD/when_to_show/get_code_label.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0490eac5bafb4137"}},{"code_sha256_prefix":"81dc0848a2719a79","entry":"discretemap","repo":"microsoft/coderec_programming_states","repo_kind":"official","path":"between_the_lines/user_study_analysis/colorblind_colors.py","file_url":"https://github.com/microsoft/coderec_programming_states/blob/HEAD/between_the_lines/user_study_analysis/colorblind_colors.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"81dc0848a2719a79"}},{"code_sha256_prefix":"5397058661fb4a5c","entry":"entropy_single_probability","repo":"microsoft/coderec_programming_states","repo_kind":"official","path":"when_to_show/cdhf.py","file_url":"https://github.com/microsoft/coderec_programming_states/blob/HEAD/when_to_show/cdhf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5397058661fb4a5c"}},{"code_sha256_prefix":"943614b7c2a70665","entry":"get_df_all","repo":"microsoft/coderec_programming_states","repo_kind":"official","path":"when_to_show/extended_logs/extended_logs_mass.py","file_url":"https://github.com/microsoft/coderec_programming_states/blob/HEAD/when_to_show/extended_logs/extended_logs_mass.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"943614b7c2a70665"}},{"code_sha256_prefix":"213f0c818c404589","entry":"get_embedding_list","repo":"microsoft/coderec_programming_states","repo_kind":"official","path":"when_to_show/generate_features.py","file_url":"https://github.com/microsoft/coderec_programming_states/blob/HEAD/when_to_show/generate_features.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"213f0c818c404589"}},{"code_sha256_prefix":"434292e5dfea98da","entry":"get_extended_states","repo":"microsoft/coderec_programming_states","repo_kind":"official","path":"when_to_show/extended_logs/extended_logs_mass.py","file_url":"https://github.com/microsoft/coderec_programming_states/blob/HEAD/when_to_show/extended_logs/extended_logs_mass.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"434292e5dfea98da"}},{"code_sha256_prefix":"e57f8705e5725835","entry":"get_features_labels","repo":"microsoft/coderec_programming_states","repo_kind":"official","path":"when_to_show/data_split.py","file_url":"https://github.com/microsoft/coderec_programming_states/blob/HEAD/when_to_show/data_split.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e57f8705e5725835"}},{"code_sha256_prefix":"1bfeef7a43e6e84b","entry":"get_features_labels_user_study","repo":"microsoft/coderec_programming_states","repo_kind":"official","path":"when_to_show/data_split.py","file_url":"https://github.com/microsoft/coderec_programming_states/blob/HEAD/when_to_show/data_split.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1bfeef7a43e6e84b"}},{"code_sha256_prefix":"53988409e494aa17","entry":"get_index","repo":"microsoft/coderec_programming_states","repo_kind":"official","path":"between_the_lines/predict_cups/data_split.py","file_url":"https://github.com/microsoft/coderec_programming_states/blob/HEAD/between_the_lines/predict_cups/data_split.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"53988409e494aa17"}},{"code_sha256_prefix":"73a8698bc54cc3fb","entry":"local_entropy","repo":"microsoft/coderec_programming_states","repo_kind":"official","path":"between_the_lines/user_study_analysis/entropy_utils.py","file_url":"https://github.com/microsoft/coderec_programming_states/blob/HEAD/between_the_lines/user_study_analysis/entropy_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"73a8698bc54cc3fb"}},{"code_sha256_prefix":"4b248dd17b83c8a3","entry":"process_data_user_study","repo":"microsoft/coderec_programming_states","repo_kind":"official","path":"when_to_show/data_split.py","file_url":"https://github.com/microsoft/coderec_programming_states/blob/HEAD/when_to_show/data_split.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4b248dd17b83c8a3"}},{"code_sha256_prefix":"380ce40e3722bc67","entry":"retreive_logs","repo":"microsoft/coderec_programming_states","repo_kind":"official","path":"when_to_show/extended_logs/extended_logs.py","file_url":"https://github.com/microsoft/coderec_programming_states/blob/HEAD/when_to_show/extended_logs/extended_logs.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"380ce40e3722bc67"}},{"code_sha256_prefix":"dfef76f9d70850bf","entry":"stationary_distribution","repo":"microsoft/coderec_programming_states","repo_kind":"official","path":"between_the_lines/user_study_analysis/entropy_utils.py","file_url":"https://github.com/microsoft/coderec_programming_states/blob/HEAD/between_the_lines/user_study_analysis/entropy_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dfef76f9d70850bf"}},{"code_sha256_prefix":"6545175d9ac9fa91","entry":"text_features","repo":"microsoft/coderec_programming_states","repo_kind":"official","path":"when_to_show/generate_features.py","file_url":"https://github.com/microsoft/coderec_programming_states/blob/HEAD/when_to_show/generate_features.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6545175d9ac9fa91"}},{"code_sha256_prefix":"9d13457c7a04bdb3","entry":"tol_cmap","repo":"microsoft/coderec_programming_states","repo_kind":"official","path":"between_the_lines/user_study_analysis/colorblind_colors.py","file_url":"https://github.com/microsoft/coderec_programming_states/blob/HEAD/between_the_lines/user_study_analysis/colorblind_colors.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9d13457c7a04bdb3"}},{"code_sha256_prefix":"52fb834d8edc9f71","entry":"tol_cset","repo":"microsoft/coderec_programming_states","repo_kind":"official","path":"between_the_lines/user_study_analysis/colorblind_colors.py","file_url":"https://github.com/microsoft/coderec_programming_states/blob/HEAD/between_the_lines/user_study_analysis/colorblind_colors.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"52fb834d8edc9f71"}},{"code_sha256_prefix":"855f8589acc66aff","entry":"trajectory_entropy","repo":"microsoft/coderec_programming_states","repo_kind":"official","path":"between_the_lines/user_study_analysis/entropy_utils.py","file_url":"https://github.com/microsoft/coderec_programming_states/blob/HEAD/between_the_lines/user_study_analysis/entropy_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"855f8589acc66aff"}},{"code_sha256_prefix":"cf65280e02f4e416","entry":"traverse_tree","repo":"microsoft/coderec_programming_states","repo_kind":"official","path":"when_to_show/get_code_label.py","file_url":"https://github.com/microsoft/coderec_programming_states/blob/HEAD/when_to_show/get_code_label.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cf65280e02f4e416"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}