{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/guiding-language-models-of-code-with-global","title":"Guiding Language Models of Code with Global Context using Monitors","arxiv_id":"2306.10763","date":"2023-06-19","proceeding":null,"authors":["Lakshya A Agrawal","Aditya Kanade","Navin Goyal","Shuvendu K. Lahiri","Sriram K. Rajamani"],"abstract":"Language models of code (LMs) work well when the surrounding code provides sufficient context. This is not true when it becomes necessary to use types, functionality or APIs defined elsewhere in the repository or a linked library, especially those not seen during training. LMs suffer from limited awareness of such global context and end up hallucinating. Integrated development environments (IDEs) assist developers in understanding repository context using static analysis. We extend this assistance, enjoyed by developers, to LMs. We propose monitor-guided decoding (MGD) where a monitor uses static analysis to guide the decoding. We construct a repository-level dataset PragmaticCode for method-completion in Java and evaluate MGD on it. On models of varying parameter scale, by monitoring for type-consistent object dereferences, MGD consistently improves compilation rates and agreement with ground truth. Further, LMs with fewer parameters, when augmented with MGD, can outperform larger LMs. With MGD, SantaCoder-1.1B achieves better compilation rate and next-identifier match than the much larger text-davinci-003 model. We also conduct a generalizability study to evaluate the ability of MGD to generalize to multiple programming languages (Java, C# and Rust), coding scenarios (e.g., correct number of arguments to method calls), and to enforce richer semantic constraints (e.g., stateful API protocols). Our data and implementation are available at https://github.com/microsoft/monitors4codegen .","url_abs":"https://arxiv.org/abs/2306.10763v2","url_pdf":"https://arxiv.org/pdf/2306.10763v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"guiding-language-models-of-code-with-global","repo_url":"https://github.com/microsoft/monitors4codegen","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"code-completion","task_name":"Code Completion"},{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"code-repair","task_name":"Code Repair"},{"task_slug":"code-translation","task_name":"Code Translation"},{"task_slug":"text-to-code-generation","task_name":"Text-to-Code Generation"}],"methods":[],"datasets_introduced":[{"slug":"dotprompts","name":"DotPrompts","full_name":""},{"slug":"pragmaticcode","name":"PragmaticCode","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/code-completion-on-dotprompts","task":"Code Completion","dataset":"DotPrompts","model":"SantaCoder-MGD","rank_in_archive_order":1,"of":3,"metrics":{"Compilation Rate":"73.03"},"uses_additional_data":false},{"leaderboard":"/sota/code-completion-on-dotprompts","task":"Code Completion","dataset":"DotPrompts","model":"SantaCoder","rank_in_archive_order":3,"of":3,"metrics":{"Compilation Rate":"59.79"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2306.10763","atlas_url":"https://app.syntology.ai/?focus=2306.10763","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.10763"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/microsoft/monitors4codegen","reach":null}],"summary":{"ran_honours":1,"ran_fixture":1},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"81302d4943b3cecc","entry":"pass_at_k","repo":"microsoft/monitors4codegen","repo_kind":"official","path":"evaluation_scripts/eval_results.py","file_url":"https://github.com/microsoft/monitors4codegen/blob/HEAD/evaluation_scripts/eval_results.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"81302d4943b3cecc"}},{"code_sha256_prefix":"e5f4563d018e7be4","entry":"score_at_k","repo":"microsoft/monitors4codegen","repo_kind":"official","path":"evaluation_scripts/eval_results.py","file_url":"https://github.com/microsoft/monitors4codegen/blob/HEAD/evaluation_scripts/eval_results.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e5f4563d018e7be4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}