{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/automating-human-tutor-style-programming","title":"Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hint Validation","arxiv_id":"2310.03780","date":"2023-10-05","proceeding":null,"authors":["Tung Phung","Victor-Alexandru Pădurean","Anjali Singh","Christopher Brooks","José Cambronero","Sumit Gulwani","Adish Singla","Gustavo Soares"],"abstract":"Generative AI and large language models hold great promise in enhancing programming education by automatically generating individualized feedback for students. We investigate the role of generative AI models in providing human tutor-style programming hints to help students resolve errors in their buggy programs. Recent works have benchmarked state-of-the-art models for various feedback generation scenarios; however, their overall quality is still inferior to human tutors and not yet ready for real-world deployment. In this paper, we seek to push the limits of generative AI models toward providing high-quality programming hints and develop a novel technique, GPT4Hints-GPT3.5Val. As a first step, our technique leverages GPT-4 as a ``tutor'' model to generate hints -- it boosts the generative quality by using symbolic information of failing test cases and fixes in prompts. As a next step, our technique leverages GPT-3.5, a weaker model, as a ``student'' model to further validate the hint quality -- it performs an automatic quality validation by simulating the potential utility of providing this feedback. We show the efficacy of our technique via extensive evaluation using three real-world datasets of Python programs covering a variety of concepts ranging from basic algorithms to regular expressions and data analysis using pandas library.","url_abs":"https://arxiv.org/abs/2310.03780v4","url_pdf":"https://arxiv.org/pdf/2310.03780v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"automating-human-tutor-style-programming","repo_url":"https://github.com/machine-teaching-group/lak2024_gpt4-hints-gpt3.5val","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"automating-human-tutor-style-programming","repo_url":"https://github.com/machine-teaching-group/lak2024_gpt4hints-gpt3.5val","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"hint-generation","task_name":"Hint Generation"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"hint","method_name":"HINT"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.03780","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.03780"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/machine-teaching-group/lak2024_gpt4-hints-gpt3.5val","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/machine-teaching-group/lak2024_gpt4hints-gpt3.5val","reach":{"status":"ok"}}],"summary":{"ran":10},"by_repo_kind":{"official":{"samples":10,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"3e7a5e860669ee13","entry":"compute_edit_distance","repo":"machine-teaching-group/lak2024_gpt4-hints-gpt3.5val","repo_kind":"official","path":"src/utils/edit_distance_utils.py","file_url":"https://github.com/machine-teaching-group/lak2024_gpt4-hints-gpt3.5val/blob/HEAD/src/utils/edit_distance_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3e7a5e860669ee13"}},{"code_sha256_prefix":"69c0fd6000b270f4","entry":"format_BasicAlgo_failing_testcase","repo":"machine-teaching-group/lak2024_gpt4-hints-gpt3.5val","repo_kind":"official","path":"src/utils/basicalgo_utils.py","file_url":"https://github.com/machine-teaching-group/lak2024_gpt4-hints-gpt3.5val/blob/HEAD/src/utils/basicalgo_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"69c0fd6000b270f4"}},{"code_sha256_prefix":"58e327c1ae57c2e1","entry":"format_BasicAlgo_problem_description","repo":"machine-teaching-group/lak2024_gpt4-hints-gpt3.5val","repo_kind":"official","path":"src/utils/basicalgo_utils.py","file_url":"https://github.com/machine-teaching-group/lak2024_gpt4-hints-gpt3.5val/blob/HEAD/src/utils/basicalgo_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"58e327c1ae57c2e1"}},{"code_sha256_prefix":"f819e5e08327840f","entry":"format_BasicAlgo_resources","repo":"machine-teaching-group/lak2024_gpt4-hints-gpt3.5val","repo_kind":"official","path":"src/utils/basicalgo_utils.py","file_url":"https://github.com/machine-teaching-group/lak2024_gpt4-hints-gpt3.5val/blob/HEAD/src/utils/basicalgo_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f819e5e08327840f"}},{"code_sha256_prefix":"c2ba6334a2c16138","entry":"lex_program","repo":"machine-teaching-group/lak2024_gpt4-hints-gpt3.5val","repo_kind":"official","path":"src/utils/program_utils.py","file_url":"https://github.com/machine-teaching-group/lak2024_gpt4-hints-gpt3.5val/blob/HEAD/src/utils/program_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c2ba6334a2c16138"}},{"code_sha256_prefix":"fb40912c4c110ea2","entry":"load_file","repo":"machine-teaching-group/lak2024_gpt4-hints-gpt3.5val","repo_kind":"official","path":"src/utils/IO_utils.py","file_url":"https://github.com/machine-teaching-group/lak2024_gpt4-hints-gpt3.5val/blob/HEAD/src/utils/IO_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fb40912c4c110ea2"}},{"code_sha256_prefix":"fe9ad0cf4f179871","entry":"load_json_file","repo":"machine-teaching-group/lak2024_gpt4-hints-gpt3.5val","repo_kind":"official","path":"src/utils/IO_utils.py","file_url":"https://github.com/machine-teaching-group/lak2024_gpt4-hints-gpt3.5val/blob/HEAD/src/utils/IO_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fe9ad0cf4f179871"}},{"code_sha256_prefix":"b5fb7d9d80d26fc0","entry":"load_openai_api","repo":"machine-teaching-group/lak2024_gpt4-hints-gpt3.5val","repo_kind":"official","path":"src/utils/openai_utils.py","file_url":"https://github.com/machine-teaching-group/lak2024_gpt4-hints-gpt3.5val/blob/HEAD/src/utils/openai_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b5fb7d9d80d26fc0"}},{"code_sha256_prefix":"763f55e28e917d6a","entry":"load_program","repo":"machine-teaching-group/lak2024_gpt4-hints-gpt3.5val","repo_kind":"official","path":"src/utils/program_utils.py","file_url":"https://github.com/machine-teaching-group/lak2024_gpt4-hints-gpt3.5val/blob/HEAD/src/utils/program_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"763f55e28e917d6a"}},{"code_sha256_prefix":"5b099dfa4568e68b","entry":"program_to_essential_tokens","repo":"machine-teaching-group/lak2024_gpt4-hints-gpt3.5val","repo_kind":"official","path":"src/utils/program_utils.py","file_url":"https://github.com/machine-teaching-group/lak2024_gpt4-hints-gpt3.5val/blob/HEAD/src/utils/program_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5b099dfa4568e68b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}