{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/zero-shot-learning-for-code-education-rubric","title":"Zero Shot Learning for Code Education: Rubric Sampling with Deep Learning Inference","arxiv_id":"1809.01357","date":"2018-09-05","proceeding":null,"authors":["Mike Wu","Milan Mosse","Noah Goodman","Chris Piech"],"abstract":"In modern computer science education, massive open online courses (MOOCs) log\nthousands of hours of data about how students solve coding challenges. Being so\nrich in data, these platforms have garnered the interest of the machine\nlearning community, with many new algorithms attempting to autonomously provide\nfeedback to help future students learn. But what about those first hundred\nthousand students? In most educational contexts (i.e. classrooms), assignments\ndo not have enough historical data for supervised learning. In this paper, we\nintroduce a human-in-the-loop \"rubric sampling\" approach to tackle the \"zero\nshot\" feedback challenge. We are able to provide autonomous feedback for the\nfirst students working on an introductory programming assignment with accuracy\nthat substantially outperforms data-hungry algorithms and approaches human\nlevel fidelity. Rubric sampling requires minimal teacher effort, can associate\nfeedback with specific parts of a student's solution and can articulate a\nstudent's misconceptions in the language of the instructor. Deep learning\ninference enables rubric sampling to further improve as more assignment\nspecific student data is acquired. We demonstrate our results on a novel\ndataset from Code.org, the world's largest programming education platform.","url_abs":"http://arxiv.org/abs/1809.01357v2","url_pdf":"http://arxiv.org/pdf/1809.01357v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"zero-shot-learning-for-code-education-rubric","repo_url":"https://github.com/mhw32/rubric-sampling-public","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"misconceptions","task_name":"Misconceptions"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1809.01357","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1809.01357"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mhw32/rubric-sampling-public","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"unverified":5},"by_repo_kind":{"official":{"samples":6,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0f786c407fb1ee4c","entry":"swish","repo":"mhw32/rubric-sampling-public","repo_kind":"official","path":"rubric_sampling/experiments/models.py","file_url":"https://github.com/mhw32/rubric-sampling-public/blob/HEAD/rubric_sampling/experiments/models.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f786c407fb1ee4c"}},{"code_sha256_prefix":"7ec7551b16cec75e","entry":"build_rank_map_from_count_map","repo":"mhw32/rubric-sampling-public","repo_kind":"official","path":"rubric_sampling/experiments/datasets.py","file_url":"https://github.com/mhw32/rubric-sampling-public/blob/HEAD/rubric_sampling/experiments/datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7ec7551b16cec75e"}},{"code_sha256_prefix":"c704b64855b7e5f8","entry":"build_zipf_map","repo":"mhw32/rubric-sampling-public","repo_kind":"official","path":"rubric_sampling/experiments/datasets.py","file_url":"https://github.com/mhw32/rubric-sampling-public/blob/HEAD/rubric_sampling/experiments/datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c704b64855b7e5f8"}},{"code_sha256_prefix":"c3152b97d79c813a","entry":"p_program_elbo","repo":"mhw32/rubric-sampling-public","repo_kind":"official","path":"rubric_sampling/experiments/loss.py","file_url":"https://github.com/mhw32/rubric-sampling-public/blob/HEAD/rubric_sampling/experiments/loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c3152b97d79c813a"}},{"code_sha256_prefix":"6c1e981c6d209765","entry":"p_program_label_elbo","repo":"mhw32/rubric-sampling-public","repo_kind":"official","path":"rubric_sampling/experiments/loss.py","file_url":"https://github.com/mhw32/rubric-sampling-public/blob/HEAD/rubric_sampling/experiments/loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6c1e981c6d209765"}},{"code_sha256_prefix":"bb59142c78cc142e","entry":"p_program_label_melbo","repo":"mhw32/rubric-sampling-public","repo_kind":"official","path":"rubric_sampling/experiments/loss.py","file_url":"https://github.com/mhw32/rubric-sampling-public/blob/HEAD/rubric_sampling/experiments/loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bb59142c78cc142e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}