{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rainier-reinforced-knowledge-introspector-for","title":"Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering","arxiv_id":"2210.03078","date":"2022-10-06","proceeding":null,"authors":["Jiacheng Liu","Skyler Hallinan","Ximing Lu","Pengfei He","Sean Welleck","Hannaneh Hajishirzi","Yejin Choi"],"abstract":"Knowledge underpins reasoning. Recent research demonstrates that when relevant knowledge is provided as additional context to commonsense question answering (QA), it can substantially enhance the performance even on top of state-of-the-art. The fundamental challenge is where and how to find such knowledge that is high quality and on point with respect to the question; knowledge retrieved from knowledge bases are incomplete and knowledge generated from language models are inconsistent. We present Rainier, or Reinforced Knowledge Introspector, that learns to generate contextually relevant knowledge in response to given questions. Our approach starts by imitating knowledge generated by GPT-3, then learns to generate its own knowledge via reinforcement learning where rewards are shaped based on the increased performance on the resulting question answering. Rainier demonstrates substantial and consistent performance gains when tested over 9 different commonsense benchmarks: including 5 datasets that are seen during model training, as well as 4 datasets that are kept unseen. Our work is the first to report that knowledge generated by models that are orders of magnitude smaller than GPT-3, even without direct supervision on the knowledge itself, can exceed the quality of commonsense knowledge elicited from GPT-3.","url_abs":"https://arxiv.org/abs/2210.03078v2","url_pdf":"https://arxiv.org/pdf/2210.03078v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rainier-reinforced-knowledge-introspector-for","repo_url":"https://github.com/liujch1998/rainier","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2210.03078","atlas_url":"https://app.syntology.ai/?focus=2210.03078","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.03078"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/liujch1998/rainier","reach":null}],"summary":{"ran_honours":4,"ran_draft_wrong":2,"ran_fixture":1,"unverified":6},"by_repo_kind":{"official":{"samples":13,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"37a3441d06cf812e","entry":"clamp","repo":"liujch1998/rainier","repo_kind":"official","path":"rainier/ppo.py","file_url":"https://github.com/liujch1998/rainier/blob/HEAD/rainier/ppo.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"37a3441d06cf812e"}},{"code_sha256_prefix":"601afc113e2c5295","entry":"logits_to_entropy","repo":"liujch1998/rainier","repo_kind":"official","path":"rainier/ppo.py","file_url":"https://github.com/liujch1998/rainier/blob/HEAD/rainier/ppo.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"601afc113e2c5295"}},{"code_sha256_prefix":"a9c6a287d240022c","entry":"mask_pad","repo":"liujch1998/rainier","repo_kind":"official","path":"rainier/ppo.py","file_url":"https://github.com/liujch1998/rainier/blob/HEAD/rainier/ppo.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a9c6a287d240022c"}},{"code_sha256_prefix":"e5bacba887a721ef","entry":"reduce_mean","repo":"liujch1998/rainier","repo_kind":"official","path":"rainier/ppo.py","file_url":"https://github.com/liujch1998/rainier/blob/HEAD/rainier/ppo.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e5bacba887a721ef"}},{"code_sha256_prefix":"24a6def7a7175be1","entry":"reduce_std","repo":"liujch1998/rainier","repo_kind":"official","path":"rainier/ppo.py","file_url":"https://github.com/liujch1998/rainier/blob/HEAD/rainier/ppo.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"24a6def7a7175be1"}},{"code_sha256_prefix":"5874b0a8910897fd","entry":"reduce_sum","repo":"liujch1998/rainier","repo_kind":"official","path":"rainier/ppo.py","file_url":"https://github.com/liujch1998/rainier/blob/HEAD/rainier/ppo.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5874b0a8910897fd"}},{"code_sha256_prefix":"f22da26515d11255","entry":"whiten","repo":"liujch1998/rainier","repo_kind":"official","path":"rainier/ppo.py","file_url":"https://github.com/liujch1998/rainier/blob/HEAD/rainier/ppo.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f22da26515d11255"}},{"code_sha256_prefix":"702f0dd46130de79","entry":"PPOTrainer","repo":"liujch1998/rainier","repo_kind":"official","path":"rainier/ppo.py","file_url":"https://github.com/liujch1998/rainier/blob/HEAD/rainier/ppo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"702f0dd46130de79"}},{"code_sha256_prefix":"af8e5334f66e1240","entry":"Policy","repo":"liujch1998/rainier","repo_kind":"official","path":"rainier/ppo.py","file_url":"https://github.com/liujch1998/rainier/blob/HEAD/rainier/ppo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"af8e5334f66e1240"}},{"code_sha256_prefix":"8e4768065a6e2e46","entry":"Reward","repo":"liujch1998/rainier","repo_kind":"official","path":"rainier/ppo.py","file_url":"https://github.com/liujch1998/rainier/blob/HEAD/rainier/ppo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8e4768065a6e2e46"}},{"code_sha256_prefix":"793d7e2a816c5031","entry":"T5ForConditionalGenerationAndTokenRegression","repo":"liujch1998/rainier","repo_kind":"official","path":"rainier/ppo.py","file_url":"https://github.com/liujch1998/rainier/blob/HEAD/rainier/ppo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"793d7e2a816c5031"}},{"code_sha256_prefix":"1fa7b1b16bcc201e","entry":"T5ForTokenRegression","repo":"liujch1998/rainier","repo_kind":"official","path":"rainier/ppo.py","file_url":"https://github.com/liujch1998/rainier/blob/HEAD/rainier/ppo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1fa7b1b16bcc201e"}},{"code_sha256_prefix":"d77834c9fb30a1e7","entry":"Value","repo":"liujch1998/rainier","repo_kind":"official","path":"rainier/ppo.py","file_url":"https://github.com/liujch1998/rainier/blob/HEAD/rainier/ppo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d77834c9fb30a1e7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}