{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2606-06673","title":"Uncertainty-Aware LLM-Guided Policy Shaping for Sparse-Reward Reinforcement Learning","arxiv_id":"2606.06673","date":"2026-06-04","proceeding":null,"authors":["Ujjwal Bhatta","Utsabi Dangol","Sumaly Bajracharya","Rodrigue Rizk","KC Santosh"],"abstract":"Sparse rewards and heterogeneous task sequences remain persistent challenges in Reinforcement Learning (RL), often resulting in slow convergence, weak generalization, and inefficient exploration. We propose Uncertainty-Aware LLM-Guided Policy Shaping (ULPS), a novel framework that integrates a calibrated Large Language Model (LLM) into the RL training loop to provide structured, uncertainty-modulated behavioral guidance. ULPS employs an A*-based oracle to synthesize optimal symbolic trajectories, which are used to fine-tune a BERT-based language model. During training, this model supplies action suggestions whose influence is conditioned on epistemic uncertainty estimated via Monte Carlo (MC) dropout. An entropy-based blending mechanism adaptively balances LLM guidance and the learned policy (via Proximal Policy Optimization, PPO), allowing the agent to prioritize reliable priors while preserving adaptability. We evaluate ULPS on the MiniGridUnlockPickup benchmark and observe consistent improvements in success rate, reward efficiency, and sample complexity over unguided, uncalibrated, and standard RL baselines. ULPS achieves more than 9% improvement in execution accuracy after fine-tuning, requires fewer environment interactions, and yields higher reward AUC. Our results demonstrate that integrating symbolic A* trajectories, pretrained language priors, and uncertainty-aware control offers a principled and effective approach to multi-task reinforcement learning in sparse-reward domains, with potential extensibility to partially observable and multi-agent settings.","url_abs":"https://arxiv.org/abs/2606.06673","url_pdf":"https://arxiv.org/pdf/2606.06673","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2606.06673","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2606.06673"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/USD-AI-ResearchLab/uncertainty-aware-llm-rl","reach":null}],"summary":{"ran":5,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"88a6169feb702691","entry":"calculate_performance_metrics","repo":"USD-AI-ResearchLab/uncertainty-aware-llm-rl","repo_kind":"found_in_text","path":"MainCode/plot.py","file_url":"https://github.com/USD-AI-ResearchLab/uncertainty-aware-llm-rl/blob/HEAD/MainCode/plot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"88a6169feb702691"}},{"code_sha256_prefix":"9df81ed6266bc82a","entry":"compute_brier_score","repo":"USD-AI-ResearchLab/uncertainty-aware-llm-rl","repo_kind":"found_in_text","path":"DQN/calibrated_llm_core.py","file_url":"https://github.com/USD-AI-ResearchLab/uncertainty-aware-llm-rl/blob/HEAD/DQN/calibrated_llm_core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9df81ed6266bc82a"}},{"code_sha256_prefix":"fe42a728494144d0","entry":"compute_discrimination","repo":"USD-AI-ResearchLab/uncertainty-aware-llm-rl","repo_kind":"found_in_text","path":"DQN/calibrated_llm_core.py","file_url":"https://github.com/USD-AI-ResearchLab/uncertainty-aware-llm-rl/blob/HEAD/DQN/calibrated_llm_core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fe42a728494144d0"}},{"code_sha256_prefix":"70a93d1cf92af554","entry":"compute_ece","repo":"USD-AI-ResearchLab/uncertainty-aware-llm-rl","repo_kind":"found_in_text","path":"DQN/calibrated_llm_core.py","file_url":"https://github.com/USD-AI-ResearchLab/uncertainty-aware-llm-rl/blob/HEAD/DQN/calibrated_llm_core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"70a93d1cf92af554"}},{"code_sha256_prefix":"4de5f1119e2fa623","entry":"load_experiment_data","repo":"USD-AI-ResearchLab/uncertainty-aware-llm-rl","repo_kind":"found_in_text","path":"MainCode/plot.py","file_url":"https://github.com/USD-AI-ResearchLab/uncertainty-aware-llm-rl/blob/HEAD/MainCode/plot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4de5f1119e2fa623"}},{"code_sha256_prefix":"b5de7ba519d7357b","entry":"plot_performance_curves","repo":"USD-AI-ResearchLab/uncertainty-aware-llm-rl","repo_kind":"found_in_text","path":"MainCode/plot.py","file_url":"https://github.com/USD-AI-ResearchLab/uncertainty-aware-llm-rl/blob/HEAD/MainCode/plot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b5de7ba519d7357b"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}