{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cautious-next-token-prediction","title":"Cautious Next Token Prediction","arxiv_id":"2507.03038","date":"2025-07-03","proceeding":null,"authors":["Yizhou Wang","Lingzhi Zhang","Yue Bai","Mang Tik Chiu","Zhengmian Hu","Mingyuan Zhang","Qihua Dong","Yu Yin","Sohrab Amirghodsi","Yun Fu"],"abstract":"Next token prediction paradigm has been prevailing for autoregressive models in the era of LLMs. The current default sampling choice for popular LLMs is temperature scaling together with nucleus sampling to balance diversity and coherence. Nevertheless, such approach leads to inferior performance in various NLP tasks when the model is not certain about testing questions. To this end, we propose a brand new training-free decoding strategy, dubbed as Cautious Next Token Prediction (CNTP). In the decoding process, if the model has comparatively high prediction entropy at a certain step, we sample multiple trials starting from the step independently and stop when encountering any punctuation. Then we select the trial with the lowest perplexity score viewed as the most probable and reliable trial path given the model's capacity. The trial number is negatively correlated with the prediction confidence, i.e., the less confident the model is, the more trials it should sample. This is consistent with human beings' behaviour: when feeling uncertain or unconfident, one tends to think more creatively, exploring multiple thinking paths, to cautiously select the path one feels most confident about. Extensive experiments on both LLMs and MLLMs show that our proposed CNTP approach outperforms existing standard decoding strategies consistently by a clear margin. Moreover, the integration of CNTP with self consistency can further improve over vanilla self consistency. We believe our proposed CNTP has the potential to become one of the default choices for LLM decoding. Code is available at https://github.com/wyzjack/CNTP.","url_abs":"https://arxiv.org/abs/2507.03038v1","url_pdf":"https://arxiv.org/pdf/2507.03038v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cautious-next-token-prediction","repo_url":"https://github.com/wyzjack/CNTP","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"prediction","task_name":"Prediction"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2507.03038","atlas_url":"https://app.syntology.ai/?focus=2507.03038","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2507.03038"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wyzjack/CNTP","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7,"unverified":7},"by_repo_kind":{"official":{"samples":14,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1adcf2d05b2faed1","entry":"get_imports","repo":"wyzjack/CNTP","repo_kind":"official","path":"custom_transformers_packages/gsm8k_strategyqa/transformers/dynamic_module_utils.py","file_url":"https://github.com/wyzjack/CNTP/blob/HEAD/custom_transformers_packages/gsm8k_strategyqa/transformers/dynamic_module_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1adcf2d05b2faed1"}},{"code_sha256_prefix":"518f6a89865f25d2","entry":"get_relative_import_files","repo":"wyzjack/CNTP","repo_kind":"official","path":"custom_transformers_packages/gsm8k_strategyqa/transformers/dynamic_module_utils.py","file_url":"https://github.com/wyzjack/CNTP/blob/HEAD/custom_transformers_packages/gsm8k_strategyqa/transformers/dynamic_module_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"518f6a89865f25d2"}},{"code_sha256_prefix":"4af5faa701d0f2c9","entry":"get_relative_imports","repo":"wyzjack/CNTP","repo_kind":"official","path":"custom_transformers_packages/gsm8k_strategyqa/transformers/dynamic_module_utils.py","file_url":"https://github.com/wyzjack/CNTP/blob/HEAD/custom_transformers_packages/gsm8k_strategyqa/transformers/dynamic_module_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4af5faa701d0f2c9"}},{"code_sha256_prefix":"dd8f7751ef859581","entry":"infer_metric_tags_from_eval_results","repo":"wyzjack/CNTP","repo_kind":"official","path":"custom_transformers_packages/gsm8k_strategyqa/transformers/modelcard.py","file_url":"https://github.com/wyzjack/CNTP/blob/HEAD/custom_transformers_packages/gsm8k_strategyqa/transformers/modelcard.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dd8f7751ef859581"}},{"code_sha256_prefix":"e6632a5722f0840d","entry":"parse_keras_history","repo":"wyzjack/CNTP","repo_kind":"official","path":"custom_transformers_packages/gsm8k_strategyqa/transformers/modelcard.py","file_url":"https://github.com/wyzjack/CNTP/blob/HEAD/custom_transformers_packages/gsm8k_strategyqa/transformers/modelcard.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e6632a5722f0840d"}},{"code_sha256_prefix":"a66d2b78670c85e1","entry":"quick_gelu","repo":"wyzjack/CNTP","repo_kind":"official","path":"custom_transformers_packages/gsm8k_strategyqa/transformers/modeling_flax_utils.py","file_url":"https://github.com/wyzjack/CNTP/blob/HEAD/custom_transformers_packages/gsm8k_strategyqa/transformers/modeling_flax_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a66d2b78670c85e1"}},{"code_sha256_prefix":"30d50ff2ced6b7a3","entry":"rename_key_and_reshape_tensor","repo":"wyzjack/CNTP","repo_kind":"official","path":"custom_transformers_packages/gsm8k_strategyqa/transformers/modeling_flax_pytorch_utils.py","file_url":"https://github.com/wyzjack/CNTP/blob/HEAD/custom_transformers_packages/gsm8k_strategyqa/transformers/modeling_flax_pytorch_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30d50ff2ced6b7a3"}},{"code_sha256_prefix":"b1260d802f5e3997","entry":"convert_pytorch_state_dict_to_flax","repo":"wyzjack/CNTP","repo_kind":"official","path":"custom_transformers_packages/gsm8k_strategyqa/transformers/modeling_flax_pytorch_utils.py","file_url":"https://github.com/wyzjack/CNTP/blob/HEAD/custom_transformers_packages/gsm8k_strategyqa/transformers/modeling_flax_pytorch_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1260d802f5e3997"}},{"code_sha256_prefix":"5e01aab2a01fb6db","entry":"dtype_byte_size","repo":"wyzjack/CNTP","repo_kind":"official","path":"custom_transformers_packages/gsm8k_strategyqa/transformers/modeling_flax_utils.py","file_url":"https://github.com/wyzjack/CNTP/blob/HEAD/custom_transformers_packages/gsm8k_strategyqa/transformers/modeling_flax_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5e01aab2a01fb6db"}},{"code_sha256_prefix":"80d41fe6d81de3dd","entry":"flax_shard_checkpoint","repo":"wyzjack/CNTP","repo_kind":"official","path":"custom_transformers_packages/gsm8k_strategyqa/transformers/modeling_flax_utils.py","file_url":"https://github.com/wyzjack/CNTP/blob/HEAD/custom_transformers_packages/gsm8k_strategyqa/transformers/modeling_flax_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"80d41fe6d81de3dd"}},{"code_sha256_prefix":"ec7b1528a936234d","entry":"is_hf_dataset","repo":"wyzjack/CNTP","repo_kind":"official","path":"custom_transformers_packages/gsm8k_strategyqa/transformers/modelcard.py","file_url":"https://github.com/wyzjack/CNTP/blob/HEAD/custom_transformers_packages/gsm8k_strategyqa/transformers/modelcard.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec7b1528a936234d"}},{"code_sha256_prefix":"a77b72cc8d5c90f0","entry":"prepare_fa2_from_position_ids","repo":"wyzjack/CNTP","repo_kind":"official","path":"custom_transformers_packages/gsm8k_strategyqa/transformers/modeling_flash_attention_utils.py","file_url":"https://github.com/wyzjack/CNTP/blob/HEAD/custom_transformers_packages/gsm8k_strategyqa/transformers/modeling_flash_attention_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a77b72cc8d5c90f0"}},{"code_sha256_prefix":"befefbd3afa0731c","entry":"read_field","repo":"wyzjack/CNTP","repo_kind":"official","path":"custom_transformers_packages/gsm8k_strategyqa/transformers/modeling_gguf_pytorch_utils.py","file_url":"https://github.com/wyzjack/CNTP/blob/HEAD/custom_transformers_packages/gsm8k_strategyqa/transformers/modeling_gguf_pytorch_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"befefbd3afa0731c"}},{"code_sha256_prefix":"bb3c4ecbbdf8a9c1","entry":"reverse_permute_weights","repo":"wyzjack/CNTP","repo_kind":"official","path":"custom_transformers_packages/gsm8k_strategyqa/transformers/modeling_gguf_pytorch_utils.py","file_url":"https://github.com/wyzjack/CNTP/blob/HEAD/custom_transformers_packages/gsm8k_strategyqa/transformers/modeling_gguf_pytorch_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bb3c4ecbbdf8a9c1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}