{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/from-loops-to-oops-fallback-behaviors-of","title":"From Loops to Oops: Fallback Behaviors of Language Models Under Uncertainty","arxiv_id":"2407.06071","date":"2024-07-08","proceeding":null,"authors":["Maor Ivgi","Ori Yoran","Jonathan Berant","Mor Geva"],"abstract":"Large language models (LLMs) often exhibit undesirable behaviors, such as hallucinations and sequence repetitions. We propose to view these behaviors as fallbacks that models exhibit under uncertainty, and investigate the connection between them. We categorize fallback behaviors -- sequence repetitions, degenerate text, and hallucinations -- and extensively analyze them in models from the same family that differ by the amount of pretraining tokens, parameter count, or the inclusion of instruction-following training. Our experiments reveal a clear and consistent ordering of fallback behaviors, across all these axes: the more advanced an LLM is (i.e., trained on more tokens, has more parameters, or instruction-tuned), its fallback behavior shifts from sequence repetitions, to degenerate text, and then to hallucinations. Moreover, the same ordering is observed throughout a single generation, even for the best-performing models; as uncertainty increases, models shift from generating hallucinations to producing degenerate text and then sequence repetitions. Lastly, we demonstrate that while common decoding techniques, such as random sampling, might alleviate some unwanted behaviors like sequence repetitions, they increase harder-to-detect hallucinations.","url_abs":"https://arxiv.org/abs/2407.06071v1","url_pdf":"https://arxiv.org/pdf/2407.06071v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"from-loops-to-oops-fallback-behaviors-of","repo_url":"https://github.com/mivg/fallbacks","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"instruction-following","task_name":"Instruction Following"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2407.06071","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.06071"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Mivg/fallbacks","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mivg/fallbacks","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5,"ran_draft_wrong":2,"ran_fixture":1,"unverified":4},"by_repo_kind":{"official":{"samples":12,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1321602fe1a91715","entry":"clean_sample","repo":"mivg/fallbacks","repo_kind":"official","path":"utils/degeneration_detector.py","file_url":"https://github.com/mivg/fallbacks/blob/HEAD/utils/degeneration_detector.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1321602fe1a91715"}},{"code_sha256_prefix":"4976dd040a789ec3","entry":"ends_in_punctuation","repo":"mivg/fallbacks","repo_kind":"official","path":"utils/degeneration_detector.py","file_url":"https://github.com/mivg/fallbacks/blob/HEAD/utils/degeneration_detector.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4976dd040a789ec3"}},{"code_sha256_prefix":"808e0a22c84796cd","entry":"extract_answers","repo":"Mivg/fallbacks","repo_kind":"official","path":"run_hallucinations_check.py","file_url":"https://github.com/Mivg/fallbacks/blob/HEAD/run_hallucinations_check.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"808e0a22c84796cd"}},{"code_sha256_prefix":"fe2f94dc0eff579c","entry":"get_generate_for_pipeline","repo":"mivg/fallbacks","repo_kind":"official","path":"run_hallucinations_check.py","file_url":"https://github.com/mivg/fallbacks/blob/HEAD/run_hallucinations_check.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fe2f94dc0eff579c"}},{"code_sha256_prefix":"fc69a31389986f33","entry":"get_params_and_model","repo":"Mivg/fallbacks","repo_kind":"official","path":"scripts/create_paper_plots.py","file_url":"https://github.com/Mivg/fallbacks/blob/HEAD/scripts/create_paper_plots.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fc69a31389986f33"}},{"code_sha256_prefix":"03f72536317546c1","entry":"load_model_and_tokenizer","repo":"mivg/fallbacks","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/mivg/fallbacks/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"03f72536317546c1"}},{"code_sha256_prefix":"2d5f654e9c61426f","entry":"read_jsonl_to_list","repo":"mivg/fallbacks","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/mivg/fallbacks/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2d5f654e9c61426f"}},{"code_sha256_prefix":"472ca68ee68861b4","entry":"sort_by","repo":"Mivg/fallbacks","repo_kind":"official","path":"scripts/create_paper_plots.py","file_url":"https://github.com/Mivg/fallbacks/blob/HEAD/scripts/create_paper_plots.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"472ca68ee68861b4"}},{"code_sha256_prefix":"7136968f74981fc8","entry":"generate_predictions","repo":"mivg/fallbacks","repo_kind":"official","path":"run_hallucinations_check.py","file_url":"https://github.com/mivg/fallbacks/blob/HEAD/run_hallucinations_check.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7136968f74981fc8"}},{"code_sha256_prefix":"c9db7bee70245214","entry":"get_tokenizer_encode_decode","repo":"mivg/fallbacks","repo_kind":"official","path":"utils/degeneration_detector.py","file_url":"https://github.com/mivg/fallbacks/blob/HEAD/utils/degeneration_detector.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c9db7bee70245214"}},{"code_sha256_prefix":"f7cc1f5104a407c1","entry":"is_valid_fact","repo":"mivg/fallbacks","repo_kind":"official","path":"open_ended_evaluator.py","file_url":"https://github.com/mivg/fallbacks/blob/HEAD/open_ended_evaluator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f7cc1f5104a407c1"}},{"code_sha256_prefix":"2520b075c722cf7f","entry":"prepare_results_dataframe","repo":"Mivg/fallbacks","repo_kind":"official","path":"scripts/create_paper_plots.py","file_url":"https://github.com/Mivg/fallbacks/blob/HEAD/scripts/create_paper_plots.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2520b075c722cf7f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}