{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2604-15306","title":"Generalization in LLM Problem Solving: The Case of the Shortest Path","arxiv_id":"2604.15306","date":"2026-04-16","proceeding":"ICLR","authors":["Yao Tong","Jiayuan Ye","Anastasia Borovykh","Reza Shokri"],"abstract":"Whether language models can systematically generalize remains actively debated. Yet empirical performance is jointly shaped by multiple factors such as training data, training paradigms, and inference-time strategies, making failures difficult to interpret. We introduce a controlled synthetic environment based on shortest-path planning, a canonical composable sequential optimization problem. The setup enables clean separation of these factors and supports two orthogonal axes of generalization: spatial transfer to unseen maps and length scaling to longer-horizon problems. We find that models exhibit strong spatial transfer but consistently fail under length scaling due to recursive instability. We further analyze how distinct stages of the learning pipeline influence systematic problem-solving: for example, data coverage sets capability limits; reinforcement learning improves training stability but does not expand those limits; and inference-time scaling enhances performance but cannot rescue length-scaling failures.","url_abs":"https://arxiv.org/abs/2604.15306","url_pdf":"https://arxiv.org/pdf/2604.15306","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2604.15306","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2604.15306"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/privacytrustlab/PathGeneralization","reach":null}],"summary":{"ran_fixture":1,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":4,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"25e5c366a21243bb","entry":"get_neighbors","repo":"privacytrustlab/PathGeneralization","repo_kind":"found_in_text","path":"utils/model.py","file_url":"https://github.com/privacytrustlab/PathGeneralization/blob/HEAD/utils/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"25e5c366a21243bb"}},{"code_sha256_prefix":"2b3628d06b233681","entry":"get_valid_neighbors","repo":"privacytrustlab/PathGeneralization","repo_kind":"found_in_text","path":"utils/model.py","file_url":"https://github.com/privacytrustlab/PathGeneralization/blob/HEAD/utils/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2b3628d06b233681"}},{"code_sha256_prefix":"ef8403978e5a4764","entry":"Model","repo":"privacytrustlab/PathGeneralization","repo_kind":"found_in_text","path":"utils/model.py","file_url":"https://github.com/privacytrustlab/PathGeneralization/blob/HEAD/utils/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ef8403978e5a4764"}},{"code_sha256_prefix":"c947b0afb65543fe","entry":"PathGenModel","repo":"privacytrustlab/PathGeneralization","repo_kind":"found_in_text","path":"utils/model.py","file_url":"https://github.com/privacytrustlab/PathGeneralization/blob/HEAD/utils/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c947b0afb65543fe"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}