{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/kinetics-rethinking-test-time-scaling-laws","title":"Kinetics: Rethinking Test-Time Scaling Laws","arxiv_id":"2506.05333","date":"2025-06-05","proceeding":null,"authors":["Ranajoy Sadhukhan","Zhuoming Chen","Haizhong Zheng","Yang Zhou","Emma Strubell","Beidi Chen"],"abstract":"We rethink test-time scaling laws from a practical efficiency perspective, revealing that the effectiveness of smaller models is significantly overestimated. Prior work, grounded in compute-optimality, overlooks critical memory access bottlenecks introduced by inference-time strategies (e.g., Best-of-$N$, long CoTs). Our holistic analysis, spanning models from 0.6B to 32B parameters, reveals a new Kinetics Scaling Law that better guides resource allocation by incorporating both computation and memory access costs. Kinetics Scaling Law suggests that test-time compute is more effective when used on models above a threshold than smaller ones. A key reason is that in TTS, attention, rather than parameter count, emerges as the dominant cost factor. Motivated by this, we propose a new scaling paradigm centered on sparse attention, which lowers per-token cost and enables longer generations and more parallel samples within the same resource budget. Empirically, we show that sparse attention models consistently outperform dense counterparts, achieving over 60 points gains in low-cost regimes and over 5 points gains in high-cost regimes for problem-solving accuracy on AIME, encompassing evaluations on state-of-the-art MoEs. These results suggest that sparse attention is essential and increasingly important with more computing invested, for realizing the full potential of test-time scaling where, unlike training, accuracy has yet to saturate as a function of computation, and continues to improve through increased generation. The code is available at https://github.com/Infini-AI-Lab/Kinetics.","url_abs":"https://arxiv.org/abs/2506.05333v2","url_pdf":"https://arxiv.org/pdf/2506.05333v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"kinetics-rethinking-test-time-scaling-laws","repo_url":"https://github.com/infini-ai-lab/kinetics","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2506.05333","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.05333"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/infini-ai-lab/kinetics","reach":null}],"summary":{"ran_fixture":3,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"e717d106264e65c0","entry":"blockwise_topk_sparsity","repo":"infini-ai-lab/kinetics","repo_kind":"official","path":"LiteSys/litesys/attention/sparsity_utils.py","file_url":"https://github.com/infini-ai-lab/kinetics/blob/HEAD/LiteSys/litesys/attention/sparsity_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e717d106264e65c0"}},{"code_sha256_prefix":"dfaeb291c6e3d5a3","entry":"run_benchmark","repo":"infini-ai-lab/kinetics","repo_kind":"official","path":"benchmark/blocktopk.py","file_url":"https://github.com/infini-ai-lab/kinetics/blob/HEAD/benchmark/blocktopk.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dfaeb291c6e3d5a3"}},{"code_sha256_prefix":"c072d99dcac78bf6","entry":"strided_blockwise_topk_sparsity","repo":"infini-ai-lab/kinetics","repo_kind":"official","path":"LiteSys/litesys/attention/sparsity_utils.py","file_url":"https://github.com/infini-ai-lab/kinetics/blob/HEAD/LiteSys/litesys/attention/sparsity_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c072d99dcac78bf6"}},{"code_sha256_prefix":"24f9ea72fe6de122","entry":"topk_sparsity","repo":"infini-ai-lab/kinetics","repo_kind":"official","path":"LiteSys/litesys/attention/sparsity_utils.py","file_url":"https://github.com/infini-ai-lab/kinetics/blob/HEAD/LiteSys/litesys/attention/sparsity_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"24f9ea72fe6de122"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}