{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/let-s-think-dot-by-dot-hidden-computation-in","title":"Let's Think Dot by Dot: Hidden Computation in Transformer Language Models","arxiv_id":"2404.15758","date":"2024-04-24","proceeding":null,"authors":["Jacob Pfau","William Merrill","Samuel R. Bowman"],"abstract":"Chain-of-thought responses from language models improve performance across most benchmarks. However, it remains unclear to what extent these performance gains can be attributed to human-like task decomposition or simply the greater computation that additional tokens allow. We show that transformers can use meaningless filler tokens (e.g., '......') in place of a chain of thought to solve two hard algorithmic tasks they could not solve when responding without intermediate tokens. However, we find empirically that learning to use filler tokens is difficult and requires specific, dense supervision to converge. We also provide a theoretical characterization of the class of problems where filler tokens are useful in terms of the quantifier depth of a first-order formula. For problems satisfying this characterization, chain-of-thought tokens need not provide information about the intermediate computational steps involved in multi-token computations. In summary, our results show that additional tokens can provide computational benefits independent of token choice. The fact that intermediate tokens can act as filler tokens raises concerns about large language models engaging in unauditable, hidden computations that are increasingly detached from the observed chain-of-thought tokens.","url_abs":"https://arxiv.org/abs/2404.15758v1","url_pdf":"https://arxiv.org/pdf/2404.15758v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"let-s-think-dot-by-dot-hidden-computation-in","repo_url":"https://github.com/jacobpfau/fillertokens","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"let-s-think-dot-by-dot-hidden-computation-in","repo_url":"https://github.com/rokosbasilisk/filler_tokens","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2404.15758","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.15758"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rokosbasilisk/filler_tokens","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jacobpfau/fillertokens","reach":{"status":"ok"}}],"summary":{"unverified":14},"by_repo_kind":{"official":{"samples":14,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"5ec0baefcc7f9cec","entry":"cat_row","repo":"jacobpfau/fillertokens","repo_kind":"official","path":"src/match3.py","file_url":"https://github.com/jacobpfau/fillertokens/blob/HEAD/src/match3.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5ec0baefcc7f9cec"}},{"code_sha256_prefix":"b34a52e418beb7bc","entry":"dot_tf_batch_to_type","repo":"jacobpfau/fillertokens","repo_kind":"official","path":"src/train_match3.py","file_url":"https://github.com/jacobpfau/fillertokens/blob/HEAD/src/train_match3.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b34a52e418beb7bc"}},{"code_sha256_prefix":"6dd6c76d6f6ed809","entry":"find_consecutive_dots_mask","repo":"jacobpfau/fillertokens","repo_kind":"official","path":"src/train_match3.py","file_url":"https://github.com/jacobpfau/fillertokens/blob/HEAD/src/train_match3.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6dd6c76d6f6ed809"}},{"code_sha256_prefix":"1a03d75c0e35e8e7","entry":"get_optimizer","repo":"jacobpfau/fillertokens","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/jacobpfau/fillertokens/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1a03d75c0e35e8e7"}},{"code_sha256_prefix":"16b09a12d091a303","entry":"identity_transform","repo":"jacobpfau/fillertokens","repo_kind":"official","path":"src/match2.py","file_url":"https://github.com/jacobpfau/fillertokens/blob/HEAD/src/match2.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"16b09a12d091a303"}},{"code_sha256_prefix":"e8da9e3e5a9a10e5","entry":"lookup_transform","repo":"jacobpfau/fillertokens","repo_kind":"official","path":"src/match2.py","file_url":"https://github.com/jacobpfau/fillertokens/blob/HEAD/src/match2.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e8da9e3e5a9a10e5"}},{"code_sha256_prefix":"84ab83ab5190ada5","entry":"match2_vector_eval_loop","repo":"jacobpfau/fillertokens","repo_kind":"official","path":"src/train_match2.py","file_url":"https://github.com/jacobpfau/fillertokens/blob/HEAD/src/train_match2.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"84ab83ab5190ada5"}},{"code_sha256_prefix":"441af884cc5dd3fe","entry":"match2_vector_eval_step","repo":"jacobpfau/fillertokens","repo_kind":"official","path":"src/train_match2.py","file_url":"https://github.com/jacobpfau/fillertokens/blob/HEAD/src/train_match2.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"441af884cc5dd3fe"}},{"code_sha256_prefix":"0f608d73aca2f8a6","entry":"match3_vector_eval_loop","repo":"jacobpfau/fillertokens","repo_kind":"official","path":"src/train_match3.py","file_url":"https://github.com/jacobpfau/fillertokens/blob/HEAD/src/train_match3.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0f608d73aca2f8a6"}},{"code_sha256_prefix":"2c3e6f2e8c98b4c7","entry":"no_filler_parallel","repo":"jacobpfau/fillertokens","repo_kind":"official","path":"src/match3.py","file_url":"https://github.com/jacobpfau/fillertokens/blob/HEAD/src/match3.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2c3e6f2e8c98b4c7"}},{"code_sha256_prefix":"452ad0e734c595f4","entry":"no_filler_serial","repo":"jacobpfau/fillertokens","repo_kind":"official","path":"src/match3.py","file_url":"https://github.com/jacobpfau/fillertokens/blob/HEAD/src/match3.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"452ad0e734c595f4"}},{"code_sha256_prefix":"d3cbc2bc4d10c134","entry":"random_lookup_params","repo":"jacobpfau/fillertokens","repo_kind":"official","path":"src/match2.py","file_url":"https://github.com/jacobpfau/fillertokens/blob/HEAD/src/match2.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d3cbc2bc4d10c134"}},{"code_sha256_prefix":"b7c1c5ed38cedeb0","entry":"reshape_crossent","repo":"jacobpfau/fillertokens","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/jacobpfau/fillertokens/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b7c1c5ed38cedeb0"}},{"code_sha256_prefix":"523caa47bb2d16e6","entry":"train_steps","repo":"jacobpfau/fillertokens","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/jacobpfau/fillertokens/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"523caa47bb2d16e6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}