{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2601-22002","title":"Understanding Rate-Distortion Performance in Distributed Transformer Inference","arxiv_id":"2601.22002","date":"2026-01-29","proceeding":null,"authors":["Anderson de Andrade","Alon Harell","Ivan V. Bajić"],"abstract":"Transformers achieve superior performance on many tasks, but impose heavy compute and memory requirements during inference. This inference can be made more efficient by partitioning the process across multiple devices, which, in turn, requires compressing its intermediate representations. We study compressibility of transformer's intermediate representations via learned compression through the lens of rate-distortion and the theory of usable information. Our study reveals that, unlike convolutional models, deeper representations in transformers become more difficult to compress. The reason for this behavior is twofold: first, the complexity of representation increases as we move deeper into the transformer, necessitating a higher rate; and second, the higher complexity of representations worsens the generaliztion bound for learned entropy estimates, which further compromises compression performance. Through a combination of experiments and theory, we characterize and analyze the compressibility of transformer representations, derive bounds on the achievable rate of learned codecs applied to these representations, and offer a unified lens for understanding rate-distortion performance in representation coding.","url_abs":"https://arxiv.org/abs/2601.22002","url_pdf":"https://arxiv.org/pdf/2601.22002","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2601.22002","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2601.22002"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/adeandrade/research","reach":null}],"summary":{"unverified":4},"by_repo_kind":{"found_in_text":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"bb51e0169df0d66d","entry":"elic_lst_downsample","repo":"adeandrade/research","repo_kind":"found_in_text","path":"projects/common-information/common_information/model_lst.py","file_url":"https://github.com/adeandrade/research/blob/HEAD/projects/common-information/common_information/model_lst.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bb51e0169df0d66d"}},{"code_sha256_prefix":"ebf6ed43a5aac579","entry":"elic_lst_downsample","repo":"adeandrade/research","repo_kind":"found_in_text","path":"projects/compatible-representations/compatible_representations/model_lst.py","file_url":"https://github.com/adeandrade/research/blob/HEAD/projects/compatible-representations/compatible_representations/model_lst.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ebf6ed43a5aac579"}},{"code_sha256_prefix":"54bb98012dc43029","entry":"elic_lst_upsample","repo":"adeandrade/research","repo_kind":"found_in_text","path":"projects/common-information/common_information/model_lst.py","file_url":"https://github.com/adeandrade/research/blob/HEAD/projects/common-information/common_information/model_lst.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"54bb98012dc43029"}},{"code_sha256_prefix":"db92bcc7d307141c","entry":"shuffle_lst_downsample","repo":"adeandrade/research","repo_kind":"found_in_text","path":"projects/common-information/common_information/model_lst.py","file_url":"https://github.com/adeandrade/research/blob/HEAD/projects/common-information/common_information/model_lst.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"db92bcc7d307141c"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}