Papers › A Parallel Data Compression Framework for Large Scale 3D Scientific Data
A Parallel Data Compression Framework for Large Scale 3D Scientific Data
Panagiotis Hadjidoukas, Fabian Wermelinger
The archive published only this paper's code-link row. Authors, date and abstract are from arXiv's metadata (CC0), read from the Kaggle arXiv metadata snapshot of 2026-09-12 where its title matched the archive's; the title is the archive's.
Large scale simulations of complex systems ranging from climate and astrophysics to crowd dynamics, produce routinely petabytes of data and are projected to reach the zettabytes level in the coming decade. These simulations enable unprecedented insights but at the same their effectiveness is hindered by the enormous data sizes associated with the computational elements and respective output quantities of interest that impose severe constraints on storage and I/O time. In this work, we address these challenges through a novel software framework for scientific data compression. The software (CubismZ) incorporates efficient wavelet based techniques and the state-of-the-art ZFP, SZ and FPZIP floating point compressors. The framework relies on a block-structured data layout, benefits from OpenMP and MPI and targets supercomputers based on multicores. CubismZ can be used as a tool for ex situ (offline) compression of scientific datasets and supports conventional Computational Fluid Dynamics (CFD) file formats. Moreover, it provides a testbed of comparison, in terms of compression factor and peak signal-to-noise ratio, for a number of available data compression methods. The software yields in situ compression ratios of 100x or higher for fluid dynamics data produced by petascale simulations of cloud cavitation collapse using 𝒪(10¹¹) grid cells, with negligible impact on the total simulation time.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Results from the paper archive 2025-07-28
No leaderboard rows for this paper in the archive.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections