{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/impacts-of-floating-point-non-associativity","title":"Impacts of floating-point non-associativity on reproducibility for HPC and deep learning applications","arxiv_id":"2408.05148","date":"2024-08-09","proceeding":null,"authors":["Sanjif Shanmugavelu","Mathieu Taillefumier","Christopher Culver","Oscar Hernandez","Mark Coletti","Ada Sedova"],"abstract":"Run to run variability in parallel programs caused by floating-point non-associativity has been known to significantly affect reproducibility in iterative algorithms, due to accumulating errors. Non-reproducibility can critically affect the efficiency and effectiveness of correctness testing for stochastic programs. Recently, the sensitivity of deep learning training and inference pipelines to floating-point non-associativity has been found to sometimes be extreme. It can prevent certification for commercial applications, accurate assessment of robustness and sensitivity, and bug detection. New approaches in scientific computing applications have coupled deep learning models with high-performance computing, leading to an aggravation of debugging and testing challenges. Here we perform an investigation of the statistical properties of floating-point non-associativity within modern parallel programming models, and analyze performance and productivity impacts of replacing atomic operations with deterministic alternatives on GPUs. We examine the recently-added deterministic options in PyTorch within the context of GPU deployment for deep learning, uncovering and quantifying the impacts of input parameters triggering run to run variability and reporting on the reliability and completeness of the documentation. Finally, we evaluate the strategy of exploiting automatic determinism that could be provided by deterministic hardware, using the Groq accelerator for inference portions of the deep learning pipeline. We demonstrate the benefits that a hardware-based strategy can provide within reproducibility and correctness efforts.","url_abs":"https://arxiv.org/abs/2408.05148v3","url_pdf":"https://arxiv.org/pdf/2408.05148v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"impacts-of-floating-point-non-associativity","repo_url":"https://github.com/minnervva/correctness","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"deep-learning","task_name":"Deep Learning"},{"task_slug":null,"task_name":"GPU"},{"task_slug":"sensitivity","task_name":"Sensitivity"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2408.05148","atlas_url":"https://app.syntology.ai/?focus=2408.05148","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.05148"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/minnervva/correctness","reach":{"status":"ok"}}],"summary":{"ran":4,"unverified":2},"by_repo_kind":{"official":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"e1af5393c12652ab","entry":"error","repo":"minnervva/correctness","repo_kind":"official","path":"codes/test-suite/scaffold.py","file_url":"https://github.com/minnervva/correctness/blob/HEAD/codes/test-suite/scaffold.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e1af5393c12652ab"}},{"code_sha256_prefix":"d1eb8634b73137c8","entry":"func_latency","repo":"minnervva/correctness","repo_kind":"official","path":"codes/test-suite/kernels/benchmark.py","file_url":"https://github.com/minnervva/correctness/blob/HEAD/codes/test-suite/kernels/benchmark.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d1eb8634b73137c8"}},{"code_sha256_prefix":"26656f8b5a29be6d","entry":"nn_get_data","repo":"minnervva/correctness","repo_kind":"official","path":"codes/test-suite/kernels/benchmark.py","file_url":"https://github.com/minnervva/correctness/blob/HEAD/codes/test-suite/kernels/benchmark.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"26656f8b5a29be6d"}},{"code_sha256_prefix":"4bca94c491f93398","entry":"nn_latency","repo":"minnervva/correctness","repo_kind":"official","path":"codes/test-suite/kernels/benchmark.py","file_url":"https://github.com/minnervva/correctness/blob/HEAD/codes/test-suite/kernels/benchmark.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4bca94c491f93398"}},{"code_sha256_prefix":"39e71e53f18485c7","entry":"benchmark","repo":"minnervva/correctness","repo_kind":"official","path":"codes/test-suite/scaffold.py","file_url":"https://github.com/minnervva/correctness/blob/HEAD/codes/test-suite/scaffold.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"39e71e53f18485c7"}},{"code_sha256_prefix":"617854884f5a9e34","entry":"benchmark","repo":"minnervva/correctness","repo_kind":"official","path":"codes/test-suite/.ipynb_checkpoints/scaffold-checkpoint.py","file_url":"https://github.com/minnervva/correctness/blob/HEAD/codes/test-suite/.ipynb_checkpoints/scaffold-checkpoint.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"617854884f5a9e34"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}