{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/parallel-training-of-gru-networks-with-a-1","title":"Parallel Training of GRU Networks with a Multi-Grid Solver for Long Sequences","arxiv_id":"2203.04738","date":"2022-03-07","proceeding":null,"authors":["Gordon Euhyun Moon","Eric C. Cyr"],"abstract":"Parallelizing Gated Recurrent Unit (GRU) networks is a challenging task, as the training procedure of GRU is inherently sequential. Prior efforts to parallelize GRU have largely focused on conventional parallelization strategies such as data-parallel and model-parallel training algorithms. However, when the given sequences are very long, existing approaches are still inevitably performance limited in terms of training time. In this paper, we present a novel parallel training scheme (called parallel-in-time) for GRU based on a multigrid reduction in time (MGRIT) solver. MGRIT partitions a sequence into multiple shorter sub-sequences and trains the sub-sequences on different processors in parallel. The key to achieving speedup is a hierarchical correction of the hidden state to accelerate end-to-end communication in both the forward and backward propagation phases of gradient descent. Experimental results on the HMDB51 dataset, where each video is an image sequence, demonstrate that the new parallel training scheme achieves up to 6.5$\\times$ speedup over a serial approach. As efficiency of our new parallelization strategy is associated with the sequence length, our parallel GRU algorithm achieves significant performance improvement as the sequence length increases.","url_abs":"https://arxiv.org/abs/2203.04738v1","url_pdf":"https://arxiv.org/pdf/2203.04738v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"parallel-training-of-gru-networks-with-a-1","repo_url":"https://github.com/Multilevel-NN/torchbraid","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[],"methods":[{"method_slug":"gru","method_name":"GRU"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2203.04738","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2203.04738"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Multilevel-NN/torchbraid","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran_honours":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"84273d11996ee085","entry":"test","repo":"multilevel-nn/torchbraid","repo_kind":"official","path":"examples/UCI_HAR_Recurrent/UCI_HAR_GRU_script.py","file_url":"https://github.com/multilevel-nn/torchbraid/blob/HEAD/examples/UCI_HAR_Recurrent/UCI_HAR_GRU_script.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"84273d11996ee085"}},{"code_sha256_prefix":"7ba32c48be99b4d3","entry":"train","repo":"multilevel-nn/torchbraid","repo_kind":"official","path":"examples/UCI_HAR_Recurrent/UCI_HAR_GRU_script.py","file_url":"https://github.com/multilevel-nn/torchbraid/blob/HEAD/examples/UCI_HAR_Recurrent/UCI_HAR_GRU_script.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7ba32c48be99b4d3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}