{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/efficient-training-of-deep-equilibrium-models","title":"Efficient Training of Deep Equilibrium Models","arxiv_id":"2304.11663","date":"2023-04-23","proceeding":null,"authors":["Bac Nguyen","Lukas Mauch"],"abstract":"Deep equilibrium models (DEQs) have proven to be very powerful for learning data representations. The idea is to replace traditional (explicit) feedforward neural networks with an implicit fixed-point equation, which allows to decouple the forward and backward passes. In particular, training DEQ layers becomes very memory-efficient via the implicit function theorem. However, backpropagation through DEQ layers still requires solving an expensive Jacobian-based equation. In this paper, we introduce a simple but effective strategy to avoid this computational burden. Our method relies on the Jacobian approximation of Broyden's method after the forward pass to compute the gradients during the backward pass. Experiments show that simply re-using this approximation can significantly speed up the training while not causing any performance degradation.","url_abs":"https://arxiv.org/abs/2304.11663v1","url_pdf":"https://arxiv.org/pdf/2304.11663v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"efficient-training-of-deep-equilibrium-models","repo_url":"https://github.com/locuslab/deq","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"deq","method_name":"DEQ"},{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2304.11663","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2304.11663"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/locuslab/deq","reach":null}],"summary":{"ran_fixture":1,"ran_draft_wrong":3,"ran_honours":1,"unverified":3},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"823ec2857d23bdea","entry":"broyden","repo":"locuslab/deq","repo_kind":"official","path":"lib/solvers.py","file_url":"https://github.com/locuslab/deq/blob/HEAD/lib/solvers.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"823ec2857d23bdea"}},{"code_sha256_prefix":"cf5710371bcfb0ab","entry":"line_search","repo":"locuslab/deq","repo_kind":"official","path":"lib/solvers.py","file_url":"https://github.com/locuslab/deq/blob/HEAD/lib/solvers.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cf5710371bcfb0ab"}},{"code_sha256_prefix":"3c120cc50bc8c55f","entry":"power_method","repo":"locuslab/deq","repo_kind":"official","path":"lib/jacobian.py","file_url":"https://github.com/locuslab/deq/blob/HEAD/lib/jacobian.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c120cc50bc8c55f"}},{"code_sha256_prefix":"968a8f446ad3c5b1","entry":"rmatvec","repo":"locuslab/deq","repo_kind":"official","path":"lib/solvers.py","file_url":"https://github.com/locuslab/deq/blob/HEAD/lib/solvers.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"968a8f446ad3c5b1"}},{"code_sha256_prefix":"b2b5022f654ee4d3","entry":"scalar_search_armijo","repo":"locuslab/deq","repo_kind":"official","path":"lib/solvers.py","file_url":"https://github.com/locuslab/deq/blob/HEAD/lib/solvers.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b2b5022f654ee4d3"}},{"code_sha256_prefix":"d41246cdbd511eff","entry":"jac_loss_estimate","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"d41246cdbd511eff"}},{"code_sha256_prefix":"3d07af47a6ba65bd","entry":"jac_loss_estimate","repo":"locuslab/deq","repo_kind":"official","path":"lib/jacobian.py","file_url":"https://github.com/locuslab/deq/blob/HEAD/lib/jacobian.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3d07af47a6ba65bd"}},{"code_sha256_prefix":"8a2cfe51ec1c04e7","entry":"power_method","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"8a2cfe51ec1c04e7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}