{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/iq-learn-inverse-soft-q-learning-for","title":"IQ-Learn: Inverse soft-Q Learning for Imitation","arxiv_id":"2106.12142","date":"2021-06-23","proceeding":"NeurIPS 2021 12","authors":["Divyansh Garg","Shuvam Chakraborty","Chris Cundy","Jiaming Song","Matthieu Geist","Stefano Ermon"],"abstract":"In many sequential decision-making problems (e.g., robotics control, game playing, sequential prediction), human or expert data is available containing useful information about the task. However, imitation learning (IL) from a small amount of expert data can be challenging in high-dimensional environments with complex dynamics. Behavioral cloning is a simple method that is widely used due to its simplicity of implementation and stable convergence but doesn't utilize any information involving the environment's dynamics. Many existing methods that exploit dynamics information are difficult to train in practice due to an adversarial optimization process over reward and policy approximators or biased, high variance gradient estimators. We introduce a method for dynamics-aware IL which avoids adversarial training by learning a single Q-function, implicitly representing both reward and policy. On standard benchmarks, the implicitly learned rewards show a high positive correlation with the ground-truth rewards, illustrating our method can also be used for inverse reinforcement learning (IRL). Our method, Inverse soft-Q learning (IQ-Learn) obtains state-of-the-art results in offline and online imitation learning settings, significantly outperforming existing methods both in the number of required environment interactions and scalability in high-dimensional spaces, often by more than 3x.","url_abs":"https://arxiv.org/abs/2106.12142v4","url_pdf":"https://arxiv.org/pdf/2106.12142v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"iq-learn-inverse-soft-q-learning-for","repo_url":"https://github.com/Div99/IQ-Learn","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"iq-learn-inverse-soft-q-learning-for","repo_url":"https://github.com/MilkSilk/masters_thesis","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"iq-learn-inverse-soft-q-learning-for","repo_url":"https://github.com/edmundmills/basalt-competition","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"iq-learn-inverse-soft-q-learning-for","repo_url":"https://github.com/google-deepmind/csil","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"iq-learn-inverse-soft-q-learning-for","repo_url":"https://github.com/robfiras/ls-iq","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"atari-games","task_name":"Atari Games"},{"task_slug":"continuous-control","task_name":"Continuous Control"},{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"imitation-learning","task_name":"Imitation Learning"},{"task_slug":"mujoco-games","task_name":"MuJoCo Games"},{"task_slug":"q-learning","task_name":"Q-Learning"},{"task_slug":"sequential-decision-making","task_name":"Sequential Decision Making"}],"methods":[{"method_slug":"iq-learn","method_name":"IQ-Learn"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/atari-games-on-atari-2600-beam-rider","task":"Atari Games","dataset":"Atari 2600 Beam Rider","model":"IQ-Learn","rank_in_archive_order":49,"of":49,"metrics":{"Return":"3025"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-qbert","task":"Atari Games","dataset":"Atari 2600 Q*Bert","model":"IQ-Learn","rank_in_archive_order":57,"of":57,"metrics":{"Return":"12940"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-seaquest","task":"Atari Games","dataset":"Atari 2600 Seaquest","model":"IQ-Learn","rank_in_archive_order":57,"of":57,"metrics":{"Return":"2349"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-space-invaders","task":"Atari Games","dataset":"Atari 2600 Space Invaders","model":"IQ-Learn","rank_in_archive_order":55,"of":55,"metrics":{"Return":"507"},"uses_additional_data":false},{"leaderboard":"/sota/mujoco-games-on-ant","task":"MuJoCo Games","dataset":"Ant","model":"IQ-Learn","rank_in_archive_order":1,"of":3,"metrics":{"Average Return":"4362.9"},"uses_additional_data":false},{"leaderboard":"/sota/mujoco-games-on-humanoid-v2","task":"MuJoCo Games","dataset":"Humanoid-v2","model":"IQ-Learn","rank_in_archive_order":1,"of":1,"metrics":{"Return":"5227.1"},"uses_additional_data":false},{"leaderboard":"/sota/mujoco-games-on-walker2d","task":"MuJoCo Games","dataset":"Walker2d","model":"IQ-Learn","rank_in_archive_order":1,"of":2,"metrics":{"Mean":"5134"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2106.12142","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.12142"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Div99/IQ-Learn","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/edmundmills/basalt-competition","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google-deepmind/csil","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/robfiras/ls-iq","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MilkSilk/masters_thesis","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"listed":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"d188609497f5bd0e","entry":"cat_states","repo":"edmundmills/basalt-competition","repo_kind":"listed","path":"algorithms/loss_functions/iqlearn.py","file_url":"https://github.com/edmundmills/basalt-competition/blob/HEAD/algorithms/loss_functions/iqlearn.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d188609497f5bd0e"}},{"code_sha256_prefix":"a14c0c0e85392744","entry":"cat_transitions","repo":"edmundmills/basalt-competition","repo_kind":"listed","path":"algorithms/loss_functions/iqlearn.py","file_url":"https://github.com/edmundmills/basalt-competition/blob/HEAD/algorithms/loss_functions/iqlearn.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a14c0c0e85392744"}},{"code_sha256_prefix":"b9c72ed3c904ebc9","entry":"IQLearnLoss","repo":"edmundmills/basalt-competition","repo_kind":"listed","path":"algorithms/loss_functions/iqlearn.py","file_url":"https://github.com/edmundmills/basalt-competition/blob/HEAD/algorithms/loss_functions/iqlearn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b9c72ed3c904ebc9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}