{"url":"/method/deepcubeai","slug":"deepcubeai","name":"DeepCubeAI","full_name":"DeepCubeA + Imagination","full_name_withheld":false,"description_markdown":"# About DeepCubeAI\r\n\r\nDeepCubeAI is an algorithm that learns a discrete world model and employs Deep Reinforcement Learning methods to learn a heuristic function that generalizes over start and goal states. We then integrate the learned model and the learned heuristic function with heuristic search, such as Q* search, to solve sequential decision making problems [[paper]](https://rlj.cs.umass.edu/2024/papers/Paper225.html) [[Code]](https://github.com/misaghsoltani/DeepCubeAI) [[PyPI]](https://pypi.org/project/deepcubeai/) [[Slides]](https://cse.sc.edu/~foresta/assets/files/Slides--LearningDiscreteWorldModelsforHeuristicSearch.pdf) [[Poster]](https://cse.sc.edu/~foresta/assets/files/Poster--LearningDiscreteWorldModelsforHeuristicSearch.pdf) \r\n\r\n‌\r\n## Key Contributions\r\n\r\nDeepCubeAI is comprised of three key components:\r\n\r\n1. **Discrete World Model**\r\n   - Learns a world model that represents states in a discrete latent space.\r\n   - This approach tackles two challenges: model degradation and state re-identification.\r\n     - Prediction errors less than 0.5 are corrected by rounding.\r\n     - Re-identifies states by comparing two binary vectors.\r\n\r\n2. **Generalizable Heuristic Function**\r\n   - Utilizes Deep Q-Network (DQN) and hindsight experience replay (HER) to learn a heuristic function that generalizes over start and goal states.\r\n\r\n3. **Optimized Search**\r\n   - Integrates the learned model and the learned heuristic function with heuristic search to solve problems. It uses [Q* search](https://prl-theworkshop.github.io/prl2024-icaps/papers/9.pdf), a variant of A* search optimized for DQNs, which enables faster and more memory-efficient planning.\r\n‌\r\n## Main Results\r\n* Accurate reconstruction of ground truth images after thousands of timesteps.\r\n* Achieved 100% success on Rubik's Cube (canonical goal), Sokoban, IceSlider, and DigitJump.\r\n* 99.9% success on Rubik's Cube with reversed start/goal states.\r\n* Demonstrated significant improvement in solving complex planning problems and generalizing to unseen goals.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Learning Discrete World Models for Heuristic Search","paper":"/paper/learning-discrete-world-models-for-heuristic","first_author":"Forest Agostinelli","n_authors":2,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/learning-discrete-world-models-for-heuristic"},"source":{"url":"https://rlj.cs.umass.edu/2024/papers/Paper225.html","title":"Learning Discrete World Models for Heuristic Search","url_on_a_paper_host":false},"code_snippet_url":"https://github.com/misaghsoltani/DeepCubeAI","code_snippet_url_on_a_code_host":true,"categories":[{"area":"Reinforcement Learning","area_id":"reinforcement-learning","collection":"Q-Learning Networks","url":"/methods/category/q-learning-networks","pwc_aliases":["q-learning"]},{"area":"Reinforcement Learning","area_id":"reinforcement-learning","collection":"Offline Reinforcement Learning Methods","url":"/methods/category/offline-reinforcement-learning-methods","pwc_aliases":[]}],"n_papers_tagged":1,"archive_num_papers":1,"papers_newest_first":[{"paper":"/paper/learning-discrete-world-models-for-heuristic","title":"Learning Discrete World Models for Heuristic Search","date":"2024-09-14","arxiv_id":null,"n_code_links":1,"syntology":null}],"papers_shown":1,"tasks":[{"task":"/task/deep-reinforcement-learning","name":"Deep Reinforcement Learning","papers":1},{"task":"/task/heuristic-search","name":"Heuristic Search","papers":1},{"task":"/task/model-based-reinforcement-learning","name":"Model-based Reinforcement Learning","papers":1},{"task":"/task/reinforcement-learning","name":"Reinforcement Learning","papers":1},{"task":"/task/rubik-s-cube","name":"Rubik's Cube","papers":1},{"task":"/task/sequential-decision-making","name":"Sequential Decision Making","papers":1},{"task":"/task/sokoban","name":"Sokoban","papers":1},{"task":"/task/trajectory-planning","name":"Trajectory Planning","papers":1}],"tasks_shown":8,"n_tasks":8,"usage_by_year":[{"year":"2024","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/deepcubeai"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}