{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2509-24282","title":"SimuHome: A Temporal- and Environment-Aware Benchmark for Smart Home LLM Agents","arxiv_id":"2509.24282","date":"2025-09-29","proceeding":"ICLR","authors":["Gyuhyeon Seo","Jungwoo Yang","Junseong Pyo","Nalim Kim","Jonggeun Lee","Yohan Jo"],"abstract":"We introduce $\\textbf{SimuHome}$, a high-fidelity smart home simulator and a benchmark of 600 episodes for LLM-based smart home agents. Existing smart home benchmarks treat the home as a static system, neither simulating how device operations affect environmental variables over time nor supporting workflow scheduling of device commands. SimuHome is grounded in the Matter protocol, the industry standard that defines how real smart home devices communicate and operate. Agents interact with devices through SimuHome's APIs and observe how their actions continuously affect environmental variables such as temperature and humidity. Our benchmark covers state inquiry, implicit user intent inference, explicit device control, and workflow scheduling, each with both feasible and infeasible requests. For workflow scheduling, the simulator accelerates time so that scheduled workflows can be evaluated immediately. An evaluation of 18 agents reveals that workflow scheduling is the hardest category, with failures persisting across alternative agent frameworks and fine-tuning. These findings suggest that SimuHome's time-accelerated simulation could serve as an environment for agents to pre-validate their actions before committing them to the real world.","url_abs":"https://arxiv.org/abs/2509.24282","url_pdf":"https://arxiv.org/pdf/2509.24282","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2509.24282","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2509.24282"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/holi-lab/SimuHome","reach":{"status":"ok"}}],"summary":{"unverified":9},"by_repo_kind":{"found_in_text":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"40fa005f0896c89a","entry":"get_pids","repo":"holi-lab/SimuHome","repo_kind":"found_in_text","path":"src/cli/stop_servers.py","file_url":"https://github.com/holi-lab/SimuHome/blob/HEAD/src/cli/stop_servers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"40fa005f0896c89a"}},{"code_sha256_prefix":"7ca394f112065546","entry":"get_safe_model_name","repo":"holi-lab/SimuHome","repo_kind":"found_in_text","path":"src/cli/episode_evaluator.py","file_url":"https://github.com/holi-lab/SimuHome/blob/HEAD/src/cli/episode_evaluator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7ca394f112065546"}},{"code_sha256_prefix":"1ef5ea97c91c3ad7","entry":"get_safe_model_name","repo":"holi-lab/SimuHome","repo_kind":"found_in_text","path":"src/cli/parallel_model_evaluation.py","file_url":"https://github.com/holi-lab/SimuHome/blob/HEAD/src/cli/parallel_model_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1ef5ea97c91c3ad7"}},{"code_sha256_prefix":"e29551bf4a4ff2e9","entry":"is_local_api_base","repo":"holi-lab/SimuHome","repo_kind":"found_in_text","path":"src/cli/config_resolver.py","file_url":"https://github.com/holi-lab/SimuHome/blob/HEAD/src/cli/config_resolver.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e29551bf4a4ff2e9"}},{"code_sha256_prefix":"79bccc4df1912814","entry":"parse_args","repo":"holi-lab/SimuHome","repo_kind":"found_in_text","path":"src/cli/artifact_audit.py","file_url":"https://github.com/holi-lab/SimuHome/blob/HEAD/src/cli/artifact_audit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"79bccc4df1912814"}},{"code_sha256_prefix":"2a1e2f48a086da2b","entry":"parse_integer_spec","repo":"holi-lab/SimuHome","repo_kind":"found_in_text","path":"src/cli/arg_utils.py","file_url":"https://github.com/holi-lab/SimuHome/blob/HEAD/src/cli/arg_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2a1e2f48a086da2b"}},{"code_sha256_prefix":"d3bcf4b95da31c00","entry":"resolve_api_key_for_base","repo":"holi-lab/SimuHome","repo_kind":"found_in_text","path":"src/cli/config_resolver.py","file_url":"https://github.com/holi-lab/SimuHome/blob/HEAD/src/cli/config_resolver.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d3bcf4b95da31c00"}},{"code_sha256_prefix":"2dc36acef66754a9","entry":"resolve_secret_value","repo":"holi-lab/SimuHome","repo_kind":"found_in_text","path":"src/cli/config_resolver.py","file_url":"https://github.com/holi-lab/SimuHome/blob/HEAD/src/cli/config_resolver.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2dc36acef66754a9"}},{"code_sha256_prefix":"c542feeb96c25810","entry":"resolve_simulator_base_url","repo":"holi-lab/SimuHome","repo_kind":"found_in_text","path":"src/clients/smarthome_client.py","file_url":"https://github.com/holi-lab/SimuHome/blob/HEAD/src/clients/smarthome_client.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c542feeb96c25810"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_api"},"syntology_extracted_results":null}