{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/assisted-perception-optimizing-observations","title":"Assisted Perception: Optimizing Observations to Communicate State","arxiv_id":"2008.02840","date":"2020-08-06","proceeding":null,"authors":["Siddharth Reddy","Sergey Levine","Anca D. Dragan"],"abstract":"We aim to help users estimate the state of the world in tasks like robotic teleoperation and navigation with visual impairments, where users may have systematic biases that lead to suboptimal behavior: they might struggle to process observations from multiple sensors simultaneously, receive delayed observations, or overestimate distances to obstacles. While we cannot directly change the user's internal beliefs or their internal state estimation process, our insight is that we can still assist them by modifying the user's observations. Instead of showing the user their true observations, we synthesize new observations that lead to more accurate internal state estimates when processed by the user. We refer to this method as assistive state estimation (ASE): an automated assistant uses the true observations to infer the state of the world, then generates a modified observation for the user to consume (e.g., through an augmented reality interface), and optimizes the modification to induce the user's new beliefs to match the assistant's current beliefs. We evaluate ASE in a user study with 12 participants who each perform four tasks: two tasks with known user biases -- bandwidth-limited image classification and a driving video game with observation delay -- and two with unknown biases that our method has to learn -- guided 2D navigation and a lunar lander teleoperation video game. A different assistance strategy emerges in each domain, such as quickly revealing informative pixels to speed up image classification, using a dynamics model to undo observation delay in driving, identifying nearby landmarks for navigation, and exaggerating a visual indicator of tilt in the lander game. The results show that ASE substantially improves the task performance of users with bandwidth constraints, observation delay, and other unknown biases.","url_abs":"https://arxiv.org/abs/2008.02840v1","url_pdf":"https://arxiv.org/pdf/2008.02840v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"assisted-perception-optimizing-observations","repo_url":"https://github.com/rddy/ASE","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"state-estimation","task_name":"State Estimation"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2008.02840","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2008.02840"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rddy/ASE","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":7},"by_repo_kind":{"listed":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bf550853d4df4f1a","entry":"build_reward","repo":"rddy/ASE","repo_kind":"listed","path":"sensei/gw_viz.py","file_url":"https://github.com/rddy/ASE/blob/HEAD/sensei/gw_viz.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bf550853d4df4f1a"}},{"code_sha256_prefix":"6040cf7fff57d910","entry":"build_transitions","repo":"rddy/ASE","repo_kind":"listed","path":"sensei/gw_viz.py","file_url":"https://github.com/rddy/ASE/blob/HEAD/sensei/gw_viz.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6040cf7fff57d910"}},{"code_sha256_prefix":"de52590f0c1a044d","entry":"evaluate_baseline_guides","repo":"rddy/ASE","repo_kind":"listed","path":"sensei/ase.py","file_url":"https://github.com/rddy/ASE/blob/HEAD/sensei/ase.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"de52590f0c1a044d"}},{"code_sha256_prefix":"10da81242bba7f38","entry":"get_display","repo":"rddy/ASE","repo_kind":"listed","path":"deps/rendering.py","file_url":"https://github.com/rddy/ASE/blob/HEAD/deps/rendering.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"10da81242bba7f38"}},{"code_sha256_prefix":"2c06fa905ea7fdda","entry":"make_circle","repo":"rddy/ASE","repo_kind":"listed","path":"deps/rendering.py","file_url":"https://github.com/rddy/ASE/blob/HEAD/deps/rendering.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2c06fa905ea7fdda"}},{"code_sha256_prefix":"d11e06eecd6b20b0","entry":"make_polygon","repo":"rddy/ASE","repo_kind":"listed","path":"deps/rendering.py","file_url":"https://github.com/rddy/ASE/blob/HEAD/deps/rendering.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d11e06eecd6b20b0"}},{"code_sha256_prefix":"f35cb9713634b966","entry":"shape","repo":"rddy/ASE","repo_kind":"listed","path":"sensei/gw_viz.py","file_url":"https://github.com/rddy/ASE/blob/HEAD/sensei/gw_viz.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f35cb9713634b966"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}