{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2608-03743","title":"Can LLMs Test Terminal User Interfaces?","arxiv_id":"2608.03743","date":"2026-08-04","proceeding":null,"authors":["Chao Peng","Ruida Hu","Ajitha Rajan","Tegawendé F Bissyandé","Jacques Klein","Cuiyun Gao"],"abstract":"Terminal User Interfaces (TUIs) combine the stateful, screen-oriented behaviour of GUIs with terminal deployment and are now common in developer tools. Yet they lack a dedicated testing methodology. We survey 197 real-world TUI applications: only 12% of test code exercises the interface, and 45% of those tests never send input, checking a static frame instead. We turn these applications into a headless benchmark spanning ratatui/Rust, bubbletea/Go, textual/Python, and ink/TypeScript, packaging each as an instrumented Docker image. We record line and widget coverage where reliable, rendered terminal states, and crashes. Under equal wall-clock budgets, we compare four frontier LLMs with random exploration. No model dominates. Random is a strong time-budgeted baseline, but its crash advantage comes from higher throughput: per interaction, LLM guidance is more efficient and uniquely reaches input-gated faults. Automatically deriving launch inputs yields the largest practical gain, enabling applications that otherwise never start. Line coverage poorly predicts crash discovery, weakening it as a proxy for test effectiveness. Automated TUI testing is feasible but far from solved, and honest baselines matter more than model choice. We release the coverage tool tuicov at https://github.com/tui-testing/tuicov and the testing framework tuibot at https://github.com/tui-testing/tuibot.","url_abs":"https://arxiv.org/abs/2608.03743","url_pdf":"https://arxiv.org/pdf/2608.03743","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2608.03743","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2608.03743"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/tui-testing/tuicov","reach":null},{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/tui-testing/tuibot","reach":null}],"summary":{"ran":9},"by_repo_kind":{"found_in_text":{"samples":9,"ran":9,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1e6beec9f1848a60","entry":"build_recap","repo":"tui-testing/tuibot","repo_kind":"found_in_text","path":"src/tuibot/agents/explorer.py","file_url":"https://github.com/tui-testing/tuibot/blob/HEAD/src/tuibot/agents/explorer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1e6beec9f1848a60"}},{"code_sha256_prefix":"1d2df3f0f117473c","entry":"inputs_dir","repo":"tui-testing/tuibot","repo_kind":"found_in_text","path":"src/tuibot/entrypoint.py","file_url":"https://github.com/tui-testing/tuibot/blob/HEAD/src/tuibot/entrypoint.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1d2df3f0f117473c"}},{"code_sha256_prefix":"c865fb88e9a79cde","entry":"normalized_file_name","repo":"tui-testing/tuicov","repo_kind":"found_in_text","path":"src/tuicov/models.py","file_url":"https://github.com/tui-testing/tuicov/blob/HEAD/src/tuicov/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c865fb88e9a79cde"}},{"code_sha256_prefix":"d13b1f314eae32ac","entry":"parse_duration","repo":"tui-testing/tuicov","repo_kind":"found_in_text","path":"src/tuicov/util.py","file_url":"https://github.com/tui-testing/tuicov/blob/HEAD/src/tuicov/util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d13b1f314eae32ac"}},{"code_sha256_prefix":"b7e42a0d48fbc6b8","entry":"parse_listen","repo":"tui-testing/tuicov","repo_kind":"found_in_text","path":"src/tuicov/service.py","file_url":"https://github.com/tui-testing/tuicov/blob/HEAD/src/tuicov/service.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b7e42a0d48fbc6b8"}},{"code_sha256_prefix":"59fbe599a8d2d6de","entry":"read_text","repo":"tui-testing/tuicov","repo_kind":"found_in_text","path":"src/tuicov/util.py","file_url":"https://github.com/tui-testing/tuicov/blob/HEAD/src/tuicov/util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"59fbe599a8d2d6de"}},{"code_sha256_prefix":"3651aace47411392","entry":"relative_capture","repo":"tui-testing/tuibot","repo_kind":"found_in_text","path":"src/tuibot/agents/explorer.py","file_url":"https://github.com/tui-testing/tuibot/blob/HEAD/src/tuibot/agents/explorer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3651aace47411392"}},{"code_sha256_prefix":"f12b4c26b065495c","entry":"run_command","repo":"tui-testing/tuibot","repo_kind":"found_in_text","path":"src/tuibot/process.py","file_url":"https://github.com/tui-testing/tuibot/blob/HEAD/src/tuibot/process.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f12b4c26b065495c"}},{"code_sha256_prefix":"1a5bcb6f5c36002e","entry":"stable_widget_id","repo":"tui-testing/tuicov","repo_kind":"found_in_text","path":"src/tuicov/util.py","file_url":"https://github.com/tui-testing/tuicov/blob/HEAD/src/tuicov/util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1a5bcb6f5c36002e"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}