{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/gazeformer-scalable-effective-and-fast","title":"Gazeformer: Scalable, Effective and Fast Prediction of Goal-Directed Human Attention","arxiv_id":"2303.15274","date":"2023-03-27","proceeding":"CVPR 2023 1","authors":["Sounak Mondal","Zhibo Yang","Seoyoung Ahn","Dimitris Samaras","Gregory Zelinsky","Minh Hoai"],"abstract":"Predicting human gaze is important in Human-Computer Interaction (HCI). However, to practically serve HCI applications, gaze prediction models must be scalable, fast, and accurate in their spatial and temporal gaze predictions. Recent scanpath prediction models focus on goal-directed attention (search). Such models are limited in their application due to a common approach relying on trained target detectors for all possible objects, and the availability of human gaze data for their training (both not scalable). In response, we pose a new task called ZeroGaze, a new variant of zero-shot learning where gaze is predicted for never-before-searched objects, and we develop a novel model, Gazeformer, to solve the ZeroGaze problem. In contrast to existing methods using object detector modules, Gazeformer encodes the target using a natural language model, thus leveraging semantic similarities in scanpath prediction. We use a transformer-based encoder-decoder architecture because transformers are particularly useful for generating contextual representations. Gazeformer surpasses other models by a large margin on the ZeroGaze setting. It also outperforms existing target-detection models on standard gaze prediction for both target-present and target-absent search tasks. In addition to its improved performance, Gazeformer is more than five times faster than the state-of-the-art target-present visual search model.","url_abs":"https://arxiv.org/abs/2303.15274v3","url_pdf":"https://arxiv.org/pdf/2303.15274v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"gazeformer-scalable-effective-and-fast","repo_url":"https://github.com/cvlab-stonybrook/gazeformer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"eye-tracking","task_name":"Gaze Prediction"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"prediction","task_name":"Prediction"},{"task_slug":"scanpath-prediction","task_name":"Scanpath prediction"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2303.15274","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.15274"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cvlab-stonybrook/gazeformer","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/cvlab-stonybrook/Gazeformer","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":2},"by_repo_kind":{"official":{"samples":2,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5b328d3dec45baa1","entry":"PositionEmbeddingSine2d","repo":"cvlab-stonybrook/Gazeformer","repo_kind":"official","path":"gazeformer.py","file_url":"https://github.com/cvlab-stonybrook/Gazeformer/blob/HEAD/gazeformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5b328d3dec45baa1"}},{"code_sha256_prefix":"72ed4d9954e8e05f","entry":"gazeformer","repo":"cvlab-stonybrook/Gazeformer","repo_kind":"official","path":"gazeformer.py","file_url":"https://github.com/cvlab-stonybrook/Gazeformer/blob/HEAD/gazeformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"72ed4d9954e8e05f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}