{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/transgop-transformer-based-gaze-object","title":"TransGOP: Transformer-Based Gaze Object Prediction","arxiv_id":"2402.13578","date":"2024-02-21","proceeding":null,"authors":["Binglu Wang","Chenxi Guo","Yang Jin","Haisheng Xia","Nian Liu"],"abstract":"Gaze object prediction aims to predict the location and category of the object that is watched by a human. Previous gaze object prediction works use CNN-based object detectors to predict the object's location. However, we find that Transformer-based object detectors can predict more accurate object location for dense objects in retail scenarios. Moreover, the long-distance modeling capability of the Transformer can help to build relationships between the human head and the gaze object, which is important for the GOP task. To this end, this paper introduces Transformer into the fields of gaze object prediction and proposes an end-to-end Transformer-based gaze object prediction method named TransGOP. Specifically, TransGOP uses an off-the-shelf Transformer-based object detector to detect the location of objects and designs a Transformer-based gaze autoencoder in the gaze regressor to establish long-distance gaze relationships. Moreover, to improve gaze heatmap regression, we propose an object-to-gaze cross-attention mechanism to let the queries of the gaze autoencoder learn the global-memory position knowledge from the object detector. Finally, to make the whole framework end-to-end trained, we propose a Gaze Box loss to jointly optimize the object detector and gaze regressor by enhancing the gaze heatmap energy in the box of the gaze object. Extensive experiments on the GOO-Synth and GOO-Real datasets demonstrate that our TransGOP achieves state-of-the-art performance on all tracks, i.e., object detection, gaze estimation, and gaze object prediction. Our code will be available at https://github.com/chenxi-Guo/TransGOP.git.","url_abs":"https://arxiv.org/abs/2402.13578v1","url_pdf":"https://arxiv.org/pdf/2402.13578v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"transgop-transformer-based-gaze-object","repo_url":"https://github.com/chenxi-guo/transgop","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"gaze-estimation","task_name":"Gaze Estimation"},{"task_slug":"object","task_name":"Object"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"prediction","task_name":"Prediction"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"heatmap","method_name":"Heatmap"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.13578","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.13578"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/chenxi-Guo/TransGOP","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":4,"ran":4,"ran_violates":1,"ran_fixture":1,"unverified":2},"by_repo_kind":{"official":{"samples":12,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6ef392f365cc5703","entry":"conv1x1","repo":"chenxi-Guo/TransGOP","repo_kind":"official","path":"models/TransGOP/resnet.py","file_url":"https://github.com/chenxi-Guo/TransGOP/blob/HEAD/models/TransGOP/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6ef392f365cc5703"}},{"code_sha256_prefix":"e360ecbff4906503","entry":"conv2d","repo":"chenxi-Guo/TransGOP","repo_kind":"official","path":"models/TransGOP/gazeregressor.py","file_url":"https://github.com/chenxi-Guo/TransGOP/blob/HEAD/models/TransGOP/gazeregressor.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e360ecbff4906503"}},{"code_sha256_prefix":"794a2ed91a5175bd","entry":"conv3x3","repo":"chenxi-Guo/TransGOP","repo_kind":"official","path":"models/TransGOP/resnet.py","file_url":"https://github.com/chenxi-Guo/TransGOP/blob/HEAD/models/TransGOP/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"794a2ed91a5175bd"}},{"code_sha256_prefix":"ac8fe530cdad4d8c","entry":"dice_loss","repo":"chenxi-Guo/TransGOP","repo_kind":"official","path":"models/TransGOP/segmentation.py","file_url":"https://github.com/chenxi-Guo/TransGOP/blob/HEAD/models/TransGOP/segmentation.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ac8fe530cdad4d8c"}},{"code_sha256_prefix":"dc279ea790da466f","entry":"dn_post_process","repo":"chenxi-Guo/TransGOP","repo_kind":"official","path":"models/TransGOP/dn_components.py","file_url":"https://github.com/chenxi-Guo/TransGOP/blob/HEAD/models/TransGOP/dn_components.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dc279ea790da466f"}},{"code_sha256_prefix":"c86e1b214c40abea","entry":"make_five_conv","repo":"chenxi-Guo/TransGOP","repo_kind":"official","path":"models/TransGOP/gazeregressor.py","file_url":"https://github.com/chenxi-Guo/TransGOP/blob/HEAD/models/TransGOP/gazeregressor.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c86e1b214c40abea"}},{"code_sha256_prefix":"fa46ce91ea26bece","entry":"make_three_conv","repo":"chenxi-Guo/TransGOP","repo_kind":"official","path":"models/TransGOP/gazeregressor.py","file_url":"https://github.com/chenxi-Guo/TransGOP/blob/HEAD/models/TransGOP/gazeregressor.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fa46ce91ea26bece"}},{"code_sha256_prefix":"5c0711aada67957e","entry":"sigmoid_focal_loss","repo":"chenxi-Guo/TransGOP","repo_kind":"official","path":"models/TransGOP/segmentation.py","file_url":"https://github.com/chenxi-Guo/TransGOP/blob/HEAD/models/TransGOP/segmentation.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5c0711aada67957e"}},{"code_sha256_prefix":"f9fd6241d935f07b","entry":"window_partition","repo":"chenxi-Guo/TransGOP","repo_kind":"official","path":"models/TransGOP/swin_transformer.py","file_url":"https://github.com/chenxi-Guo/TransGOP/blob/HEAD/models/TransGOP/swin_transformer.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f9fd6241d935f07b"}},{"code_sha256_prefix":"fb32094c6dbece71","entry":"window_reverse","repo":"chenxi-Guo/TransGOP","repo_kind":"official","path":"models/TransGOP/swin_transformer.py","file_url":"https://github.com/chenxi-Guo/TransGOP/blob/HEAD/models/TransGOP/swin_transformer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fb32094c6dbece71"}},{"code_sha256_prefix":"a82091666b9db8cf","entry":"multi_head_attention_forward","repo":"chenxi-Guo/TransGOP","repo_kind":"official","path":"models/TransGOP/attention.py","file_url":"https://github.com/chenxi-Guo/TransGOP/blob/HEAD/models/TransGOP/attention.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a82091666b9db8cf"}},{"code_sha256_prefix":"347e83ff50f45f03","entry":"resnet18","repo":"chenxi-Guo/TransGOP","repo_kind":"official","path":"models/TransGOP/resnet.py","file_url":"https://github.com/chenxi-Guo/TransGOP/blob/HEAD/models/TransGOP/resnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"347e83ff50f45f03"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}