{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/2408-01044","title":"Boosting Gaze Object Prediction via Pixel-level Supervision from Vision Foundation Model","arxiv_id":"2408.01044","date":"2024-08-02","proceeding":null,"authors":["Yang Jin","Lei Zhang","Shi Yan","Bin Fan","Binglu Wang"],"abstract":"Gaze object prediction (GOP) aims to predict the category and location of the object that a human is looking at. Previous methods utilized box-level supervision to identify the object that a person is looking at, but struggled with semantic ambiguity, ie, a single box may contain several items since objects are close together. The Vision foundation model (VFM) has improved in object segmentation using box prompts, which can reduce confusion by more precisely locating objects, offering advantages for fine-grained prediction of gaze objects. This paper presents a more challenging gaze object segmentation (GOS) task, which involves inferring the pixel-level mask corresponding to the object captured by human gaze behavior. In particular, we propose that the pixel-level supervision provided by VFM can be integrated into gaze object prediction to mitigate semantic ambiguity. This leads to our gaze object detection and segmentation framework that enables accurate pixel-level predictions. Different from previous methods that require additional head input or ignore head features, we propose to automatically obtain head features from scene features to ensure the model's inference efficiency and flexibility in the real world. Moreover, rather than directly fuse features to predict gaze heatmap as in existing methods, which may overlook spatial location and subtle details of the object, we develop a space-to-object gaze regression method to facilitate human-object gaze interaction. Specifically, it first constructs an initial human-object spatial connection, then refines this connection by interacting with semantically clear features in the segmentation branch, ultimately predicting a gaze heatmap for precise localization. Extensive experiments on GOO-Synth and GOO-Real datasets demonstrate the effectiveness of our method.","url_abs":"https://arxiv.org/abs/2408.01044v1","url_pdf":"https://arxiv.org/pdf/2408.01044v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"2408-01044","repo_url":"https://github.com/jinyang06/SamGOP","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"object","task_name":"Object"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"heatmap","method_name":"Heatmap"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2408.01044","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.01044"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jinyang06/SamGOP","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":3,"ran":4,"unverified":5},"by_repo_kind":{"official":{"samples":12,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6ef392f365cc5703","entry":"conv1x1","repo":"jinyang06/SamGOP","repo_kind":"official","path":"maskGOP/resnet.py","file_url":"https://github.com/jinyang06/SamGOP/blob/HEAD/maskGOP/resnet.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6ef392f365cc5703"}},{"code_sha256_prefix":"cc5b66776a7eeaff","entry":"conv2d","repo":"jinyang06/SamGOP","repo_kind":"official","path":"maskGOP/maskGOP.py","file_url":"https://github.com/jinyang06/SamGOP/blob/HEAD/maskGOP/maskGOP.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cc5b66776a7eeaff"}},{"code_sha256_prefix":"794a2ed91a5175bd","entry":"conv3x3","repo":"jinyang06/SamGOP","repo_kind":"official","path":"maskGOP/resnet.py","file_url":"https://github.com/jinyang06/SamGOP/blob/HEAD/maskGOP/resnet.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"794a2ed91a5175bd"}},{"code_sha256_prefix":"12e95d4ff97d5ae5","entry":"gaze_loss","repo":"jinyang06/SamGOP","repo_kind":"official","path":"maskGOP/modeling/criterion.py","file_url":"https://github.com/jinyang06/SamGOP/blob/HEAD/maskGOP/modeling/criterion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"12e95d4ff97d5ae5"}},{"code_sha256_prefix":"02607cd42ff778d8","entry":"get_head_channel","repo":"jinyang06/SamGOP","repo_kind":"official","path":"maskGOP/maskGOP.py","file_url":"https://github.com/jinyang06/SamGOP/blob/HEAD/maskGOP/maskGOP.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"02607cd42ff778d8"}},{"code_sha256_prefix":"e977ad9fb0c42add","entry":"resnet50","repo":"jinyang06/SamGOP","repo_kind":"official","path":"maskGOP/resnet.py","file_url":"https://github.com/jinyang06/SamGOP/blob/HEAD/maskGOP/resnet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e977ad9fb0c42add"}},{"code_sha256_prefix":"5c0711aada67957e","entry":"sigmoid_focal_loss","repo":"jinyang06/SamGOP","repo_kind":"official","path":"maskGOP/modeling/criterion.py","file_url":"https://github.com/jinyang06/SamGOP/blob/HEAD/maskGOP/modeling/criterion.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5c0711aada67957e"}},{"code_sha256_prefix":"bc2cb481a75c370d","entry":"batch_dice_loss","repo":"jinyang06/SamGOP","repo_kind":"official","path":"maskGOP/modeling/detr_matcher.py","file_url":"https://github.com/jinyang06/SamGOP/blob/HEAD/maskGOP/modeling/detr_matcher.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bc2cb481a75c370d"}},{"code_sha256_prefix":"1edd24985036b0bf","entry":"batch_sigmoid_ce_loss","repo":"jinyang06/SamGOP","repo_kind":"official","path":"maskGOP/modeling/detr_matcher.py","file_url":"https://github.com/jinyang06/SamGOP/blob/HEAD/maskGOP/modeling/detr_matcher.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1edd24985036b0bf"}},{"code_sha256_prefix":"f52bf9a389e50bbc","entry":"build_transformer","repo":"jinyang06/SamGOP","repo_kind":"official","path":"maskGOP/transformer.py","file_url":"https://github.com/jinyang06/SamGOP/blob/HEAD/maskGOP/transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f52bf9a389e50bbc"}},{"code_sha256_prefix":"89f75e54ff128be0","entry":"dice_loss","repo":"jinyang06/SamGOP","repo_kind":"official","path":"maskGOP/modeling/criterion.py","file_url":"https://github.com/jinyang06/SamGOP/blob/HEAD/maskGOP/modeling/criterion.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"89f75e54ff128be0"}},{"code_sha256_prefix":"ca2fbb7977db964a","entry":"select_head","repo":"jinyang06/SamGOP","repo_kind":"official","path":"maskGOP/maskGOP.py","file_url":"https://github.com/jinyang06/SamGOP/blob/HEAD/maskGOP/maskGOP.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ca2fbb7977db964a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}