{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/gsrformer-grounded-situation-recognition","title":"GSRFormer: Grounded Situation Recognition Transformer with Alternate Semantic Attention Refinement","arxiv_id":"2208.08965","date":"2022-08-18","proceeding":null,"authors":["Zhi-Qi Cheng","Qi Dai","SiYao Li","Teruko Mitamura","Alexander G. Hauptmann"],"abstract":"Grounded Situation Recognition (GSR) aims to generate structured semantic summaries of images for \"human-like\" event understanding. Specifically, GSR task not only detects the salient activity verb (e.g. buying), but also predicts all corresponding semantic roles (e.g. agent and goods). Inspired by object detection and image captioning tasks, existing methods typically employ a two-stage framework: 1) detect the activity verb, and then 2) predict semantic roles based on the detected verb. Obviously, this illogical framework constitutes a huge obstacle to semantic understanding. First, pre-detecting verbs solely without semantic roles inevitably fails to distinguish many similar daily activities (e.g., offering and giving, buying and selling). Second, predicting semantic roles in a closed auto-regressive manner can hardly exploit the semantic relations among the verb and roles. To this end, in this paper we propose a novel two-stage framework that focuses on utilizing such bidirectional relations within verbs and roles. In the first stage, instead of pre-detecting the verb, we postpone the detection step and assume a pseudo label, where an intermediate representation for each corresponding semantic role is learned from images. In the second stage, we exploit transformer layers to unearth the potential semantic relations within both verbs and semantic roles. With the help of a set of support images, an alternate learning scheme is designed to simultaneously optimize the results: update the verb using nouns corresponding to the image, and update nouns using verbs from support images. Extensive experimental results on challenging SWiG benchmarks show that our renovated framework outperforms other state-of-the-art methods under various metrics.","url_abs":"https://arxiv.org/abs/2208.08965v4","url_pdf":"https://arxiv.org/pdf/2208.08965v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"gsrformer-grounded-situation-recognition","repo_url":"https://github.com/zhiqic/gsrformer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"grounded-situation-recognition","task_name":"Grounded Situation Recognition"},{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"pseudo-label","task_name":"Pseudo Label"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2208.08965","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2208.08965"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhiqic/gsrformer","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":2,"ran_honours":2,"unverified":7},"by_repo_kind":{"official":{"samples":11,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"aa6da43d05cfaf65","entry":"all_gather","repo":"zhiqic/gsrformer","repo_kind":"official","path":"util/misc.py","file_url":"https://github.com/zhiqic/gsrformer/blob/HEAD/util/misc.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"aa6da43d05cfaf65"}},{"code_sha256_prefix":"e0a06ded5d4f6c3c","entry":"box_cxcywh_to_xyxy","repo":"zhiqic/gsrformer","repo_kind":"official","path":"util/box_ops.py","file_url":"https://github.com/zhiqic/gsrformer/blob/HEAD/util/box_ops.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"e0a06ded5d4f6c3c"}},{"code_sha256_prefix":"031b02fb990284ba","entry":"box_xyxy_to_cxcywh","repo":"zhiqic/gsrformer","repo_kind":"official","path":"util/box_ops.py","file_url":"https://github.com/zhiqic/gsrformer/blob/HEAD/util/box_ops.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"031b02fb990284ba"}},{"code_sha256_prefix":"05992505df9ef0cd","entry":"reduce_dict","repo":"zhiqic/gsrformer","repo_kind":"official","path":"util/misc.py","file_url":"https://github.com/zhiqic/gsrformer/blob/HEAD/util/misc.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"05992505df9ef0cd"}},{"code_sha256_prefix":"e335e7e2c2f17628","entry":"build","repo":"zhiqic/gsrformer","repo_kind":"official","path":"datasets/swig.py","file_url":"https://github.com/zhiqic/gsrformer/blob/HEAD/datasets/swig.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"e335e7e2c2f17628"}},{"code_sha256_prefix":"5acac9cbe77725d2","entry":"build_decoder_transformer","repo":"zhiqic/gsrformer","repo_kind":"official","path":"models/transformer.py","file_url":"https://github.com/zhiqic/gsrformer/blob/HEAD/models/transformer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"5acac9cbe77725d2"}},{"code_sha256_prefix":"c45c284723243254","entry":"build_encoder_decoder","repo":"zhiqic/gsrformer","repo_kind":"official","path":"models/transformer.py","file_url":"https://github.com/zhiqic/gsrformer/blob/HEAD/models/transformer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"c45c284723243254"}},{"code_sha256_prefix":"bb3aeef20beceb00","entry":"build_encoder_encoder","repo":"zhiqic/gsrformer","repo_kind":"official","path":"models/transformer.py","file_url":"https://github.com/zhiqic/gsrformer/blob/HEAD/models/transformer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"bb3aeef20beceb00"}},{"code_sha256_prefix":"458a46394aa87a2f","entry":"collate_fn","repo":"zhiqic/gsrformer","repo_kind":"official","path":"util/misc.py","file_url":"https://github.com/zhiqic/gsrformer/blob/HEAD/util/misc.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"458a46394aa87a2f"}},{"code_sha256_prefix":"7f06889563fd00ce","entry":"noun2synset","repo":"zhiqic/gsrformer","repo_kind":"official","path":"inference.py","file_url":"https://github.com/zhiqic/gsrformer/blob/HEAD/inference.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"7f06889563fd00ce"}},{"code_sha256_prefix":"fdd34e83fa6d7086","entry":"swig_box_xyxy_to_cxcywh","repo":"zhiqic/gsrformer","repo_kind":"official","path":"util/box_ops.py","file_url":"https://github.com/zhiqic/gsrformer/blob/HEAD/util/box_ops.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"fdd34e83fa6d7086"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}