{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rethinking-the-two-stage-framework-for","title":"Rethinking the Two-Stage Framework for Grounded Situation Recognition","arxiv_id":"2112.05375","date":"2021-12-10","proceeding":null,"authors":["Meng Wei","Long Chen","Wei Ji","Xiaoyu Yue","Tat-Seng Chua"],"abstract":"Grounded Situation Recognition (GSR), i.e., recognizing the salient activity (or verb) category in an image (e.g., buying) and detecting all corresponding semantic roles (e.g., agent and goods), is an essential step towards \"human-like\" event understanding. Since each verb is associated with a specific set of semantic roles, all existing GSR methods resort to a two-stage framework: predicting the verb in the first stage and detecting the semantic roles in the second stage. However, there are obvious drawbacks in both stages: 1) The widely-used cross-entropy (XE) loss for object recognition is insufficient in verb classification due to the large intra-class variation and high inter-class similarity among daily activities. 2) All semantic roles are detected in an autoregressive manner, which fails to model the complex semantic relations between different roles. To this end, we propose a novel SituFormer for GSR which consists of a Coarse-to-Fine Verb Model (CFVM) and a Transformer-based Noun Model (TNM). CFVM is a two-step verb prediction model: a coarse-grained model trained with XE loss first proposes a set of verb candidates, and then a fine-grained model trained with triplet loss re-ranks these candidates with enhanced verb features (not only separable but also discriminative). TNM is a transformer-based semantic role detection model, which detects all roles parallelly. Owing to the global relation modeling ability and flexibility of the transformer decoder, TNM can fully explore the statistical dependency of the roles. Extensive validations on the challenging SWiG benchmark show that SituFormer achieves a new state-of-the-art performance with significant gains under various metrics. Code is available at https://github.com/kellyiss/SituFormer.","url_abs":"https://arxiv.org/abs/2112.05375v1","url_pdf":"https://arxiv.org/pdf/2112.05375v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rethinking-the-two-stage-framework-for","repo_url":"https://github.com/kellyiss/situformer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"grounded-situation-recognition","task_name":"Grounded Situation Recognition"},{"task_slug":"object-recognition","task_name":"Object Recognition"},{"task_slug":"situation-recognition","task_name":"Situation Recognition"},{"task_slug":null,"task_name":"Triplet"},{"task_slug":"two","task_name":"Vocal Bursts Valence Prediction"}],"methods":[{"method_slug":"triplet-loss","method_name":"Triplet Loss"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/grounded-situation-recognition-on-swig","task":"Grounded Situation Recognition","dataset":"SWiG","model":"SituFormer","rank_in_archive_order":4,"of":13,"metrics":{"Top-1 Verb":"44.2","Top-1 Verb & Grounded-Value":"29.22","Top-1 Verb & Value":"35.24","Top-5 Verbs":"71.21","Top-5 Verbs & Grounded-Value":"46","Top-5 Verbs & Value":"55.75"},"uses_additional_data":false},{"leaderboard":"/sota/situation-recognition-on-imsitu","task":"Situation Recognition","dataset":"imSitu","model":"SituFormer","rank_in_archive_order":4,"of":13,"metrics":{"Top-1 Verb":"44.2","Top-1 Verb & Value":"35.24","Top-5 Verbs":"71.21","Top-5 Verbs & Value":"55.75"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2112.05375","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2112.05375"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kellyiss/situformer","reach":{"status":"ok"}}],"summary":{"ran":6,"ran_honours":3,"unverified":3},"by_repo_kind":{"official":{"samples":12,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"aa6da43d05cfaf65","entry":"all_gather","repo":"kellyiss/situformer","repo_kind":"official","path":"util/misc.py","file_url":"https://github.com/kellyiss/situformer/blob/HEAD/util/misc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa6da43d05cfaf65"}},{"code_sha256_prefix":"e0a06ded5d4f6c3c","entry":"box_cxcywh_to_xyxy","repo":"kellyiss/situformer","repo_kind":"official","path":"util/box_ops.py","file_url":"https://github.com/kellyiss/situformer/blob/HEAD/util/box_ops.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e0a06ded5d4f6c3c"}},{"code_sha256_prefix":"ce749f837424dd8c","entry":"box_iou","repo":"kellyiss/situformer","repo_kind":"official","path":"util/box_ops.py","file_url":"https://github.com/kellyiss/situformer/blob/HEAD/util/box_ops.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ce749f837424dd8c"}},{"code_sha256_prefix":"031b02fb990284ba","entry":"box_xyxy_to_cxcywh","repo":"kellyiss/situformer","repo_kind":"official","path":"util/box_ops.py","file_url":"https://github.com/kellyiss/situformer/blob/HEAD/util/box_ops.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"031b02fb990284ba"}},{"code_sha256_prefix":"74f3f5c1bab38c14","entry":"crop","repo":"kellyiss/situformer","repo_kind":"official","path":"datasets/transforms.py","file_url":"https://github.com/kellyiss/situformer/blob/HEAD/datasets/transforms.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"74f3f5c1bab38c14"}},{"code_sha256_prefix":"9be9e658b5bd688a","entry":"hflip","repo":"kellyiss/situformer","repo_kind":"official","path":"datasets/transforms.py","file_url":"https://github.com/kellyiss/situformer/blob/HEAD/datasets/transforms.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9be9e658b5bd688a"}},{"code_sha256_prefix":"d5c2ac8ea46e788e","entry":"make_sr_transforms","repo":"kellyiss/situformer","repo_kind":"official","path":"datasets/swig.py","file_url":"https://github.com/kellyiss/situformer/blob/HEAD/datasets/swig.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d5c2ac8ea46e788e"}},{"code_sha256_prefix":"f46bad6811c75514","entry":"parse_with_config","repo":"kellyiss/situformer","repo_kind":"official","path":"util/misc.py","file_url":"https://github.com/kellyiss/situformer/blob/HEAD/util/misc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f46bad6811c75514"}},{"code_sha256_prefix":"05992505df9ef0cd","entry":"reduce_dict","repo":"kellyiss/situformer","repo_kind":"official","path":"util/misc.py","file_url":"https://github.com/kellyiss/situformer/blob/HEAD/util/misc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"05992505df9ef0cd"}},{"code_sha256_prefix":"569e2388775fa1b7","entry":"all_reduce_sum","repo":"kellyiss/situformer","repo_kind":"official","path":"util/metrics.py","file_url":"https://github.com/kellyiss/situformer/blob/HEAD/util/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"569e2388775fa1b7"}},{"code_sha256_prefix":"da89ce279ae764b6","entry":"build_transformer","repo":"kellyiss/situformer","repo_kind":"official","path":"models/transformer.py","file_url":"https://github.com/kellyiss/situformer/blob/HEAD/models/transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"da89ce279ae764b6"}},{"code_sha256_prefix":"fe699ae8220daf07","entry":"plot_precision_recall","repo":"kellyiss/situformer","repo_kind":"official","path":"util/plot_utils.py","file_url":"https://github.com/kellyiss/situformer/blob/HEAD/util/plot_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fe699ae8220daf07"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}