{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/visual-distant-supervision-for-scene-graph","title":"Visual Distant Supervision for Scene Graph Generation","arxiv_id":"2103.15365","date":"2021-03-29","proceeding":"ICCV 2021 10","authors":["Yuan YAO","Ao Zhang","Xu Han","Mengdi Li","Cornelius Weber","Zhiyuan Liu","Stefan Wermter","Maosong Sun"],"abstract":"Scene graph generation aims to identify objects and their relations in images, providing structured image representations that can facilitate numerous applications in computer vision. However, scene graph models usually require supervised learning on large quantities of labeled data with intensive human annotation. In this work, we propose visual distant supervision, a novel paradigm of visual relation learning, which can train scene graph models without any human-labeled data. The intuition is that by aligning commonsense knowledge bases and images, we can automatically create large-scale labeled data to provide distant supervision for visual relation learning. To alleviate the noise in distantly labeled data, we further propose a framework that iteratively estimates the probabilistic relation labels and eliminates the noisy ones. Comprehensive experimental results show that our distantly supervised model outperforms strong weakly supervised and semi-supervised baselines. By further incorporating human-labeled data in a semi-supervised fashion, our model outperforms state-of-the-art fully supervised models by a large margin (e.g., 8.3 micro- and 7.8 macro-recall@50 improvements for predicate classification in Visual Genome evaluation). We make the data and code for this paper publicly available at https://github.com/thunlp/VisualDS.","url_abs":"https://arxiv.org/abs/2103.15365v2","url_pdf":"https://arxiv.org/pdf/2103.15365v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"visual-distant-supervision-for-scene-graph","repo_url":"https://github.com/thunlp/visualds","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"graph-generation","task_name":"Graph Generation"},{"task_slug":"predicate-classification","task_name":"Predicate Classification"},{"task_slug":null,"task_name":"Relation"},{"task_slug":"scene-graph-generation","task_name":"Scene Graph Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2103.15365","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2103.15365"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thunlp/visualds","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/thunlp/VisualDS","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":4},"by_repo_kind":{"official":{"samples":5,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e261fa29066b37e5","entry":"smooth_l1_loss","repo":"thunlp/VisualDS","repo_kind":"official","path":"maskrcnn_benchmark/layers/smooth_l1_loss.py","file_url":"https://github.com/thunlp/VisualDS/blob/HEAD/maskrcnn_benchmark/layers/smooth_l1_loss.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e261fa29066b37e5"}},{"code_sha256_prefix":"2d943c0c96e61f16","entry":"entropy_loss","repo":"thunlp/VisualDS","repo_kind":"official","path":"maskrcnn_benchmark/layers/entropy_loss.py","file_url":"https://github.com/thunlp/VisualDS/blob/HEAD/maskrcnn_benchmark/layers/entropy_loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2d943c0c96e61f16"}},{"code_sha256_prefix":"2902bf4410253ff7","entry":"interpolate","repo":"thunlp/VisualDS","repo_kind":"official","path":"maskrcnn_benchmark/layers/misc.py","file_url":"https://github.com/thunlp/VisualDS/blob/HEAD/maskrcnn_benchmark/layers/misc.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2902bf4410253ff7"}},{"code_sha256_prefix":"382942f3ada73679","entry":"kl_div_loss","repo":"thunlp/VisualDS","repo_kind":"official","path":"maskrcnn_benchmark/layers/kl_div_loss.py","file_url":"https://github.com/thunlp/VisualDS/blob/HEAD/maskrcnn_benchmark/layers/kl_div_loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"382942f3ada73679"}},{"code_sha256_prefix":"426fd02a2beaf826","entry":"sigmoid_focal_loss_cpu","repo":"thunlp/VisualDS","repo_kind":"official","path":"maskrcnn_benchmark/layers/sigmoid_focal_loss.py","file_url":"https://github.com/thunlp/VisualDS/blob/HEAD/maskrcnn_benchmark/layers/sigmoid_focal_loss.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"426fd02a2beaf826"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}