{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-to-compose-dynamic-tree-structures","title":"Learning to Compose Dynamic Tree Structures for Visual Contexts","arxiv_id":"1812.01880","date":"2018-12-05","proceeding":"CVPR 2019 6","authors":["Kaihua Tang","Hanwang Zhang","Baoyuan Wu","Wenhan Luo","Wei Liu"],"abstract":"We propose to compose dynamic tree structures that place the objects in an\nimage into a visual context, helping visual reasoning tasks such as scene graph\ngeneration and visual Q&A. Our visual context tree model, dubbed VCTree, has\ntwo key advantages over existing structured object representations including\nchains and fully-connected graphs: 1) The efficient and expressive binary tree\nencodes the inherent parallel/hierarchical relationships among objects, e.g.,\n\"clothes\" and \"pants\" are usually co-occur and belong to \"person\"; 2) the\ndynamic structure varies from image to image and task to task, allowing more\ncontent-/task-specific message passing among objects. To construct a VCTree, we\ndesign a score function that calculates the task-dependent validity between\neach object pair, and the tree is the binary version of the maximum spanning\ntree from the score matrix. Then, visual contexts are encoded by bidirectional\nTreeLSTM and decoded by task-specific models. We develop a hybrid learning\nprocedure which integrates end-task supervised learning and the tree structure\nreinforcement learning, where the former's evaluation result serves as a\nself-critic for the latter's structure exploration. Experimental results on two\nbenchmarks, which require reasoning over contexts: Visual Genome for scene\ngraph generation and VQA2.0 for visual Q&A, show that VCTree outperforms\nstate-of-the-art results while discovering interpretable visual context\nstructures.","url_abs":"http://arxiv.org/abs/1812.01880v1","url_pdf":"http://arxiv.org/pdf/1812.01880v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-to-compose-dynamic-tree-structures","repo_url":"https://github.com/KaihuaTang/Scene-Graph-Benchmark.pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"learning-to-compose-dynamic-tree-structures","repo_url":"https://github.com/KaihuaTang/VCTree-Scene-Graph-Generation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"learning-to-compose-dynamic-tree-structures","repo_url":"https://github.com/KaihuaTang/VCTree-Visual-Question-Answering","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"learning-to-compose-dynamic-tree-structures","repo_url":"https://github.com/Karim-53/SGG","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"learning-to-compose-dynamic-tree-structures","repo_url":"https://github.com/ihaeyong/unbiased-sgg","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"learning-to-compose-dynamic-tree-structures","repo_url":"https://github.com/zacharie12/zacharie","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"graph-generation","task_name":"Graph Generation"},{"task_slug":"panoptic-scene-graph-generation","task_name":"Panoptic Scene Graph Generation"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"scene-graph-generation","task_name":"Scene Graph Generation"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/panoptic-scene-graph-generation-on-psg","task":"Panoptic Scene Graph Generation","dataset":"PSG Dataset","model":"VCTree","rank_in_archive_order":7,"of":9,"metrics":{"R@20":"20.6","mR@20":"9.70"},"uses_additional_data":false},{"leaderboard":"/sota/scene-graph-generation-on-visual-genome","task":"Scene Graph Generation","dataset":"Visual Genome","model":"VCTree","rank_in_archive_order":7,"of":19,"metrics":{"Recall@50":"27.9"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1812.01880","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1812.01880"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/KaihuaTang/VCTree-Visual-Question-Answering","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ihaeyong/unbiased-sgg","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/KaihuaTang/VCTree-Scene-Graph-Generation","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Karim-53/SGG","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/KaihuaTang/Scene-Graph-Benchmark.pytorch","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zacharie12/zacharie","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran_fixture":1,"ran":2,"ran_violates":1,"unverified":10},"by_repo_kind":{"listed":{"samples":14,"ran":4,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fb40006c9cd123ac","entry":"apply_attention","repo":"KaihuaTang/VCTree-Visual-Question-Answering","repo_kind":"listed","path":"model.py","file_url":"https://github.com/KaihuaTang/VCTree-Visual-Question-Answering/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fb40006c9cd123ac"}},{"code_sha256_prefix":"a1e587129dd2dfaf","entry":"get_ranking","repo":"KaihuaTang/VCTree-Scene-Graph-Generation","repo_kind":"listed","path":"lib/pytorch_misc.py","file_url":"https://github.com/KaihuaTang/VCTree-Scene-Graph-Generation/blob/HEAD/lib/pytorch_misc.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a1e587129dd2dfaf"}},{"code_sha256_prefix":"fef49730c9e9a2b8","entry":"pairwise","repo":"KaihuaTang/VCTree-Scene-Graph-Generation","repo_kind":"listed","path":"lib/pytorch_misc.py","file_url":"https://github.com/KaihuaTang/VCTree-Scene-Graph-Generation/blob/HEAD/lib/pytorch_misc.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fef49730c9e9a2b8"}},{"code_sha256_prefix":"b3299f7858d7d2f2","entry":"tile_2d_over_nd","repo":"KaihuaTang/VCTree-Visual-Question-Answering","repo_kind":"listed","path":"model.py","file_url":"https://github.com/KaihuaTang/VCTree-Visual-Question-Answering/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b3299f7858d7d2f2"}},{"code_sha256_prefix":"bacd2c4e4708f3a1","entry":"arTree_to_biTree","repo":"KaihuaTang/VCTree-Scene-Graph-Generation","repo_kind":"listed","path":"lib/tree_lstm/graph_to_tree.py","file_url":"https://github.com/KaihuaTang/VCTree-Scene-Graph-Generation/blob/HEAD/lib/tree_lstm/graph_to_tree.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bacd2c4e4708f3a1"}},{"code_sha256_prefix":"cd49722cb881389f","entry":"arbitraryForest_to_biForest","repo":"KaihuaTang/VCTree-Scene-Graph-Generation","repo_kind":"listed","path":"lib/tree_lstm/graph_to_tree.py","file_url":"https://github.com/KaihuaTang/VCTree-Scene-Graph-Generation/blob/HEAD/lib/tree_lstm/graph_to_tree.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cd49722cb881389f"}},{"code_sha256_prefix":"23725d7c2832de1b","entry":"filter_dets","repo":"KaihuaTang/VCTree-Scene-Graph-Generation","repo_kind":"listed","path":"lib/surgery.py","file_url":"https://github.com/KaihuaTang/VCTree-Scene-Graph-Generation/blob/HEAD/lib/surgery.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"23725d7c2832de1b"}},{"code_sha256_prefix":"013f03d1624b019d","entry":"gather_res","repo":"KaihuaTang/VCTree-Scene-Graph-Generation","repo_kind":"listed","path":"lib/object_detector.py","file_url":"https://github.com/KaihuaTang/VCTree-Scene-Graph-Generation/blob/HEAD/lib/object_detector.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"013f03d1624b019d"}},{"code_sha256_prefix":"da7576453008a270","entry":"load_word_vectors","repo":"KaihuaTang/VCTree-Scene-Graph-Generation","repo_kind":"listed","path":"lib/word_vectors.py","file_url":"https://github.com/KaihuaTang/VCTree-Scene-Graph-Generation/blob/HEAD/lib/word_vectors.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"da7576453008a270"}},{"code_sha256_prefix":"6d0daf87a6ec7e39","entry":"optimistic_restore","repo":"KaihuaTang/VCTree-Scene-Graph-Generation","repo_kind":"listed","path":"lib/pytorch_misc.py","file_url":"https://github.com/KaihuaTang/VCTree-Scene-Graph-Generation/blob/HEAD/lib/pytorch_misc.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6d0daf87a6ec7e39"}},{"code_sha256_prefix":"5d64438de6614360","entry":"path","repo":"KaihuaTang/VCTree-Scene-Graph-Generation","repo_kind":"listed","path":"config.py","file_url":"https://github.com/KaihuaTang/VCTree-Scene-Graph-Generation/blob/HEAD/config.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5d64438de6614360"}},{"code_sha256_prefix":"b0ccc0da43f7ec20","entry":"reporthook","repo":"KaihuaTang/VCTree-Scene-Graph-Generation","repo_kind":"listed","path":"lib/word_vectors.py","file_url":"https://github.com/KaihuaTang/VCTree-Scene-Graph-Generation/blob/HEAD/lib/word_vectors.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b0ccc0da43f7ec20"}},{"code_sha256_prefix":"081b0b43ff7d9810","entry":"stanford_path","repo":"KaihuaTang/VCTree-Scene-Graph-Generation","repo_kind":"listed","path":"config.py","file_url":"https://github.com/KaihuaTang/VCTree-Scene-Graph-Generation/blob/HEAD/config.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"081b0b43ff7d9810"}},{"code_sha256_prefix":"16d7f949c9b0c390","entry":"vgg_fc","repo":"KaihuaTang/VCTree-Scene-Graph-Generation","repo_kind":"listed","path":"lib/resnet.py","file_url":"https://github.com/KaihuaTang/VCTree-Scene-Graph-Generation/blob/HEAD/lib/resnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"16d7f949c9b0c390"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}