{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/block-bilinear-superdiagonal-fusion-for","title":"BLOCK: Bilinear Superdiagonal Fusion for Visual Question Answering and Visual Relationship Detection","arxiv_id":"1902.00038","date":"2019-01-31","proceeding":null,"authors":["Hedi Ben-Younes","Rémi Cadene","Nicolas Thome","Matthieu Cord"],"abstract":"Multimodal representation learning is gaining more and more interest within\nthe deep learning community. While bilinear models provide an interesting\nframework to find subtle combination of modalities, their number of parameters\ngrows quadratically with the input dimensions, making their practical\nimplementation within classical deep learning pipelines challenging. In this\npaper, we introduce BLOCK, a new multimodal fusion based on the\nblock-superdiagonal tensor decomposition. It leverages the notion of block-term\nranks, which generalizes both concepts of rank and mode ranks for tensors,\nalready used for multimodal fusion. It allows to define new ways for optimizing\nthe tradeoff between the expressiveness and complexity of the fusion model, and\nis able to represent very fine interactions between modalities while\nmaintaining powerful mono-modal representations. We demonstrate the practical\ninterest of our fusion model by using BLOCK for two challenging tasks: Visual\nQuestion Answering (VQA) and Visual Relationship Detection (VRD), where we\ndesign end-to-end learnable architectures for representing relevant\ninteractions between modalities. Through extensive experiments, we show that\nBLOCK compares favorably with respect to state-of-the-art multimodal fusion\nmodels for both VQA and VRD tasks. Our code is available at\nhttps://github.com/Cadene/block.bootstrap.pytorch.","url_abs":"http://arxiv.org/abs/1902.00038v2","url_pdf":"http://arxiv.org/pdf/1902.00038v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"block-bilinear-superdiagonal-fusion-for","repo_url":"https://github.com/Cadene/block.bootstrap.pytorch","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"relationship-detection","task_name":"Relationship Detection"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"tensor-decomposition","task_name":"Tensor Decomposition"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"visual-relationship-detection","task_name":"Visual Relationship Detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-question-answering-on-vqa-v2-test-dev","task":"Visual Question Answering (VQA)","dataset":"VQA v2 test-dev","model":"BLOCK","rank_in_archive_order":38,"of":56,"metrics":{"Accuracy":"67.58"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-vqa-v2-test-std","task":"Visual Question Answering (VQA)","dataset":"VQA v2 test-std","model":"BLOCK","rank_in_archive_order":33,"of":38,"metrics":{"overall":"67.9"},"uses_additional_data":false},{"leaderboard":"/sota/visual-relationship-detection-on-vrd-phrase","task":"Visual Relationship Detection","dataset":"VRD Phrase Detection","model":"BLOCK","rank_in_archive_order":2,"of":7,"metrics":{"R@100":"28.96","R@50":"26.32"},"uses_additional_data":false},{"leaderboard":"/sota/visual-relationship-detection-on-vrd","task":"Visual Relationship Detection","dataset":"VRD Predicate Detection","model":"BLOCK","rank_in_archive_order":3,"of":7,"metrics":{"R@100":"92.58","R@50":"86.58"},"uses_additional_data":false},{"leaderboard":"/sota/visual-relationship-detection-on-vrd-1","task":"Visual Relationship Detection","dataset":"VRD Relationship Detection","model":"BLOCK","rank_in_archive_order":3,"of":8,"metrics":{"R@100":"20.96","R@50":"19.06"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/1902.00038","atlas_url":"https://app.syntology.ai/?focus=1902.00038","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1902.00038"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Cadene/block.bootstrap.pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"unverified":2},"by_repo_kind":{"official":{"samples":2,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"88ff7954b60fad9c","entry":"tokenize","repo":"Cadene/block.bootstrap.pytorch","repo_kind":"official","path":"block/datasets/vqa_utils.py","file_url":"https://github.com/Cadene/block.bootstrap.pytorch/blob/HEAD/block/datasets/vqa_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"88ff7954b60fad9c"}},{"code_sha256_prefix":"4efccc8130f403ef","entry":"tokenize_mcb","repo":"Cadene/block.bootstrap.pytorch","repo_kind":"official","path":"block/datasets/vqa_utils.py","file_url":"https://github.com/Cadene/block.bootstrap.pytorch/blob/HEAD/block/datasets/vqa_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"4efccc8130f403ef"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}