{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/video-text-as-game-players-hierarchical","title":"Video-Text as Game Players: Hierarchical Banzhaf Interaction for Cross-Modal Representation Learning","arxiv_id":"2303.14369","date":"2023-03-25","proceeding":"CVPR 2023 1","authors":["Peng Jin","Jinfa Huang","Pengfei Xiong","Shangxuan Tian","Chang Liu","Xiangyang Ji","Li Yuan","Jie Chen"],"abstract":"Contrastive learning-based video-language representation learning approaches, e.g., CLIP, have achieved outstanding performance, which pursue semantic interaction upon pre-defined video-text pairs. To clarify this coarse-grained global interaction and move a step further, we have to encounter challenging shell-breaking interactions for fine-grained cross-modal learning. In this paper, we creatively model video-text as game players with multivariate cooperative game theory to wisely handle the uncertainty during fine-grained semantic interaction with diverse granularity, flexible combination, and vague intensity. Concretely, we propose Hierarchical Banzhaf Interaction (HBI) to value possible correspondence between video frames and text words for sensitive and explainable cross-modal contrast. To efficiently realize the cooperative game of multiple video frames and multiple text words, the proposed method clusters the original video frames (text words) and computes the Banzhaf Interaction between the merged tokens. By stacking token merge modules, we achieve cooperative games at different semantic levels. Extensive experiments on commonly used text-video retrieval and video-question answering benchmarks with superior performances justify the efficacy of our HBI. More encouragingly, it can also serve as a visualization tool to promote the understanding of cross-modal interaction, which have a far-reaching impact on the community. Project page is available at https://jpthu17.github.io/HBI/.","url_abs":"https://arxiv.org/abs/2303.14369v1","url_pdf":"https://arxiv.org/pdf/2303.14369v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"video-text-as-game-players-hierarchical","repo_url":"https://github.com/jpthu17/HBI","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"video-text-as-game-players-hierarchical","repo_url":"https://github.com/jpthu17/dicosa","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"video-text-as-game-players-hierarchical","repo_url":"https://github.com/jpthu17/diffusionret","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"video-text-as-game-players-hierarchical","repo_url":"https://github.com/jpthu17/emcl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"video-question-answering","task_name":"Video Question Answering"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-question-answering-on-msrvtt-qa","task":"Video Question Answering","dataset":"MSRVTT-QA","model":"HBI","rank_in_archive_order":8,"of":14,"metrics":{"Accuracy":"46.2"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-activitynet","task":"Video Retrieval","dataset":"ActivityNet","model":"HBI","rank_in_archive_order":23,"of":31,"metrics":{"text-to-video Mean Rank":"6.6","text-to-video Median Rank":"2.0","text-to-video R@1":"42.2","text-to-video R@10":"84.6","text-to-video R@5":"73.0","video-to-text Mean Rank":"6.5","video-to-text Median Rank":"2.0","video-to-text R@1":"42.4","video-to-text R@10":"86.0","video-to-text R@5":"73.0"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-didemo","task":"Video Retrieval","dataset":"DiDeMo","model":"HBI","rank_in_archive_order":30,"of":40,"metrics":{"text-to-video Mean Rank":"12.1","text-to-video Median Rank":"2.0","text-to-video R@1":"46.9","text-to-video R@10":"82.7","text-to-video R@5":"74.9","video-to-text Mean Rank":"8.7","video-to-text Median Rank":"2.0","video-to-text R@1":"46.2","video-to-text R@10":"82.7","video-to-text R@5":"73.0"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-msr-vtt-1ka","task":"Video Retrieval","dataset":"MSR-VTT-1kA","model":"HBI","rank_in_archive_order":24,"of":63,"metrics":{"text-to-video Mean Rank":"12.0","text-to-video Median Rank":"2.0","text-to-video R@1":"48.6","text-to-video R@10":"83.4","text-to-video R@5":"74.6","video-to-text Mean Rank":"8.9","video-to-text Median Rank":"2.0","video-to-text R@1":"46.8","video-to-text R@10":"84.3","video-to-text R@5":"74.3"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-msrvtt-qa-1","task":"Visual Question Answering (VQA)","dataset":"MSRVTT-QA","model":"HBI","rank_in_archive_order":11,"of":34,"metrics":{"Accuracy":"0.462"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2303.14369","atlas_url":"https://app.syntology.ai/?focus=2303.14369","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.14369"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jpthu17/dicosa","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jpthu17/HBI","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jpthu17/emcl","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jpthu17/diffusionret","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":11,"ran_draft_wrong":1,"unverified":4},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":15,"ran":11,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"72bb773921596f80","entry":"AllGather","repo":"jpthu17/dicosa","repo_kind":"listed","path":"tvr/models/modeling.py","file_url":"https://github.com/jpthu17/dicosa/blob/HEAD/tvr/models/modeling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"72bb773921596f80"}},{"code_sha256_prefix":"9dbc40efee885693","entry":"AttentionPool2d","repo":"jpthu17/dicosa","repo_kind":"listed","path":"tvr/models/modeling.py","file_url":"https://github.com/jpthu17/dicosa/blob/HEAD/tvr/models/modeling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9dbc40efee885693"}},{"code_sha256_prefix":"e707b0fe8aa85751","entry":"BanzhafInteraction","repo":"jpthu17/HBI","repo_kind":"official","path":"HBI/models/banzhaf.py","file_url":"https://github.com/jpthu17/HBI/blob/HEAD/HBI/models/banzhaf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e707b0fe8aa85751"}},{"code_sha256_prefix":"d9e854326a29af63","entry":"Bottleneck","repo":"jpthu17/dicosa","repo_kind":"listed","path":"tvr/models/modeling.py","file_url":"https://github.com/jpthu17/dicosa/blob/HEAD/tvr/models/modeling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d9e854326a29af63"}},{"code_sha256_prefix":"0303fad775cdc882","entry":"CrossEn","repo":"jpthu17/dicosa","repo_kind":"listed","path":"tvr/models/modeling.py","file_url":"https://github.com/jpthu17/dicosa/blob/HEAD/tvr/models/modeling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0303fad775cdc882"}},{"code_sha256_prefix":"2eac2ed706ba48cd","entry":"Emcl","repo":"jpthu17/emcl","repo_kind":"listed","path":"video_retrieval/EMCL-Net/tvr/models/modeling.py","file_url":"https://github.com/jpthu17/emcl/blob/HEAD/video_retrieval/EMCL-Net/tvr/models/modeling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2eac2ed706ba48cd"}},{"code_sha256_prefix":"09e8fef38647435a","entry":"ModifiedResNet","repo":"jpthu17/dicosa","repo_kind":"listed","path":"tvr/models/modeling.py","file_url":"https://github.com/jpthu17/dicosa/blob/HEAD/tvr/models/modeling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"09e8fef38647435a"}},{"code_sha256_prefix":"99083d92c6568c29","entry":"ResidualAttentionBlock","repo":"jpthu17/dicosa","repo_kind":"listed","path":"tvr/models/modeling.py","file_url":"https://github.com/jpthu17/dicosa/blob/HEAD/tvr/models/modeling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"99083d92c6568c29"}},{"code_sha256_prefix":"80385a1374622ebf","entry":"Transformer","repo":"jpthu17/dicosa","repo_kind":"listed","path":"tvr/models/modeling.py","file_url":"https://github.com/jpthu17/dicosa/blob/HEAD/tvr/models/modeling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"80385a1374622ebf"}},{"code_sha256_prefix":"bab52ac82134c899","entry":"VisualTransformer","repo":"jpthu17/dicosa","repo_kind":"listed","path":"tvr/models/modeling.py","file_url":"https://github.com/jpthu17/dicosa/blob/HEAD/tvr/models/modeling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bab52ac82134c899"}},{"code_sha256_prefix":"95fdd52bfeffad7d","entry":"_download","repo":"jpthu17/dicosa","repo_kind":"listed","path":"tvr/models/modeling.py","file_url":"https://github.com/jpthu17/dicosa/blob/HEAD/tvr/models/modeling.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"95fdd52bfeffad7d"}},{"code_sha256_prefix":"3a4824e9630d22f9","entry":"available_models","repo":"jpthu17/dicosa","repo_kind":"listed","path":"tvr/models/modeling.py","file_url":"https://github.com/jpthu17/dicosa/blob/HEAD/tvr/models/modeling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3a4824e9630d22f9"}},{"code_sha256_prefix":"aacf9b08feb3ea59","entry":"CLIP","repo":"jpthu17/dicosa","repo_kind":"listed","path":"tvr/models/modeling.py","file_url":"https://github.com/jpthu17/dicosa/blob/HEAD/tvr/models/modeling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aacf9b08feb3ea59"}},{"code_sha256_prefix":"d285823632f22684","entry":"DiCoSA","repo":"jpthu17/dicosa","repo_kind":"listed","path":"tvr/models/modeling.py","file_url":"https://github.com/jpthu17/dicosa/blob/HEAD/tvr/models/modeling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d285823632f22684"}},{"code_sha256_prefix":"3fe3a6ee59bad490","entry":"EMCL","repo":"jpthu17/emcl","repo_kind":"listed","path":"video_retrieval/EMCL-Net/tvr/models/modeling.py","file_url":"https://github.com/jpthu17/emcl/blob/HEAD/video_retrieval/EMCL-Net/tvr/models/modeling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3fe3a6ee59bad490"}},{"code_sha256_prefix":"56babc05b4c255dd","entry":"convert_weights","repo":"jpthu17/dicosa","repo_kind":"listed","path":"tvr/models/modeling.py","file_url":"https://github.com/jpthu17/dicosa/blob/HEAD/tvr/models/modeling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"56babc05b4c255dd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}