{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-joint-sequence-fusion-model-for-video","title":"A Joint Sequence Fusion Model for Video Question Answering and Retrieval","arxiv_id":"1808.02559","date":"2018-08-07","proceeding":"ECCV 2018 9","authors":["Youngjae Yu","Jongseok Kim","Gunhee Kim"],"abstract":"We present an approach named JSFusion (Joint Sequence Fusion) that can\nmeasure semantic similarity between any pairs of multimodal sequence data (e.g.\na video clip and a language sentence). Our multimodal matching network consists\nof two key components. First, the Joint Semantic Tensor composes a dense\npairwise representation of two sequence data into a 3D tensor. Then, the\nConvolutional Hierarchical Decoder computes their similarity score by\ndiscovering hidden hierarchical matches between the two sequence modalities.\nBoth modules leverage hierarchical attention mechanisms that learn to promote\nwell-matched representation patterns while prune out misaligned ones in a\nbottom-up manner. Although the JSFusion is a universal model to be applicable\nto any multimodal sequence data, this work focuses on video-language tasks\nincluding multimodal retrieval and video QA. We evaluate the JSFusion model in\nthree retrieval and VQA tasks in LSMDC, for which our model achieves the best\nperformance reported so far. We also perform multiple-choice and movie\nretrieval tasks for the MSR-VTT dataset, on which our approach outperforms many\nstate-of-the-art methods.","url_abs":"http://arxiv.org/abs/1808.02559v1","url_pdf":"http://arxiv.org/pdf/1808.02559v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-joint-sequence-fusion-model-for-video","repo_url":"https://github.com/antoine77340/howto100m","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"a-joint-sequence-fusion-model-for-video","repo_url":"https://github.com/ruc-aimc-lab/nt2vr","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"multiple-choice","task_name":"Multiple-choice"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"semantic-similarity","task_name":"Semantic Similarity"},{"task_slug":"semantic-textual-similarity","task_name":"Semantic Textual Similarity"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"video-question-answering","task_name":"Video Question Answering"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[],"datasets_introduced":[{"slug":"msrvtt-mc","name":"MSRVTT-MC","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-retrieval-on-lsmdc","task":"Video Retrieval","dataset":"LSMDC","model":"JSFusion","rank_in_archive_order":34,"of":38,"metrics":{"text-to-video Median Rank":"36","text-to-video R@1":"9.1","text-to-video R@10":"34.1","text-to-video R@5":"21.2"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-msr-vtt","task":"Video Retrieval","dataset":"MSR-VTT","model":"JSFusion","rank_in_archive_order":36,"of":40,"metrics":{"text-to-video Median Rank":"13","text-to-video R@1":"10.2","text-to-video R@10":"43.2","video-to-text R@5":"31.2"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-msr-vtt-1ka","task":"Video Retrieval","dataset":"MSR-VTT-1kA","model":"JSFusion","rank_in_archive_order":59,"of":63,"metrics":{"text-to-video Median Rank":"13","text-to-video R@1":"10.2","text-to-video R@10":"43.2","text-to-video R@5":"31.2"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1808.02559","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1808.02559"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ruc-aimc-lab/nt2vr","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/antoine77340/howto100m","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"unverified":1},"by_repo_kind":{"listed":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d59b4b3c004c31ff","entry":"compute_metrics","repo":"antoine77340/howto100m","repo_kind":"listed","path":"metrics.py","file_url":"https://github.com/antoine77340/howto100m/blob/HEAD/metrics.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d59b4b3c004c31ff"}},{"code_sha256_prefix":"0554231c8f070006","entry":"get_args","repo":"antoine77340/howto100m","repo_kind":"listed","path":"args.py","file_url":"https://github.com/antoine77340/howto100m/blob/HEAD/args.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0554231c8f070006"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}