{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ensemble-of-mrr-and-ndcg-models-for-visual","title":"Ensemble of MRR and NDCG models for Visual Dialog","arxiv_id":"2104.07511","date":"2021-04-15","proceeding":"NAACL 2021 4","authors":["Idan Schwartz"],"abstract":"Assessing an AI agent that can converse in human language and understand visual content is challenging. Generation metrics, such as BLEU scores favor correct syntax over semantics. Hence a discriminative approach is often used, where an agent ranks a set of candidate options. The mean reciprocal rank (MRR) metric evaluates the model performance by taking into account the rank of a single human-derived answer. This approach, however, raises a new challenge: the ambiguity and synonymy of answers, for instance, semantic equivalence (e.g., `yeah' and `yes'). To address this, the normalized discounted cumulative gain (NDCG) metric has been used to capture the relevance of all the correct answers via dense annotations. However, the NDCG metric favors the usually applicable uncertain answers such as `I don't know. Crafting a model that excels on both MRR and NDCG metrics is challenging. Ideally, an AI agent should answer a human-like reply and validate the correctness of any answer. To address this issue, we describe a two-step non-parametric ranking approach that can merge strong MRR and NDCG models. Using our approach, we manage to keep most MRR state-of-the-art performance (70.41% vs. 71.24%) and the NDCG state-of-the-art performance (72.16% vs. 75.35%). Moreover, our approach won the recent Visual Dialog 2020 challenge. Source code is available at https://github.com/idansc/mrr-ndcg.","url_abs":"https://arxiv.org/abs/2104.07511v3","url_pdf":"https://arxiv.org/pdf/2104.07511v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ensemble-of-mrr-and-ndcg-models-for-visual","repo_url":"https://github.com/idansc/mrr-ndcg","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"ai-agent","task_name":"AI Agent"},{"task_slug":"visual-dialogue","task_name":"Visual Dialog"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-dialog-on-visdial-v10-test-std","task":"Visual Dialog","dataset":"VisDial v1.0 test-std","model":"5xFGA + LS*+","rank_in_archive_order":1,"of":3,"metrics":{"MRR":"0.7124","Mean Rank":"2.96","R@1":"58.28","R@10":"94.45","R@5":"87.55"},"uses_additional_data":false},{"leaderboard":"/sota/visual-dialog-on-visdial-v10-test-std","task":"Visual Dialog","dataset":"VisDial v1.0 test-std","model":"Two-Step","rank_in_archive_order":2,"of":3,"metrics":{"MRR":"0.7041","Mean Rank":"3.66","NDCG":"72.16","R@1":"58.18","R@10":"90.83","R@5":"83.85"},"uses_additional_data":false},{"leaderboard":"/sota/visual-dialog-on-visdial-v10-test-std","task":"Visual Dialog","dataset":"VisDial v1.0 test-std","model":"5xFGA + LS","rank_in_archive_order":3,"of":3,"metrics":{"NDCG":"64.04"},"uses_additional_data":false},{"leaderboard":"/sota/visual-dialog-on-visual-dialog-v1-0-test-std","task":"Visual Dialog","dataset":"Visual Dialog v1.0 test-std","model":"2 Step: Factor Graph Attention + VD-Bert","rank_in_archive_order":16,"of":80,"metrics":{"MRR (x 100)":"69.92","Mean":"3.84","NDCG (x 100)":"72.83","R@1":"58.3","R@10":"89.6","R@5":"81.55"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2104.07511","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.07511"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/idansc/mrr-ndcg","reach":null}],"summary":{"ran":1,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"ae2328a7f30a9546","entry":"NDCG","repo":"idansc/mrr-ndcg","repo_kind":"official","path":"utils.py","file_url":"https://github.com/idansc/mrr-ndcg/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ae2328a7f30a9546"}},{"code_sha256_prefix":"bba223178ea790b8","entry":"scores_to_ranks","repo":"idansc/mrr-ndcg","repo_kind":"official","path":"utils.py","file_url":"https://github.com/idansc/mrr-ndcg/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bba223178ea790b8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}