{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pointwise-mutual-information-based-metric-and","title":"Pointwise Mutual Information Based Metric and Decoding Strategy for Faithful Generation in Document Grounded Dialogs","arxiv_id":"2305.12191","date":"2023-05-20","proceeding":null,"authors":["Yatin Nandwani","Vineet Kumar","Dinesh Raghu","Sachindra Joshi","Luis A. Lastras"],"abstract":"A major concern in using deep learning based generative models for document-grounded dialogs is the potential generation of responses that are not \\textit{faithful} to the underlying document. Existing automated metrics used for evaluating the faithfulness of response with respect to the grounding document measure the degree of similarity between the generated response and the document's content. However, these automated metrics are far from being well aligned with human judgments. Therefore, to improve the measurement of faithfulness, we propose a new metric that utilizes (Conditional) Point-wise Mutual Information (PMI) between the generated response and the source document, conditioned on the dialogue. PMI quantifies the extent to which the document influences the generated response -- with a higher PMI indicating a more faithful response. We build upon this idea to create a new decoding technique that incorporates PMI into the response generation process to predict more faithful responses. Our experiments on the BEGIN benchmark demonstrate an improved correlation of our metric with human evaluation. We also show that our decoding technique is effective in generating more faithful responses when compared to standard decoding techniques on a set of publicly available document-grounded dialog datasets.","url_abs":"https://arxiv.org/abs/2305.12191v2","url_pdf":"https://arxiv.org/pdf/2305.12191v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pointwise-mutual-information-based-metric-and","repo_url":"https://github.com/ynandwan/pmi-faith","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"response-generation","task_name":"Response Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2305.12191","atlas_url":"https://app.syntology.ai/?focus=2305.12191","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.12191"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ynandwan/pmi-faith","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":9},"by_repo_kind":{"official":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b02d06c0faf674c0","entry":"concat_padded_tensors","repo":"ynandwan/pmi-faith","repo_kind":"official","path":"faithful-decode/models/modeling_nce.py","file_url":"https://github.com/ynandwan/pmi-faith/blob/HEAD/faithful-decode/models/modeling_nce.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b02d06c0faf674c0"}},{"code_sha256_prefix":"a77daed734d009e1","entry":"convert_dial_list","repo":"ynandwan/pmi-faith","repo_kind":"official","path":"faithful-decode/models/dataset.py","file_url":"https://github.com/ynandwan/pmi-faith/blob/HEAD/faithful-decode/models/dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a77daed734d009e1"}},{"code_sha256_prefix":"856adbd7f6afcfa6","entry":"get_dataset","repo":"ynandwan/pmi-faith","repo_kind":"official","path":"faithful-decode/models/dataset.py","file_url":"https://github.com/ynandwan/pmi-faith/blob/HEAD/faithful-decode/models/dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"856adbd7f6afcfa6"}},{"code_sha256_prefix":"4a463c48c18c4b42","entry":"get_huggingface_pretrained_model","repo":"ynandwan/pmi-faith","repo_kind":"official","path":"faithfulness-metrics/src/compute_faithfulness_api.py","file_url":"https://github.com/ynandwan/pmi-faith/blob/HEAD/faithfulness-metrics/src/compute_faithfulness_api.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4a463c48c18c4b42"}},{"code_sha256_prefix":"badf073462e1fd33","entry":"get_output_name","repo":"ynandwan/pmi-faith","repo_kind":"official","path":"faithful-decode/models/pmi_generate.py","file_url":"https://github.com/ynandwan/pmi-faith/blob/HEAD/faithful-decode/models/pmi_generate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"badf073462e1fd33"}},{"code_sha256_prefix":"9c6cb7a5a08fbe5d","entry":"get_tokens","repo":"ynandwan/pmi-faith","repo_kind":"official","path":"faithfulness-metrics/src/compute_faithfulness_api.py","file_url":"https://github.com/ynandwan/pmi-faith/blob/HEAD/faithfulness-metrics/src/compute_faithfulness_api.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9c6cb7a5a08fbe5d"}},{"code_sha256_prefix":"20cb1e1782f60309","entry":"masked_log_softmax","repo":"ynandwan/pmi-faith","repo_kind":"official","path":"faithful-decode/models/modeling_nce.py","file_url":"https://github.com/ynandwan/pmi-faith/blob/HEAD/faithful-decode/models/modeling_nce.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"20cb1e1782f60309"}},{"code_sha256_prefix":"2dec641c76d04d78","entry":"pad_tensor","repo":"ynandwan/pmi-faith","repo_kind":"official","path":"faithful-decode/models/modeling_nce.py","file_url":"https://github.com/ynandwan/pmi-faith/blob/HEAD/faithful-decode/models/modeling_nce.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2dec641c76d04d78"}},{"code_sha256_prefix":"eec474cc8cda034e","entry":"top_k_top_p_filtering","repo":"ynandwan/pmi-faith","repo_kind":"official","path":"faithful-decode/models/generation_utils_pmi.py","file_url":"https://github.com/ynandwan/pmi-faith/blob/HEAD/faithful-decode/models/generation_utils_pmi.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eec474cc8cda034e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}