{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/neural-methods-for-point-wise-dependency","title":"Neural Methods for Point-wise Dependency Estimation","arxiv_id":"2006.05553","date":"2020-06-09","proceeding":"NeurIPS 2020 12","authors":["Yao-Hung Hubert Tsai","Han Zhao","Makoto Yamada","Louis-Philippe Morency","Ruslan Salakhutdinov"],"abstract":"Since its inception, the neural estimation of mutual information (MI) has demonstrated the empirical success of modeling expected dependency between high-dimensional random variables. However, MI is an aggregate statistic and cannot be used to measure point-wise dependency between different events. In this work, instead of estimating the expected dependency, we focus on estimating point-wise dependency (PD), which quantitatively measures how likely two outcomes co-occur. We show that we can naturally obtain PD when we are optimizing MI neural variational bounds. However, optimizing these bounds is challenging due to its large variance in practice. To address this issue, we develop two methods (free of optimizing MI variational bounds): Probabilistic Classifier and Density-Ratio Fitting. We demonstrate the effectiveness of our approaches in 1) MI estimation, 2) self-supervised representation learning, and 3) cross-modal retrieval task.","url_abs":"https://arxiv.org/abs/2006.05553v4","url_pdf":"https://arxiv.org/pdf/2006.05553v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"neural-methods-for-point-wise-dependency","repo_url":"https://github.com/yaohungt/Pointwise_Dependency_Neural_Estimation","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"cross-modal-retrieval","task_name":"Cross-Modal Retrieval"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2006.05553","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.05553"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yaohungt/Pointwise_Dependency_Neural_Estimation","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_violates":2,"ran_fixture":1,"ran_draft_wrong":1,"unverified":12},"by_repo_kind":{"official":{"samples":16,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c7cf2e57c9b57b27","entry":"mi_to_rho","repo":"yaohungt/Pointwise_Dependency_Neural_Estimation","repo_kind":"official","path":"MI_Est_and_CrossModal/src/utils.py","file_url":"https://github.com/yaohungt/Pointwise_Dependency_Neural_Estimation/blob/HEAD/MI_Est_and_CrossModal/src/utils.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c7cf2e57c9b57b27"}},{"code_sha256_prefix":"25d792e8c00de62b","entry":"mlp","repo":"yaohungt/Pointwise_Dependency_Neural_Estimation","repo_kind":"official","path":"MI_Est_and_CrossModal/src/models.py","file_url":"https://github.com/yaohungt/Pointwise_Dependency_Neural_Estimation/blob/HEAD/MI_Est_and_CrossModal/src/models.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"25d792e8c00de62b"}},{"code_sha256_prefix":"e26f4d70904fa7e6","entry":"rho_to_mi","repo":"yaohungt/Pointwise_Dependency_Neural_Estimation","repo_kind":"official","path":"MI_Est_and_CrossModal/src/utils.py","file_url":"https://github.com/yaohungt/Pointwise_Dependency_Neural_Estimation/blob/HEAD/MI_Est_and_CrossModal/src/utils.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e26f4d70904fa7e6"}},{"code_sha256_prefix":"93154000bc1f766f","entry":"tanh_clip","repo":"yaohungt/Pointwise_Dependency_Neural_Estimation","repo_kind":"official","path":"RepreLearn_Deep/costs.py","file_url":"https://github.com/yaohungt/Pointwise_Dependency_Neural_Estimation/blob/HEAD/RepreLearn_Deep/costs.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"93154000bc1f766f"}},{"code_sha256_prefix":"44031b849f6409b6","entry":"get_dataset","repo":"yaohungt/Pointwise_Dependency_Neural_Estimation","repo_kind":"official","path":"RepreLearn_Deep/datasets.py","file_url":"https://github.com/yaohungt/Pointwise_Dependency_Neural_Estimation/blob/HEAD/RepreLearn_Deep/datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"44031b849f6409b6"}},{"code_sha256_prefix":"13b4b14bea7e35f4","entry":"get_encoder_size","repo":"yaohungt/Pointwise_Dependency_Neural_Estimation","repo_kind":"official","path":"RepreLearn_Deep/datasets.py","file_url":"https://github.com/yaohungt/Pointwise_Dependency_Neural_Estimation/blob/HEAD/RepreLearn_Deep/datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"13b4b14bea7e35f4"}},{"code_sha256_prefix":"68d2b5229235b340","entry":"initialize","repo":"yaohungt/Pointwise_Dependency_Neural_Estimation","repo_kind":"official","path":"RepreLearn_Deep/mixed_precision.py","file_url":"https://github.com/yaohungt/Pointwise_Dependency_Neural_Estimation/blob/HEAD/RepreLearn_Deep/mixed_precision.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"68d2b5229235b340"}},{"code_sha256_prefix":"73e11d9bc93dd5dc","entry":"js_fgan_lower_bound_obj","repo":"yaohungt/Pointwise_Dependency_Neural_Estimation","repo_kind":"official","path":"MI_Est_and_CrossModal/src/estimators.py","file_url":"https://github.com/yaohungt/Pointwise_Dependency_Neural_Estimation/blob/HEAD/MI_Est_and_CrossModal/src/estimators.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"73e11d9bc93dd5dc"}},{"code_sha256_prefix":"6812721fcf21c64a","entry":"loss_xent","repo":"yaohungt/Pointwise_Dependency_Neural_Estimation","repo_kind":"official","path":"RepreLearn_Deep/costs.py","file_url":"https://github.com/yaohungt/Pointwise_Dependency_Neural_Estimation/blob/HEAD/RepreLearn_Deep/costs.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6812721fcf21c64a"}},{"code_sha256_prefix":"7ef596c2eaa1fed2","entry":"maybe_half","repo":"yaohungt/Pointwise_Dependency_Neural_Estimation","repo_kind":"official","path":"RepreLearn_Deep/mixed_precision.py","file_url":"https://github.com/yaohungt/Pointwise_Dependency_Neural_Estimation/blob/HEAD/RepreLearn_Deep/mixed_precision.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7ef596c2eaa1fed2"}},{"code_sha256_prefix":"99cb11cbb8338f7a","entry":"mine_lower_bound","repo":"yaohungt/Pointwise_Dependency_Neural_Estimation","repo_kind":"official","path":"MI_Est_and_CrossModal/src/estimators.py","file_url":"https://github.com/yaohungt/Pointwise_Dependency_Neural_Estimation/blob/HEAD/MI_Est_and_CrossModal/src/estimators.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"99cb11cbb8338f7a"}},{"code_sha256_prefix":"16a681df6830e17c","entry":"nwj_lower_bound_obj","repo":"yaohungt/Pointwise_Dependency_Neural_Estimation","repo_kind":"official","path":"MI_Est_and_CrossModal/src/estimators.py","file_url":"https://github.com/yaohungt/Pointwise_Dependency_Neural_Estimation/blob/HEAD/MI_Est_and_CrossModal/src/estimators.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"16a681df6830e17c"}},{"code_sha256_prefix":"10d196b211843339","entry":"parse_devkit","repo":"yaohungt/Pointwise_Dependency_Neural_Estimation","repo_kind":"official","path":"RepreLearn_Deep/FastAutoAugment/imagenet.py","file_url":"https://github.com/yaohungt/Pointwise_Dependency_Neural_Estimation/blob/HEAD/RepreLearn_Deep/FastAutoAugment/imagenet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"10d196b211843339"}},{"code_sha256_prefix":"97bfaa2cce837d67","entry":"parse_meta","repo":"yaohungt/Pointwise_Dependency_Neural_Estimation","repo_kind":"official","path":"RepreLearn_Deep/FastAutoAugment/imagenet.py","file_url":"https://github.com/yaohungt/Pointwise_Dependency_Neural_Estimation/blob/HEAD/RepreLearn_Deep/FastAutoAugment/imagenet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"97bfaa2cce837d67"}},{"code_sha256_prefix":"e7825a1705689714","entry":"parse_val_groundtruth","repo":"yaohungt/Pointwise_Dependency_Neural_Estimation","repo_kind":"official","path":"RepreLearn_Deep/FastAutoAugment/imagenet.py","file_url":"https://github.com/yaohungt/Pointwise_Dependency_Neural_Estimation/blob/HEAD/RepreLearn_Deep/FastAutoAugment/imagenet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7825a1705689714"}},{"code_sha256_prefix":"5aa907d081bca44c","entry":"sample_correlated_gaussian","repo":"yaohungt/Pointwise_Dependency_Neural_Estimation","repo_kind":"official","path":"MI_Est_and_CrossModal/src/utils.py","file_url":"https://github.com/yaohungt/Pointwise_Dependency_Neural_Estimation/blob/HEAD/MI_Est_and_CrossModal/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5aa907d081bca44c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}