{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-complete-decomposition-of-kl-error-using","title":"A Complete Decomposition of KL Error using Refined Information and Mode Interaction Selection","arxiv_id":"2410.11964","date":"2024-10-15","proceeding":null,"authors":["James Enouen","Mahito Sugiyama"],"abstract":"The log-linear model has received a significant amount of theoretical attention in previous decades and remains the fundamental tool used for learning probability distributions over discrete variables. Despite its large popularity in statistical mechanics and high-dimensional statistics, the vast majority of such energy-based modeling approaches only focus on the two-variable relationships, such as Boltzmann machines and Markov graphical models. Although these approaches have easier-to-solve structure learning problems and easier-to-optimize parametric distributions, they often ignore the rich structure which exists in the higher-order interactions between different variables. Using more recent tools from the field of information geometry, we revisit the classical formulation of the log-linear model with a focus on higher-order mode interactions, going beyond the 1-body modes of independent distributions and the 2-body modes of Boltzmann distributions. This perspective allows us to define a complete decomposition of the KL error. This then motivates the formulation of a sparse selection problem over the set of possible mode interactions. In the same way as sparse graph selection allows for better generalization, we find that our learned distributions are able to more efficiently use the finite amount of data which is available in practice. On both synthetic and real-world datasets, we demonstrate our algorithm's effectiveness in maximizing the log-likelihood for the generative task and also the ease of adaptability to the discriminative task of classification.","url_abs":"https://arxiv.org/abs/2410.11964v1","url_pdf":"https://arxiv.org/pdf/2410.11964v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-complete-decomposition-of-kl-error-using","repo_url":"https://github.com/EnouenJ/mode-attributing-hierarchy","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"focus","method_name":"Focus"},{"method_slug":"set","method_name":"SET"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.11964","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.11964"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/EnouenJ/mode-attributing-hierarchy","reach":{"status":"ok"}}],"summary":{"ran":5,"unverified":5},"by_repo_kind":{"official":{"samples":10,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"571d357711340c7f","entry":"KL","repo":"EnouenJ/mode-attributing-hierarchy","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/EnouenJ/mode-attributing-hierarchy/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"571d357711340c7f"}},{"code_sha256_prefix":"a465da79d2f5d42f","entry":"get_cumulative_index_sizes","repo":"EnouenJ/mode-attributing-hierarchy","repo_kind":"official","path":"datasets/dataset_utils.py","file_url":"https://github.com/EnouenJ/mode-attributing-hierarchy/blob/HEAD/datasets/dataset_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a465da79d2f5d42f"}},{"code_sha256_prefix":"20f9fde9cbfdbe18","entry":"index_sizes_from_event_dict","repo":"EnouenJ/mode-attributing-hierarchy","repo_kind":"official","path":"datasets/dataset_utils.py","file_url":"https://github.com/EnouenJ/mode-attributing-hierarchy/blob/HEAD/datasets/dataset_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"20f9fde9cbfdbe18"}},{"code_sha256_prefix":"29478703469dab21","entry":"oneless","repo":"EnouenJ/mode-attributing-hierarchy","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/EnouenJ/mode-attributing-hierarchy/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"29478703469dab21"}},{"code_sha256_prefix":"fb9020626aee2101","entry":"powerset","repo":"EnouenJ/mode-attributing-hierarchy","repo_kind":"official","path":"models.py","file_url":"https://github.com/EnouenJ/mode-attributing-hierarchy/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fb9020626aee2101"}},{"code_sha256_prefix":"690a1132df324307","entry":"build_nonstrict_frontier","repo":"EnouenJ/mode-attributing-hierarchy","repo_kind":"official","path":"utils/MIS_utils.py","file_url":"https://github.com/EnouenJ/mode-attributing-hierarchy/blob/HEAD/utils/MIS_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"690a1132df324307"}},{"code_sha256_prefix":"c4eab9ac4a12761f","entry":"build_semistrict_frontier","repo":"EnouenJ/mode-attributing-hierarchy","repo_kind":"official","path":"utils/MIS_utils.py","file_url":"https://github.com/EnouenJ/mode-attributing-hierarchy/blob/HEAD/utils/MIS_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c4eab9ac4a12761f"}},{"code_sha256_prefix":"f77b8898c144ec89","entry":"constructWeakHigherInteractions","repo":"EnouenJ/mode-attributing-hierarchy","repo_kind":"official","path":"utils/MIS_utils.py","file_url":"https://github.com/EnouenJ/mode-attributing-hierarchy/blob/HEAD/utils/MIS_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f77b8898c144ec89"}},{"code_sha256_prefix":"70a2f7a8dd235c5f","entry":"powerset","repo":"EnouenJ/mode-attributing-hierarchy","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/EnouenJ/mode-attributing-hierarchy/blob/HEAD/utils/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"70a2f7a8dd235c5f"}},{"code_sha256_prefix":"b4f39a6a8f5d8ae5","entry":"read_categorical_only_from_CSV","repo":"EnouenJ/mode-attributing-hierarchy","repo_kind":"official","path":"datasets/dataset_utils.py","file_url":"https://github.com/EnouenJ/mode-attributing-hierarchy/blob/HEAD/datasets/dataset_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b4f39a6a8f5d8ae5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}