{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2510-13212","title":"Towards Understanding Valuable Preference Data for Large Language Model Alignment","arxiv_id":"2510.13212","date":"2025-10-15","proceeding":"ICLR","authors":["Zizhuo Zhang","Qizhou Wang","Shanshan Ye","Jianing Zhu","Jiangchao Yao","Bo Han","Masashi Sugiyama"],"abstract":"Large language model (LLM) alignment is typically achieved through learning from human preference comparisons, making the quality of preference data critical to its success. Existing studies often pre-process raw training datasets to identify valuable preference pairs using external reward models or off-the-shelf LLMs, achieving improved overall performance but rarely examining whether individual, selected data point is genuinely beneficial. We assess data quality through individual influence on validation data using our newly proposed truncated influence function (TIF), which mitigates the over-scoring present in traditional measures and reveals that preference data quality is inherently a property of the model. In other words, a data pair that benefits one model may harm another. This leaves the need to improve the preference data selection approaches to be adapting to specific models. To this end, we introduce two candidate scoring functions (SFs) that are computationally simpler than TIF and positively correlated with it. They are also model dependent and can serve as potential indicators of individual data quality for preference data selection. Furthermore, we observe that these SFs inherently exhibit errors when compared to TIF. To this end, we combine them to offset their diverse error sources, resulting in a simple yet effective data selection rule that enables the models to achieve a more precise selection of valuable preference data. We conduct experiments across diverse alignment benchmarks and various LLM families, with results demonstrating that better alignment performance can be achieved using less data, showing the generality of our findings and new methods. Our code is publicly available at https://github.com/tmlr-group/TIF_LossDiff-IRM.","url_abs":"https://arxiv.org/abs/2510.13212","url_pdf":"https://arxiv.org/pdf/2510.13212","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2510.13212","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2510.13212"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/tmlr-group/TIF_LossDiff-IRM","reach":{"status":"ok"}}],"summary":{"unverified":21},"by_repo_kind":{"found_in_text":{"samples":21,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":21,"samples":[{"code_sha256_prefix":"8e680b35d845de28","entry":"build_messages","repo":"tmlr-group/TIF_LossDiff-IRM","repo_kind":"found_in_text","path":"winrate_eval/single_score_local.py","file_url":"https://github.com/tmlr-group/TIF_LossDiff-IRM/blob/HEAD/winrate_eval/single_score_local.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8e680b35d845de28"}},{"code_sha256_prefix":"ae9d6994bbaf2c28","entry":"compute_dpo_loss","repo":"tmlr-group/TIF_LossDiff-IRM","repo_kind":"found_in_text","path":"influence/analyze_loss_diff.py","file_url":"https://github.com/tmlr-group/TIF_LossDiff-IRM/blob/HEAD/influence/analyze_loss_diff.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ae9d6994bbaf2c28"}},{"code_sha256_prefix":"87d4a4018ae927d4","entry":"compute_dpo_loss","repo":"tmlr-group/TIF_LossDiff-IRM","repo_kind":"found_in_text","path":"influence/analyze_slic_loss_diff.py","file_url":"https://github.com/tmlr-group/TIF_LossDiff-IRM/blob/HEAD/influence/analyze_slic_loss_diff.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"87d4a4018ae927d4"}},{"code_sha256_prefix":"833ac53ebd095a26","entry":"compute_dpo_losses","repo":"tmlr-group/TIF_LossDiff-IRM","repo_kind":"found_in_text","path":"analysis/lossdiff.py","file_url":"https://github.com/tmlr-group/TIF_LossDiff-IRM/blob/HEAD/analysis/lossdiff.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"833ac53ebd095a26"}},{"code_sha256_prefix":"39cbc907eb087c14","entry":"compute_slice_indices","repo":"tmlr-group/TIF_LossDiff-IRM","repo_kind":"found_in_text","path":"analysis/data_select_by2_mid_k.py","file_url":"https://github.com/tmlr-group/TIF_LossDiff-IRM/blob/HEAD/analysis/data_select_by2_mid_k.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"39cbc907eb087c14"}},{"code_sha256_prefix":"422c10cc1340815b","entry":"compute_slice_indices","repo":"tmlr-group/TIF_LossDiff-IRM","repo_kind":"found_in_text","path":"analysis/data_select_mid_k.py","file_url":"https://github.com/tmlr-group/TIF_LossDiff-IRM/blob/HEAD/analysis/data_select_mid_k.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"422c10cc1340815b"}},{"code_sha256_prefix":"46e5b32f4f287fa8","entry":"filter_middle_percent","repo":"tmlr-group/TIF_LossDiff-IRM","repo_kind":"found_in_text","path":"influence/filter_lossdiff_vs_rm_v1.py","file_url":"https://github.com/tmlr-group/TIF_LossDiff-IRM/blob/HEAD/influence/filter_lossdiff_vs_rm_v1.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"46e5b32f4f287fa8"}},{"code_sha256_prefix":"f688c7a26a24d651","entry":"get_by_path","repo":"tmlr-group/TIF_LossDiff-IRM","repo_kind":"found_in_text","path":"analysis/correlation_IF_lossdiff.py","file_url":"https://github.com/tmlr-group/TIF_LossDiff-IRM/blob/HEAD/analysis/correlation_IF_lossdiff.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f688c7a26a24d651"}},{"code_sha256_prefix":"7bbf7a5207c682b0","entry":"load_dpo_dataset","repo":"tmlr-group/TIF_LossDiff-IRM","repo_kind":"found_in_text","path":"analysis/lossdiff.py","file_url":"https://github.com/tmlr-group/TIF_LossDiff-IRM/blob/HEAD/analysis/lossdiff.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7bbf7a5207c682b0"}},{"code_sha256_prefix":"1976064be25085f7","entry":"load_dpo_dataset","repo":"tmlr-group/TIF_LossDiff-IRM","repo_kind":"found_in_text","path":"influence/analyze_loss_diff.py","file_url":"https://github.com/tmlr-group/TIF_LossDiff-IRM/blob/HEAD/influence/analyze_loss_diff.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1976064be25085f7"}},{"code_sha256_prefix":"80c41f1da6f9e664","entry":"load_loss_diff_data","repo":"tmlr-group/TIF_LossDiff-IRM","repo_kind":"found_in_text","path":"influence/filter_lossdiff_vs_rm_v1.py","file_url":"https://github.com/tmlr-group/TIF_LossDiff-IRM/blob/HEAD/influence/filter_lossdiff_vs_rm_v1.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"80c41f1da6f9e664"}},{"code_sha256_prefix":"deeaaed5a48fb372","entry":"parse_rating","repo":"tmlr-group/TIF_LossDiff-IRM","repo_kind":"found_in_text","path":"winrate_eval/single_score.py","file_url":"https://github.com/tmlr-group/TIF_LossDiff-IRM/blob/HEAD/winrate_eval/single_score.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"deeaaed5a48fb372"}},{"code_sha256_prefix":"8b89fc7d1a1f608a","entry":"parse_rating","repo":"tmlr-group/TIF_LossDiff-IRM","repo_kind":"found_in_text","path":"winrate_eval/single_score_local.py","file_url":"https://github.com/tmlr-group/TIF_LossDiff-IRM/blob/HEAD/winrate_eval/single_score_local.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8b89fc7d1a1f608a"}},{"code_sha256_prefix":"a65ff6fa4d9d7cc7","entry":"read_jsonl","repo":"tmlr-group/TIF_LossDiff-IRM","repo_kind":"found_in_text","path":"winrate_eval/single_score.py","file_url":"https://github.com/tmlr-group/TIF_LossDiff-IRM/blob/HEAD/winrate_eval/single_score.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a65ff6fa4d9d7cc7"}},{"code_sha256_prefix":"07adb23040d2275c","entry":"read_jsonl","repo":"tmlr-group/TIF_LossDiff-IRM","repo_kind":"found_in_text","path":"winrate_eval/single_score_local.py","file_url":"https://github.com/tmlr-group/TIF_LossDiff-IRM/blob/HEAD/winrate_eval/single_score_local.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"07adb23040d2275c"}},{"code_sha256_prefix":"17f3dfa5c8d12d87","entry":"read_jsonl","repo":"tmlr-group/TIF_LossDiff-IRM","repo_kind":"found_in_text","path":"analysis/data_select_by2_mid_k.py","file_url":"https://github.com/tmlr-group/TIF_LossDiff-IRM/blob/HEAD/analysis/data_select_by2_mid_k.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"17f3dfa5c8d12d87"}},{"code_sha256_prefix":"e2a67da3a04cd3c7","entry":"read_jsonl","repo":"tmlr-group/TIF_LossDiff-IRM","repo_kind":"found_in_text","path":"analysis/data_select_mid_k.py","file_url":"https://github.com/tmlr-group/TIF_LossDiff-IRM/blob/HEAD/analysis/data_select_mid_k.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e2a67da3a04cd3c7"}},{"code_sha256_prefix":"f06ea4ab38142076","entry":"read_values","repo":"tmlr-group/TIF_LossDiff-IRM","repo_kind":"found_in_text","path":"analysis/correlation_IF_lossdiff.py","file_url":"https://github.com/tmlr-group/TIF_LossDiff-IRM/blob/HEAD/analysis/correlation_IF_lossdiff.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f06ea4ab38142076"}},{"code_sha256_prefix":"57554c79159f92df","entry":"to_float","repo":"tmlr-group/TIF_LossDiff-IRM","repo_kind":"found_in_text","path":"analysis/data_select_by2_mid_k.py","file_url":"https://github.com/tmlr-group/TIF_LossDiff-IRM/blob/HEAD/analysis/data_select_by2_mid_k.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"57554c79159f92df"}},{"code_sha256_prefix":"43485be04415bd36","entry":"to_float_or_none","repo":"tmlr-group/TIF_LossDiff-IRM","repo_kind":"found_in_text","path":"analysis/correlation_IF_lossdiff.py","file_url":"https://github.com/tmlr-group/TIF_LossDiff-IRM/blob/HEAD/analysis/correlation_IF_lossdiff.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"43485be04415bd36"}},{"code_sha256_prefix":"9b8c4e63551b7168","entry":"try_get_float","repo":"tmlr-group/TIF_LossDiff-IRM","repo_kind":"found_in_text","path":"analysis/data_select_mid_k.py","file_url":"https://github.com/tmlr-group/TIF_LossDiff-IRM/blob/HEAD/analysis/data_select_mid_k.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9b8c4e63551b7168"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_api"},"syntology_extracted_results":null}