{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2606-02578","title":"Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling","arxiv_id":"2606.02578","date":"2026-06-01","proceeding":"ICML","authors":["Seojeong Park","Jiho Choi","Junyong Kang","Seonho Lee","Jaeyo Shin","Hyunjung Shim"],"abstract":"Recent multimodal large language models have demonstrated strong reasoning ability, yet their reliability as automated evaluators remains limited by a critical weakness: when visual evidence conflicts with textual cues, MLLM judges tend to reward plausible narratives over perceptually correct answers. We identify and systematically analyze this phenomenon, which we term Perceptual Judgment Bias. Through controlled visual perturbations, existing multimodal judges frequently anchor on the response text instead of their own visual perception, leading to inconsistent and non-verifiable evaluations. To address this issue, we introduce the Perceptually Perturbed Judgment Dataset, which constructs minimally edited counterfactual responses that isolate perceptual errors and enable verifiable supervision. Building on this dataset, we develop a unified training framework that combines a structured GRPO-based reward with a batch-ranking objective, achieving coherent global ordering without explicit pairwise labels. Experiments across diverse MLLM-as-a-Judge benchmarks show that our approach substantially improves perceptual fidelity, ranking coherence, and alignment with human evaluation. Our results establish a scalable and generalizable pathway for training multimodal judges that are perceptually grounded, interpretable, and robust to visual-reasoning conflicts.","url_abs":"https://arxiv.org/abs/2606.02578","url_pdf":"https://arxiv.org/pdf/2606.02578","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2606.02578","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2606.02578"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/kaist-cvml/perception-judge","reach":null}],"summary":{"ran":14,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":15,"ran":14,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":15,"samples":[{"code_sha256_prefix":"28820e518316f342","entry":"batch_reward_function","repo":"kaist-cvml/perception-judge","repo_kind":"found_in_text","path":"verl/perception_judge/reward_function.py","file_url":"https://github.com/kaist-cvml/perception-judge/blob/HEAD/verl/perception_judge/reward_function.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"28820e518316f342"}},{"code_sha256_prefix":"cfeb19d54f2339b3","entry":"clean_text","repo":"kaist-cvml/perception-judge","repo_kind":"found_in_text","path":"prepare-datasets/post_processing_filter_after_gen.py","file_url":"https://github.com/kaist-cvml/perception-judge/blob/HEAD/prepare-datasets/post_processing_filter_after_gen.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cfeb19d54f2339b3"}},{"code_sha256_prefix":"664cda8c4fff72d0","entry":"infer_field_mapping","repo":"kaist-cvml/perception-judge","repo_kind":"found_in_text","path":"prepare-datasets/schema.py","file_url":"https://github.com/kaist-cvml/perception-judge/blob/HEAD/prepare-datasets/schema.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"664cda8c4fff72d0"}},{"code_sha256_prefix":"036ccfc4a7996396","entry":"merge_files","repo":"kaist-cvml/perception-judge","repo_kind":"found_in_text","path":"prepare-datasets/post_processing_merge_folders.py","file_url":"https://github.com/kaist-cvml/perception-judge/blob/HEAD/prepare-datasets/post_processing_merge_folders.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"036ccfc4a7996396"}},{"code_sha256_prefix":"2b85173eac27fd50","entry":"normalize_answer","repo":"kaist-cvml/perception-judge","repo_kind":"found_in_text","path":"prepare-datasets/utils.py","file_url":"https://github.com/kaist-cvml/perception-judge/blob/HEAD/prepare-datasets/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2b85173eac27fd50"}},{"code_sha256_prefix":"c64c86c18c70fb68","entry":"normalize_record","repo":"kaist-cvml/perception-judge","repo_kind":"found_in_text","path":"prepare-datasets/schema.py","file_url":"https://github.com/kaist-cvml/perception-judge/blob/HEAD/prepare-datasets/schema.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c64c86c18c70fb68"}},{"code_sha256_prefix":"667041a05d97cfae","entry":"pair_reward_function","repo":"kaist-cvml/perception-judge","repo_kind":"found_in_text","path":"verl/perception_judge/reward_function.py","file_url":"https://github.com/kaist-cvml/perception-judge/blob/HEAD/verl/perception_judge/reward_function.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"667041a05d97cfae"}},{"code_sha256_prefix":"3a81d8857ecf1f74","entry":"parse_json_response","repo":"kaist-cvml/perception-judge","repo_kind":"found_in_text","path":"prepare-datasets/processing_async.py","file_url":"https://github.com/kaist-cvml/perception-judge/blob/HEAD/prepare-datasets/processing_async.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3a81d8857ecf1f74"}},{"code_sha256_prefix":"8af8334abe20a4bb","entry":"parse_perception_attrs","repo":"kaist-cvml/perception-judge","repo_kind":"found_in_text","path":"prepare-datasets/processing_async.py","file_url":"https://github.com/kaist-cvml/perception-judge/blob/HEAD/prepare-datasets/processing_async.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8af8334abe20a4bb"}},{"code_sha256_prefix":"5994eb7534d88144","entry":"parse_verify_result","repo":"kaist-cvml/perception-judge","repo_kind":"found_in_text","path":"prepare-datasets/processing_async.py","file_url":"https://github.com/kaist-cvml/perception-judge/blob/HEAD/prepare-datasets/processing_async.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5994eb7534d88144"}},{"code_sha256_prefix":"9e9e9289a06c8157","entry":"process_file","repo":"kaist-cvml/perception-judge","repo_kind":"found_in_text","path":"prepare-datasets/post_processing_filter_after_gen.py","file_url":"https://github.com/kaist-cvml/perception-judge/blob/HEAD/prepare-datasets/post_processing_filter_after_gen.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9e9e9289a06c8157"}},{"code_sha256_prefix":"8d0040b5ee433fc9","entry":"process_row","repo":"kaist-cvml/perception-judge","repo_kind":"found_in_text","path":"prepare-datasets/post_processing_filter_after_gen.py","file_url":"https://github.com/kaist-cvml/perception-judge/blob/HEAD/prepare-datasets/post_processing_filter_after_gen.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8d0040b5ee433fc9"}},{"code_sha256_prefix":"efd80dae2d3566c3","entry":"run_async","repo":"kaist-cvml/perception-judge","repo_kind":"found_in_text","path":"prepare-datasets/client_async.py","file_url":"https://github.com/kaist-cvml/perception-judge/blob/HEAD/prepare-datasets/client_async.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"efd80dae2d3566c3"}},{"code_sha256_prefix":"8957a3e7937a6f2c","entry":"select_chosen","repo":"kaist-cvml/perception-judge","repo_kind":"found_in_text","path":"prepare-datasets/generate_rejected_from_score.py","file_url":"https://github.com/kaist-cvml/perception-judge/blob/HEAD/prepare-datasets/generate_rejected_from_score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8957a3e7937a6f2c"}},{"code_sha256_prefix":"2d6a14cf6379542d","entry":"read_jsonl","repo":"kaist-cvml/perception-judge","repo_kind":"found_in_text","path":"prepare-datasets/post_processing_merge_folders.py","file_url":"https://github.com/kaist-cvml/perception-judge/blob/HEAD/prepare-datasets/post_processing_merge_folders.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2d6a14cf6379542d"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CV","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}