{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/visalign-dataset-for-measuring-the-degree-of","title":"VisAlign: Dataset for Measuring the Degree of Alignment between AI and Humans in Visual Perception","arxiv_id":"2308.01525","date":"2023-08-03","proceeding":null,"authors":["Jiyoung Lee","Seungho Kim","Seunghyun Won","Joonseok Lee","Marzyeh Ghassemi","James Thorne","Jaeseok Choi","O-Kil Kwon","Edward Choi"],"abstract":"AI alignment refers to models acting towards human-intended goals, preferences, or ethical principles. Given that most large-scale deep learning models act as black boxes and cannot be manually controlled, analyzing the similarity between models and humans can be a proxy measure for ensuring AI safety. In this paper, we focus on the models' visual perception alignment with humans, further referred to as AI-human visual alignment. Specifically, we propose a new dataset for measuring AI-human visual alignment in terms of image classification, a fundamental task in machine perception. In order to evaluate AI-human visual alignment, a dataset should encompass samples with various scenarios that may arise in the real world and have gold human perception labels. Our dataset consists of three groups of samples, namely Must-Act (i.e., Must-Classify), Must-Abstain, and Uncertain, based on the quantity and clarity of visual information in an image and further divided into eight categories. All samples have a gold human perception label; even Uncertain (severely blurry) sample labels were obtained via crowd-sourcing. The validity of our dataset is verified by sampling theory, statistical theories related to survey design, and experts in the related fields. Using our dataset, we analyze the visual alignment and reliability of five popular visual perception models and seven abstention methods. Our code and data is available at https://github.com/jiyounglee-0523/VisAlign.","url_abs":"https://arxiv.org/abs/2308.01525v3","url_pdf":"https://arxiv.org/pdf/2308.01525v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"visalign-dataset-for-measuring-the-degree-of","repo_url":"https://github.com/jiyounglee-0523/visalign","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[{"slug":"visalign","name":"VisAlign","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2308.01525","atlas_url":"https://app.syntology.ai/?focus=2308.01525","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.01525"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/jiyounglee-0523/VisAlign","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jiyounglee-0523/visalign","reach":{"status":"ok"}}],"summary":{"ran":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"94ff7babd4a52bd6","entry":"to_tensor","repo":"jiyounglee-0523/VisAlign","repo_kind":"official","path":"dataset/imagenet_selfsupervised.py","file_url":"https://github.com/jiyounglee-0523/VisAlign/blob/HEAD/dataset/imagenet_selfsupervised.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"94ff7babd4a52bd6"}},{"code_sha256_prefix":"3bc5b4f052dbc620","entry":"transform_fn","repo":"jiyounglee-0523/VisAlign","repo_kind":"official","path":"dataset/imagenet.py","file_url":"https://github.com/jiyounglee-0523/VisAlign/blob/HEAD/dataset/imagenet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3bc5b4f052dbc620"}},{"code_sha256_prefix":"9ed4a397c47e5cfc","entry":"validate_dist","repo":"jiyounglee-0523/VisAlign","repo_kind":"official","path":"make_leaderboard_submission.py","file_url":"https://github.com/jiyounglee-0523/VisAlign/blob/HEAD/make_leaderboard_submission.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9ed4a397c47e5cfc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}