{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2608-11947","title":"Accuracy and Order Sensitivity Diverge Under Label-Free Strategies","arxiv_id":"2608.11947","date":"2026-08-12","proceeding":null,"authors":["Karl Hanna","Chen Feng"],"abstract":"Multiple-choice benchmarks are widely used to evaluate large language models, but MCQ scores conflate knowledge with sensitivity to option order, which makes them unreliable measures of model knowledge. In this paper, we test whether preventing a model from seeing option labels while committing to an answer removes positional influence and, in turn, improves performance. We evaluate two different strategies for mitigating bias. The first uses a generation-then-matching approach, and the second scores options in isolation, which is positionally unbiased by construction. Neither reliably improves accuracy. A complete decomposition shows that the bottleneck is withholding options, not the matching step. The only configuration that consistently matches the baseline is the one that shows the model all options paired with an LLM matcher. However, eliminating positional influence entirely still does not reliably yield accuracy gains, while cyclic permutation often improves them. For two-stage prompting, an aggregate measure of recall imbalance and a direct per-question measure of order sensitivity both fail to show reliable debiasing.","url_abs":"https://arxiv.org/abs/2608.11947","url_pdf":"https://arxiv.org/pdf/2608.11947","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2608.11947","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2608.11947"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/cotenthusiast/choicebench","reach":null}],"summary":{"ran":5,"unverified":3},"by_repo_kind":{"found_in_text":{"samples":8,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"31c8d02fd22424ff","entry":"is_credential_key","repo":"cotenthusiast/choicebench","repo_kind":"found_in_text","path":"src/choicebench/identity.py","file_url":"https://github.com/cotenthusiast/choicebench/blob/HEAD/src/choicebench/identity.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"31c8d02fd22424ff"}},{"code_sha256_prefix":"ace6e305c009562c","entry":"is_secret_key","repo":"cotenthusiast/choicebench","repo_kind":"found_in_text","path":"src/choicebench/identity.py","file_url":"https://github.com/cotenthusiast/choicebench/blob/HEAD/src/choicebench/identity.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ace6e305c009562c"}},{"code_sha256_prefix":"396beb965aacfb93","entry":"modal_k","repo":"cotenthusiast/choicebench","repo_kind":"found_in_text","path":"src/choicebench/stats.py","file_url":"https://github.com/cotenthusiast/choicebench/blob/HEAD/src/choicebench/stats.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"396beb965aacfb93"}},{"code_sha256_prefix":"62850c7e2c5064c7","entry":"permutation_filter_path_for","repo":"cotenthusiast/choicebench","repo_kind":"found_in_text","path":"src/choicebench/permutation_filter.py","file_url":"https://github.com/cotenthusiast/choicebench/blob/HEAD/src/choicebench/permutation_filter.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"62850c7e2c5064c7"}},{"code_sha256_prefix":"9ff66446a3f539d1","entry":"sanitize_url","repo":"cotenthusiast/choicebench","repo_kind":"found_in_text","path":"src/choicebench/identity.py","file_url":"https://github.com/cotenthusiast/choicebench/blob/HEAD/src/choicebench/identity.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9ff66446a3f539d1"}},{"code_sha256_prefix":"6654b16a28356e5b","entry":"letters_for","repo":"cotenthusiast/choicebench","repo_kind":"found_in_text","path":"src/choicebench/constants.py","file_url":"https://github.com/cotenthusiast/choicebench/blob/HEAD/src/choicebench/constants.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6654b16a28356e5b"}},{"code_sha256_prefix":"29a93549c3303ffb","entry":"match_permutation_unsafe_pattern","repo":"cotenthusiast/choicebench","repo_kind":"found_in_text","path":"src/choicebench/permutation_filter.py","file_url":"https://github.com/cotenthusiast/choicebench/blob/HEAD/src/choicebench/permutation_filter.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"29a93549c3303ffb"}},{"code_sha256_prefix":"541cf852b40b1742","entry":"resolve_hf_dataset_revision","repo":"cotenthusiast/choicebench","repo_kind":"found_in_text","path":"src/choicebench/provenance.py","file_url":"https://github.com/cotenthusiast/choicebench/blob/HEAD/src/choicebench/provenance.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"541cf852b40b1742"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,795 of the 9,581 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9581,"papers_checked":6795},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}