{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2604-07035","title":"Unified Deployment-Aware Evaluation of Open Reasoning Language Models","arxiv_id":"2604.07035","date":"2026-04-08","proceeding":null,"authors":["Md Motaleb Hossen Manik","Ge Wang"],"abstract":"Open reasoning language models are often compared under mixed sample sizes, partially standardized prompts, and accuracy-centered summaries, which makes practical model selection difficult to interpret. We present a unified evaluation of seven open reasoning language model configurations across four benchmarks: ARC-Challenge, GSM8K, MATH levels 1 to 3, and TruthfulQA MC1. We test zero-shot, chain-of-thought (CoT), and few-shot CoT prompting on the same 238-example subset for every model--dataset--strategy condition, yielding a complete 7 x 4 x 3 design with 84 conditions and 19,992 evaluated examples. Beyond accuracy, we report Wilson confidence intervals, latency, peak video random access memory (VRAM), weighted aggregate performance, Pareto-efficient operating points, prompt-sensitivity metrics, and compatibility diagnostics. Gemma-4-26B-A4B with zero-shot prompting achieves the highest weighted score at 0.794. Gemma-4-E4B remains close to the top across prompting settings while using substantially lower latency and memory, making it a strong practical operating point. Bootstrap and paired-permutation analyses show that the leading configurations are close enough that deployment tradeoffs remain important. We also find that prompting strategy changes model rankings rather than shifting all models uniformly. Benchmark-specific complementarity creates routing headroom, with an oracle task-aware selector reaching a weighted score of 0.825. Compatibility diagnostics show that some apparent failures, especially Phi-4-Reasoning on GSM8K, reflect robustness and interface-adherence problems under the shared evaluation pipeline. These results support a central claim: open-model evaluation should be framed as a deployment-aware, multi-objective operating-point problem rather than as a single-score leaderboard exercise.","url_abs":"https://arxiv.org/abs/2604.07035","url_pdf":"https://arxiv.org/pdf/2604.07035","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2604.07035","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2604.07035"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/mkboch/UDAE","reach":null}],"summary":{"unverified":27},"by_repo_kind":{"found_in_text":{"samples":27,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":27,"samples":[{"code_sha256_prefix":"b46b8c24217896bd","entry":"accuracy_with_ci","repo":"mkboch/UDAE","repo_kind":"found_in_text","path":"evaluation/metrics.py","file_url":"https://github.com/mkboch/UDAE/blob/HEAD/evaluation/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b46b8c24217896bd"}},{"code_sha256_prefix":"ffe57dd4455d5d52","entry":"build_few_shot_block","repo":"mkboch/UDAE","repo_kind":"found_in_text","path":"prompts/builder.py","file_url":"https://github.com/mkboch/UDAE/blob/HEAD/prompts/builder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ffe57dd4455d5d52"}},{"code_sha256_prefix":"35b0ce59f15f774d","entry":"build_prompt","repo":"mkboch/UDAE","repo_kind":"found_in_text","path":"prompts/builder.py","file_url":"https://github.com/mkboch/UDAE/blob/HEAD/prompts/builder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"35b0ce59f15f774d"}},{"code_sha256_prefix":"d1264a79572c608f","entry":"clean_generated_text","repo":"mkboch/UDAE","repo_kind":"found_in_text","path":"models/inference.py","file_url":"https://github.com/mkboch/UDAE/blob/HEAD/models/inference.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d1264a79572c608f"}},{"code_sha256_prefix":"e1ecec747bd2a353","entry":"cohens_h","repo":"mkboch/UDAE","repo_kind":"found_in_text","path":"evaluation/metrics.py","file_url":"https://github.com/mkboch/UDAE/blob/HEAD/evaluation/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e1ecec747bd2a353"}},{"code_sha256_prefix":"dae1ad66afb68436","entry":"extract_arc","repo":"mkboch/UDAE","repo_kind":"found_in_text","path":"evaluation/extractor.py","file_url":"https://github.com/mkboch/UDAE/blob/HEAD/evaluation/extractor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dae1ad66afb68436"}},{"code_sha256_prefix":"313c701d2ffb4d5a","entry":"extract_gold_gsm8k","repo":"mkboch/UDAE","repo_kind":"found_in_text","path":"evaluation/grader.py","file_url":"https://github.com/mkboch/UDAE/blob/HEAD/evaluation/grader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"313c701d2ffb4d5a"}},{"code_sha256_prefix":"4e929eef98401978","entry":"extract_gsm8k","repo":"mkboch/UDAE","repo_kind":"found_in_text","path":"evaluation/extractor.py","file_url":"https://github.com/mkboch/UDAE/blob/HEAD/evaluation/extractor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4e929eef98401978"}},{"code_sha256_prefix":"d612a82a2e6f9d63","entry":"extract_math","repo":"mkboch/UDAE","repo_kind":"found_in_text","path":"evaluation/extractor.py","file_url":"https://github.com/mkboch/UDAE/blob/HEAD/evaluation/extractor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d612a82a2e6f9d63"}},{"code_sha256_prefix":"39b49da1df05acbe","entry":"filter_models","repo":"mkboch/UDAE","repo_kind":"found_in_text","path":"experiments/run_experiment.py","file_url":"https://github.com/mkboch/UDAE/blob/HEAD/experiments/run_experiment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"39b49da1df05acbe"}},{"code_sha256_prefix":"c4f1e80f4872c9db","entry":"find_file","repo":"mkboch/UDAE","repo_kind":"found_in_text","path":"code/additional_analysis_scripts/audit_full_run.py","file_url":"https://github.com/mkboch/UDAE/blob/HEAD/code/additional_analysis_scripts/audit_full_run.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c4f1e80f4872c9db"}},{"code_sha256_prefix":"894761ab6aa57596","entry":"find_file","repo":"mkboch/UDAE","repo_kind":"found_in_text","path":"code/additional_analysis_scripts/check_experiments_1_to_5.py","file_url":"https://github.com/mkboch/UDAE/blob/HEAD/code/additional_analysis_scripts/check_experiments_1_to_5.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"894761ab6aa57596"}},{"code_sha256_prefix":"92e0dae95ed263ab","entry":"get_torch_dtype","repo":"mkboch/UDAE","repo_kind":"found_in_text","path":"models/loader.py","file_url":"https://github.com/mkboch/UDAE/blob/HEAD/models/loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"92e0dae95ed263ab"}},{"code_sha256_prefix":"d7449f228840e30b","entry":"load_dataset_records","repo":"mkboch/UDAE","repo_kind":"found_in_text","path":"experiments/run_benchmark_final_clean.py","file_url":"https://github.com/mkboch/UDAE/blob/HEAD/experiments/run_benchmark_final_clean.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d7449f228840e30b"}},{"code_sha256_prefix":"6b7c1e512fe35435","entry":"load_model","repo":"mkboch/UDAE","repo_kind":"found_in_text","path":"models/loader.py","file_url":"https://github.com/mkboch/UDAE/blob/HEAD/models/loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6b7c1e512fe35435"}},{"code_sha256_prefix":"d5e3096782eba05c","entry":"load_prepared_records","repo":"mkboch/UDAE","repo_kind":"found_in_text","path":"experiments/run_experiment.py","file_url":"https://github.com/mkboch/UDAE/blob/HEAD/experiments/run_experiment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d5e3096782eba05c"}},{"code_sha256_prefix":"09b3ca5c2d7d5b1d","entry":"load_yaml","repo":"mkboch/UDAE","repo_kind":"found_in_text","path":"experiments/run_benchmark_final_clean.py","file_url":"https://github.com/mkboch/UDAE/blob/HEAD/experiments/run_benchmark_final_clean.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"09b3ca5c2d7d5b1d"}},{"code_sha256_prefix":"32b3e0f038f60081","entry":"load_yaml","repo":"mkboch/UDAE","repo_kind":"found_in_text","path":"experiments/run_experiment.py","file_url":"https://github.com/mkboch/UDAE/blob/HEAD/experiments/run_experiment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"32b3e0f038f60081"}},{"code_sha256_prefix":"e55180874babf570","entry":"maybe_apply_chat_template","repo":"mkboch/UDAE","repo_kind":"found_in_text","path":"prompts/builder.py","file_url":"https://github.com/mkboch/UDAE/blob/HEAD/prompts/builder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e55180874babf570"}},{"code_sha256_prefix":"456270afeb970217","entry":"md_table","repo":"mkboch/UDAE","repo_kind":"found_in_text","path":"code/additional_analysis_scripts/check_experiments_1_to_5.py","file_url":"https://github.com/mkboch/UDAE/blob/HEAD/code/additional_analysis_scripts/check_experiments_1_to_5.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"456270afeb970217"}},{"code_sha256_prefix":"93eb9a3b2ead392a","entry":"normalize_number_like","repo":"mkboch/UDAE","repo_kind":"found_in_text","path":"evaluation/grader.py","file_url":"https://github.com/mkboch/UDAE/blob/HEAD/evaluation/grader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"93eb9a3b2ead392a"}},{"code_sha256_prefix":"87a574d68dd0da52","entry":"normalize_text","repo":"mkboch/UDAE","repo_kind":"found_in_text","path":"evaluation/grader.py","file_url":"https://github.com/mkboch/UDAE/blob/HEAD/evaluation/grader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"87a574d68dd0da52"}},{"code_sha256_prefix":"c56861d06ede9b74","entry":"read_condition","repo":"mkboch/UDAE","repo_kind":"found_in_text","path":"code/additional_analysis_scripts/check_experiments_1_to_5.py","file_url":"https://github.com/mkboch/UDAE/blob/HEAD/code/additional_analysis_scripts/check_experiments_1_to_5.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c56861d06ede9b74"}},{"code_sha256_prefix":"df17b9a60ed6753f","entry":"run_inference","repo":"mkboch/UDAE","repo_kind":"found_in_text","path":"models/inference.py","file_url":"https://github.com/mkboch/UDAE/blob/HEAD/models/inference.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"df17b9a60ed6753f"}},{"code_sha256_prefix":"8990f9360920f94e","entry":"select_model_cfg","repo":"mkboch/UDAE","repo_kind":"found_in_text","path":"experiments/run_single.py","file_url":"https://github.com/mkboch/UDAE/blob/HEAD/experiments/run_single.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8990f9360920f94e"}},{"code_sha256_prefix":"d6866f29cafe61d4","entry":"tokens_per_second","repo":"mkboch/UDAE","repo_kind":"found_in_text","path":"models/inference.py","file_url":"https://github.com/mkboch/UDAE/blob/HEAD/models/inference.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d6866f29cafe61d4"}},{"code_sha256_prefix":"e60660601bd75ac4","entry":"wilson_ci","repo":"mkboch/UDAE","repo_kind":"found_in_text","path":"evaluation/metrics.py","file_url":"https://github.com/mkboch/UDAE/blob/HEAD/evaluation/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e60660601bd75ac4"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}