{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2607-20491","title":"DFAH-Bench: Benchmarking Observable Agent Instability in Financial Decision-Making","arxiv_id":"2607.20491","date":"2026-06-10","proceeding":null,"authors":["Raffi Khatchadourian"],"abstract":"A financial AI agent can repeat a decision while changing the tools, order, or recorded arguments and results used to reach it. Outcome-only evaluation misses this variation, even when it matters for replay and change control. DFAH-Bench operationalizes the Determinism-Faithfulness Assurance Harness (DFAH), where faithfulness means fidelity of observable execution under replay, not answer correctness. The protocol qualifies comparable, sufficiently observed replays and measures decision agreement (DAR) and tool-path agreement (TAR) over the same eligible groups. We analyze 4,157 retrospective episodes from configurations with observed tool use across 719 synthetic compliance and financial DataOps groups, together with an argument-aware prospective extension comprising 570 eligible episodes across 190 groups. In that extension, decisions agree 94.2-95.1% while exact tool-name paths agree 66.9-69.4%, producing 25.8-27.3 percentage-point gaps; argument-and-result trajectory agreement falls to 45.0-51.5%. Even among unanimous-decision groups, paths vary in 66.7-68.9% under task weighting. DFAH-Bench makes the execution behind a stable decision visible for replay, investigation, and change review.","url_abs":"https://arxiv.org/abs/2607.20491","url_pdf":"https://arxiv.org/pdf/2607.20491","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2607.20491","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2607.20491"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/ibm-client-engineering/output-drift-financial-llms","reach":null}],"summary":{"ran":15,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":17,"ran":15,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ab9dc624b00a431e","entry":"analyze_faithfulness_determinism_tradeoff","repo":"ibm-client-engineering/output-drift-financial-llms","repo_kind":"found_in_text","path":"econometrics/analyze_real_data.py","file_url":"https://github.com/ibm-client-engineering/output-drift-financial-llms/blob/HEAD/econometrics/analyze_real_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ab9dc624b00a431e"}},{"code_sha256_prefix":"109125dea3af89f5","entry":"analyze_model_tiers","repo":"ibm-client-engineering/output-drift-financial-llms","repo_kind":"found_in_text","path":"econometrics/analyze_real_data.py","file_url":"https://github.com/ibm-client-engineering/output-drift-financial-llms/blob/HEAD/econometrics/analyze_real_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"109125dea3af89f5"}},{"code_sha256_prefix":"ad4a8061dcc44459","entry":"compute_measurement_error_covariance","repo":"ibm-client-engineering/output-drift-financial-llms","repo_kind":"found_in_text","path":"econometrics/validation_debiasing.py","file_url":"https://github.com/ibm-client-engineering/output-drift-financial-llms/blob/HEAD/econometrics/validation_debiasing.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ad4a8061dcc44459"}},{"code_sha256_prefix":"fb0cde7b90c7bbe7","entry":"compute_ngram_fingerprint","repo":"ibm-client-engineering/output-drift-financial-llms","repo_kind":"found_in_text","path":"econometrics/leakage_detection.py","file_url":"https://github.com/ibm-client-engineering/output-drift-financial-llms/blob/HEAD/econometrics/leakage_detection.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fb0cde7b90c7bbe7"}},{"code_sha256_prefix":"aeafd2672665514a","entry":"compute_tf_vector","repo":"ibm-client-engineering/output-drift-financial-llms","repo_kind":"found_in_text","path":"econometrics/semantic_divergence_econometric.py","file_url":"https://github.com/ibm-client-engineering/output-drift-financial-llms/blob/HEAD/econometrics/semantic_divergence_econometric.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aeafd2672665514a"}},{"code_sha256_prefix":"ed2de81ea3093e9c","entry":"compute_token_mismatch","repo":"ibm-client-engineering/output-drift-financial-llms","repo_kind":"found_in_text","path":"econometrics/semantic_divergence_econometric.py","file_url":"https://github.com/ibm-client-engineering/output-drift-financial-llms/blob/HEAD/econometrics/semantic_divergence_econometric.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed2de81ea3093e9c"}},{"code_sha256_prefix":"8aac78959bcae7bb","entry":"detect_temporal_leakage","repo":"ibm-client-engineering/output-drift-financial-llms","repo_kind":"found_in_text","path":"econometrics/leakage_detection.py","file_url":"https://github.com/ibm-client-engineering/output-drift-financial-llms/blob/HEAD/econometrics/leakage_detection.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8aac78959bcae7bb"}},{"code_sha256_prefix":"b96061d3b5f18fb1","entry":"estimate_categorical_drift_rate","repo":"ibm-client-engineering/output-drift-financial-llms","repo_kind":"found_in_text","path":"econometrics/drift_variance_estimator.py","file_url":"https://github.com/ibm-client-engineering/output-drift-financial-llms/blob/HEAD/econometrics/drift_variance_estimator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b96061d3b5f18fb1"}},{"code_sha256_prefix":"0e9d3bde9f3a6912","entry":"estimate_drift_variance","repo":"ibm-client-engineering/output-drift-financial-llms","repo_kind":"found_in_text","path":"econometrics/validation_debiasing.py","file_url":"https://github.com/ibm-client-engineering/output-drift-financial-llms/blob/HEAD/econometrics/validation_debiasing.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0e9d3bde9f3a6912"}},{"code_sha256_prefix":"6446d46ba40ef240","entry":"estimate_sample_drift_variance","repo":"ibm-client-engineering/output-drift-financial-llms","repo_kind":"found_in_text","path":"econometrics/drift_variance_estimator.py","file_url":"https://github.com/ibm-client-engineering/output-drift-financial-llms/blob/HEAD/econometrics/drift_variance_estimator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6446d46ba40ef240"}},{"code_sha256_prefix":"5076293b4c331597","entry":"extract_dates_from_text","repo":"ibm-client-engineering/output-drift-financial-llms","repo_kind":"found_in_text","path":"econometrics/leakage_detection.py","file_url":"https://github.com/ibm-client-engineering/output-drift-financial-llms/blob/HEAD/econometrics/leakage_detection.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5076293b4c331597"}},{"code_sha256_prefix":"ec5cceabd14e66b3","entry":"get_supported_models","repo":"ibm-client-engineering/output-drift-financial-llms","repo_kind":"found_in_text","path":"harness/load_models.py","file_url":"https://github.com/ibm-client-engineering/output-drift-financial-llms/blob/HEAD/harness/load_models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec5cceabd14e66b3"}},{"code_sha256_prefix":"8ad577384dcbd4e6","entry":"naive_ols_regression","repo":"ibm-client-engineering/output-drift-financial-llms","repo_kind":"found_in_text","path":"econometrics/validation_debiasing.py","file_url":"https://github.com/ibm-client-engineering/output-drift-financial-llms/blob/HEAD/econometrics/validation_debiasing.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8ad577384dcbd4e6"}},{"code_sha256_prefix":"f8a9742e520382f7","entry":"resolve_model_name","repo":"ibm-client-engineering/output-drift-financial-llms","repo_kind":"found_in_text","path":"harness/load_models.py","file_url":"https://github.com/ibm-client-engineering/output-drift-financial-llms/blob/HEAD/harness/load_models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f8a9742e520382f7"}},{"code_sha256_prefix":"759c8d14d87fe68c","entry":"run_multi_run_analysis","repo":"ibm-client-engineering/output-drift-financial-llms","repo_kind":"found_in_text","path":"econometrics/drift_variance_estimator.py","file_url":"https://github.com/ibm-client-engineering/output-drift-financial-llms/blob/HEAD/econometrics/drift_variance_estimator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"759c8d14d87fe68c"}},{"code_sha256_prefix":"a23a29a8df70cbb3","entry":"analyze_task_structure_effect","repo":"ibm-client-engineering/output-drift-financial-llms","repo_kind":"found_in_text","path":"econometrics/analyze_real_data.py","file_url":"https://github.com/ibm-client-engineering/output-drift-financial-llms/blob/HEAD/econometrics/analyze_real_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a23a29a8df70cbb3"}},{"code_sha256_prefix":"11ee49b80af0b23d","entry":"compute_cosine_divergence","repo":"ibm-client-engineering/output-drift-financial-llms","repo_kind":"found_in_text","path":"econometrics/semantic_divergence_econometric.py","file_url":"https://github.com/ibm-client-engineering/output-drift-financial-llms/blob/HEAD/econometrics/semantic_divergence_econometric.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"11ee49b80af0b23d"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,264 of the 9,581 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9581,"papers_checked":6264},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}