{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/context-aware-testing-a-new-paradigm-for","title":"Context-Aware Testing: A New Paradigm for Model Testing with Large Language Models","arxiv_id":"2410.24005","date":"2024-10-31","proceeding":null,"authors":["Paulius Rauba","Nabeel Seedat","Max Ruiz Luyten","Mihaela van der Schaar"],"abstract":"The predominant de facto paradigm of testing ML models relies on either using only held-out data to compute aggregate evaluation metrics or by assessing the performance on different subgroups. However, such data-only testing methods operate under the restrictive assumption that the available empirical data is the sole input for testing ML models, disregarding valuable contextual information that could guide model testing. In this paper, we challenge the go-to approach of data-only testing and introduce context-aware testing (CAT) which uses context as an inductive bias to guide the search for meaningful model failures. We instantiate the first CAT system, SMART Testing, which employs large language models to hypothesize relevant and likely failures, which are evaluated on data using a self-falsification mechanism. Through empirical evaluations in diverse settings, we show that SMART automatically identifies more relevant and impactful failures than alternatives, demonstrating the potential of CAT as a testing paradigm.","url_abs":"https://arxiv.org/abs/2410.24005v1","url_pdf":"https://arxiv.org/pdf/2410.24005v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"context-aware-testing-a-new-paradigm-for","repo_url":"https://github.com/pauliusrauba/SMART_Testing","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":null},{"paper_slug":"context-aware-testing-a-new-paradigm-for","repo_url":"https://github.com/vanderschaarlab/SMART_Testing","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"inductive-bias","task_name":"Inductive Bias"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.24005","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.24005"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vanderschaarlab/SMART_Testing","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pauliusrauba/SMART_Testing","reach":null}],"summary":{"ran_fixture":3,"ran_draft_wrong":11,"ran_honours":2,"unverified":11},"by_repo_kind":{"official":{"samples":27,"ran":16,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":27,"samples":[{"code_sha256_prefix":"980b86cc82a40ff3","entry":"add_categorical_variables","repo":"pauliusrauba/smart_testing","repo_kind":"official","path":"experiments/exp1.py","file_url":"https://github.com/pauliusrauba/smart_testing/blob/HEAD/experiments/exp1.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"980b86cc82a40ff3"}},{"code_sha256_prefix":"082775765bcf5927","entry":"bootstrapping_test_for_accuracy","repo":"pauliusrauba/SMART_Testing","repo_kind":"official","path":"src/SMART.py","file_url":"https://github.com/pauliusrauba/SMART_Testing/blob/HEAD/src/SMART.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"082775765bcf5927"}},{"code_sha256_prefix":"644cdbc5b96ea94b","entry":"bootstrapping_test_for_accuracy_string","repo":"pauliusrauba/SMART_Testing","repo_kind":"official","path":"src/SMART.py","file_url":"https://github.com/pauliusrauba/SMART_Testing/blob/HEAD/src/SMART.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"644cdbc5b96ea94b"}},{"code_sha256_prefix":"a842e24eaf74fcd2","entry":"calculate_group_statistics_string","repo":"pauliusrauba/SMART_Testing","repo_kind":"official","path":"src/SMART.py","file_url":"https://github.com/pauliusrauba/SMART_Testing/blob/HEAD/src/SMART.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a842e24eaf74fcd2"}},{"code_sha256_prefix":"04ef2bc634603850","entry":"calculate_lift","repo":"pauliusrauba/SMART_Testing","repo_kind":"official","path":"src/SMART.py","file_url":"https://github.com/pauliusrauba/SMART_Testing/blob/HEAD/src/SMART.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"04ef2bc634603850"}},{"code_sha256_prefix":"952dd38f279afee7","entry":"calculate_lift_outcome","repo":"pauliusrauba/SMART_Testing","repo_kind":"official","path":"src/SMART.py","file_url":"https://github.com/pauliusrauba/SMART_Testing/blob/HEAD/src/SMART.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"952dd38f279afee7"}},{"code_sha256_prefix":"48299ec78578a7ca","entry":"calculate_odds_ratio","repo":"pauliusrauba/SMART_Testing","repo_kind":"official","path":"src/SMART.py","file_url":"https://github.com/pauliusrauba/SMART_Testing/blob/HEAD/src/SMART.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"48299ec78578a7ca"}},{"code_sha256_prefix":"ddea085ab8189264","entry":"calculate_odds_ratio_acc","repo":"pauliusrauba/SMART_Testing","repo_kind":"official","path":"src/SMART.py","file_url":"https://github.com/pauliusrauba/SMART_Testing/blob/HEAD/src/SMART.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ddea085ab8189264"}},{"code_sha256_prefix":"fa2e7f5436b471ff","entry":"calculate_weighted_relative_accuracy","repo":"pauliusrauba/SMART_Testing","repo_kind":"official","path":"src/SMART.py","file_url":"https://github.com/pauliusrauba/SMART_Testing/blob/HEAD/src/SMART.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fa2e7f5436b471ff"}},{"code_sha256_prefix":"c1530c3092d8eb42","entry":"calculate_weighted_relative_outcomes","repo":"pauliusrauba/SMART_Testing","repo_kind":"official","path":"src/SMART.py","file_url":"https://github.com/pauliusrauba/SMART_Testing/blob/HEAD/src/SMART.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c1530c3092d8eb42"}},{"code_sha256_prefix":"a17ed6b8532eb0da","entry":"chi_square_test_for_accuracy","repo":"pauliusrauba/SMART_Testing","repo_kind":"official","path":"src/SMART.py","file_url":"https://github.com/pauliusrauba/SMART_Testing/blob/HEAD/src/SMART.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a17ed6b8532eb0da"}},{"code_sha256_prefix":"78bb22f3c0eb1ec3","entry":"clean_query_string","repo":"pauliusrauba/smart_testing","repo_kind":"official","path":"src/SMART.py","file_url":"https://github.com/pauliusrauba/smart_testing/blob/HEAD/src/SMART.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"78bb22f3c0eb1ec3"}},{"code_sha256_prefix":"861a293ada0282e4","entry":"convert_to_string_condition","repo":"pauliusrauba/smart_testing","repo_kind":"official","path":"src/SMART.py","file_url":"https://github.com/pauliusrauba/smart_testing/blob/HEAD/src/SMART.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"861a293ada0282e4"}},{"code_sha256_prefix":"fbfa3ff05c9f3076","entry":"generate_combinations_for_variable","repo":"pauliusrauba/smart_testing","repo_kind":"official","path":"src/SMART.py","file_url":"https://github.com/pauliusrauba/smart_testing/blob/HEAD/src/SMART.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fbfa3ff05c9f3076"}},{"code_sha256_prefix":"3f9858e6a6582447","entry":"mcnemars_test","repo":"pauliusrauba/SMART_Testing","repo_kind":"official","path":"src/SMART.py","file_url":"https://github.com/pauliusrauba/SMART_Testing/blob/HEAD/src/SMART.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3f9858e6a6582447"}},{"code_sha256_prefix":"dd22a0f722bbba34","entry":"welchs_t_test_for_accuracy","repo":"pauliusrauba/SMART_Testing","repo_kind":"official","path":"src/SMART.py","file_url":"https://github.com/pauliusrauba/SMART_Testing/blob/HEAD/src/SMART.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dd22a0f722bbba34"}},{"code_sha256_prefix":"d6be180787be0544","entry":"SMART","repo":"pauliusrauba/SMART_Testing","repo_kind":"official","path":"src/SMART.py","file_url":"https://github.com/pauliusrauba/SMART_Testing/blob/HEAD/src/SMART.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d6be180787be0544"}},{"code_sha256_prefix":"352bd9c2249abd79","entry":"calculate_group_statistics","repo":"pauliusrauba/SMART_Testing","repo_kind":"official","path":"src/SMART.py","file_url":"https://github.com/pauliusrauba/SMART_Testing/blob/HEAD/src/SMART.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"352bd9c2249abd79"}},{"code_sha256_prefix":"64d7572fd4d95cd6","entry":"calculcate_white_black_scores","repo":"vanderschaarlab/SMART_Testing","repo_kind":"official","path":"experiments/exp4.py","file_url":"https://github.com/vanderschaarlab/SMART_Testing/blob/HEAD/experiments/exp4.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"64d7572fd4d95cd6"}},{"code_sha256_prefix":"7b95bcfb0f225d7e","entry":"clean_features","repo":"vanderschaarlab/SMART_Testing","repo_kind":"official","path":"experiments/exp2.py","file_url":"https://github.com/vanderschaarlab/SMART_Testing/blob/HEAD/experiments/exp2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7b95bcfb0f225d7e"}},{"code_sha256_prefix":"655af21881ab2ee3","entry":"generate_dataset","repo":"vanderschaarlab/SMART_Testing","repo_kind":"official","path":"experiments/exp3.py","file_url":"https://github.com/vanderschaarlab/SMART_Testing/blob/HEAD/experiments/exp3.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"655af21881ab2ee3"}},{"code_sha256_prefix":"e041ca87d939d79f","entry":"generate_mdss_query_condition","repo":"vanderschaarlab/SMART_Testing","repo_kind":"official","path":"src/subgroup_helpers.py","file_url":"https://github.com/vanderschaarlab/SMART_Testing/blob/HEAD/src/subgroup_helpers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e041ca87d939d79f"}},{"code_sha256_prefix":"5f50e8beb718878d","entry":"get_data_breast_cancer","repo":"vanderschaarlab/SMART_Testing","repo_kind":"official","path":"src/dataset_helpers.py","file_url":"https://github.com/vanderschaarlab/SMART_Testing/blob/HEAD/src/dataset_helpers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5f50e8beb718878d"}},{"code_sha256_prefix":"8eafe28f193a5cb3","entry":"get_data_compas","repo":"vanderschaarlab/SMART_Testing","repo_kind":"official","path":"src/dataset_helpers.py","file_url":"https://github.com/vanderschaarlab/SMART_Testing/blob/HEAD/src/dataset_helpers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8eafe28f193a5cb3"}},{"code_sha256_prefix":"859fdcdc54b12008","entry":"get_data_education","repo":"vanderschaarlab/SMART_Testing","repo_kind":"official","path":"src/dataset_helpers.py","file_url":"https://github.com/vanderschaarlab/SMART_Testing/blob/HEAD/src/dataset_helpers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"859fdcdc54b12008"}},{"code_sha256_prefix":"dee4e53b7cfcbb6d","entry":"get_queries_many_exp","repo":"vanderschaarlab/SMART_Testing","repo_kind":"official","path":"experiments/exp2.py","file_url":"https://github.com/vanderschaarlab/SMART_Testing/blob/HEAD/experiments/exp2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dee4e53b7cfcbb6d"}},{"code_sha256_prefix":"b0c4fc6d6786516a","entry":"load_seer_cutract_dataset","repo":"vanderschaarlab/SMART_Testing","repo_kind":"official","path":"experiments/exp2.py","file_url":"https://github.com/vanderschaarlab/SMART_Testing/blob/HEAD/experiments/exp2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b0c4fc6d6786516a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}