{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/setlexsem-challenge-using-set-operations-to","title":"SetLexSem Challenge: Using Set Operations to Evaluate the Lexical and Semantic Robustness of Language Models","arxiv_id":"2411.07336","date":"2024-11-11","proceeding":null,"authors":["Bardiya Akhbari","Manish Gawali","Nicholas A. Dronen"],"abstract":"Set theory is foundational to mathematics and, when sets are finite, to reasoning about the world. An intelligent system should perform set operations consistently, regardless of superficial variations in the operands. Initially designed for semantically-oriented NLP tasks, large language models (LLMs) are now being evaluated on algorithmic tasks. Because sets are comprised of arbitrary symbols (e.g. numbers, words), they provide an opportunity to test, systematically, the invariance of LLMs' algorithmic abilities under simple lexical or semantic variations. To this end, we present the SetLexSem Challenge, a synthetic benchmark that evaluates the performance of LLMs on set operations. SetLexSem assesses the robustness of LLMs' instruction-following abilities under various conditions, focusing on the set operations and the nature and construction of the set members. Evaluating seven LLMs with SetLexSem, we find that they exhibit poor robustness to variation in both operation and operands. We show -- via the framework's systematic sampling of set members along lexical and semantic dimensions -- that LLMs are not only not robust to variation along these dimensions but demonstrate unique failure modes in particular, easy-to-create semantic groupings of \"deceptive\" sets. We find that rigorously measuring language model robustness to variation in frequency and length is challenging and present an analysis that measures them independently. The code for reproducing the results of this paper, and for generating the SetLexSem Challenge dataset, is available at \\href{https://github.com/amazon-science/SetLexSem-Challenge}{https://github.com/amazon-science/SetLexSem-Challenge}.","url_abs":"https://arxiv.org/abs/2411.07336v1","url_pdf":"https://arxiv.org/pdf/2411.07336v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"setlexsem-challenge-using-set-operations-to","repo_url":"https://github.com/amazon-science/setlexsem-challenge","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"instruction-following","task_name":"Instruction Following"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.07336","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.07336"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/amazon-science/SetLexSem-Challenge","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/amazon-science/setlexsem-challenge","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":3,"unverified":20},"by_repo_kind":{"official":{"samples":23,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6ee8c0ec3ee6a7a3","entry":"astype_set","repo":"amazon-science/SetLexSem-Challenge","repo_kind":"official","path":"setlexsem/generate/generate_sets.py","file_url":"https://github.com/amazon-science/SetLexSem-Challenge/blob/HEAD/setlexsem/generate/generate_sets.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6ee8c0ec3ee6a7a3"}},{"code_sha256_prefix":"c653593d59f019e1","entry":"generate_set_pair","repo":"amazon-science/SetLexSem-Challenge","repo_kind":"official","path":"setlexsem/generate/generate_sets.py","file_url":"https://github.com/amazon-science/SetLexSem-Challenge/blob/HEAD/setlexsem/generate/generate_sets.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c653593d59f019e1"}},{"code_sha256_prefix":"39c5d1d50694781a","entry":"parse_set_pair","repo":"amazon-science/SetLexSem-Challenge","repo_kind":"official","path":"setlexsem/generate/generate_sets.py","file_url":"https://github.com/amazon-science/SetLexSem-Challenge/blob/HEAD/setlexsem/generate/generate_sets.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"39c5d1d50694781a"}},{"code_sha256_prefix":"50791cd8d19ae17c","entry":"agg","repo":"amazon-science/setlexsem-challenge","repo_kind":"official","path":"setlexsem/analyze/hypothesis_testing_utils.py","file_url":"https://github.com/amazon-science/setlexsem-challenge/blob/HEAD/setlexsem/analyze/hypothesis_testing_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"50791cd8d19ae17c"}},{"code_sha256_prefix":"7ec1a56a7fb2d0a6","entry":"aws_auth","repo":"amazon-science/setlexsem-challenge","repo_kind":"official","path":"setlexsem/experiment/lmapi.py","file_url":"https://github.com/amazon-science/setlexsem-challenge/blob/HEAD/setlexsem/experiment/lmapi.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7ec1a56a7fb2d0a6"}},{"code_sha256_prefix":"7a11de2b701c8b62","entry":"calculate_extra_info","repo":"amazon-science/setlexsem-challenge","repo_kind":"official","path":"setlexsem/analyze/error_analysis.py","file_url":"https://github.com/amazon-science/setlexsem-challenge/blob/HEAD/setlexsem/analyze/error_analysis.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7a11de2b701c8b62"}},{"code_sha256_prefix":"912d808705da066a","entry":"count_tokens","repo":"amazon-science/setlexsem-challenge","repo_kind":"official","path":"setlexsem/experiment/lmapi.py","file_url":"https://github.com/amazon-science/setlexsem-challenge/blob/HEAD/setlexsem/experiment/lmapi.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"912d808705da066a"}},{"code_sha256_prefix":"382682b1f43803e2","entry":"create_single_plot","repo":"amazon-science/setlexsem-challenge","repo_kind":"official","path":"setlexsem/analyze/visualize.py","file_url":"https://github.com/amazon-science/setlexsem-challenge/blob/HEAD/setlexsem/analyze/visualize.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"382682b1f43803e2"}},{"code_sha256_prefix":"01c3ce5d9fb476e6","entry":"create_subplots_1d","repo":"amazon-science/setlexsem-challenge","repo_kind":"official","path":"setlexsem/analyze/visualize.py","file_url":"https://github.com/amazon-science/setlexsem-challenge/blob/HEAD/setlexsem/analyze/visualize.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"01c3ce5d9fb476e6"}},{"code_sha256_prefix":"fea82b54e26300ae","entry":"filter_dataframe","repo":"amazon-science/setlexsem-challenge","repo_kind":"official","path":"setlexsem/analyze/error_analysis.py","file_url":"https://github.com/amazon-science/setlexsem-challenge/blob/HEAD/setlexsem/analyze/error_analysis.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fea82b54e26300ae"}},{"code_sha256_prefix":"8431ddf5ad802651","entry":"filter_words","repo":"amazon-science/setlexsem-challenge","repo_kind":"official","path":"setlexsem/generate/sample.py","file_url":"https://github.com/amazon-science/setlexsem-challenge/blob/HEAD/setlexsem/generate/sample.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8431ddf5ad802651"}},{"code_sha256_prefix":"600bbee8e77f30f9","entry":"get_normalized_count","repo":"amazon-science/setlexsem-challenge","repo_kind":"official","path":"setlexsem/analyze/error_analysis.py","file_url":"https://github.com/amazon-science/setlexsem-challenge/blob/HEAD/setlexsem/analyze/error_analysis.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"600bbee8e77f30f9"}},{"code_sha256_prefix":"8d9b3e28920892c7","entry":"get_path_root","repo":"amazon-science/setlexsem-challenge","repo_kind":"official","path":"setlexsem/constants.py","file_url":"https://github.com/amazon-science/setlexsem-challenge/blob/HEAD/setlexsem/constants.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8d9b3e28920892c7"}},{"code_sha256_prefix":"5afc0bf83778d553","entry":"get_stats","repo":"amazon-science/setlexsem-challenge","repo_kind":"official","path":"setlexsem/analyze/hypothesis_testing_utils.py","file_url":"https://github.com/amazon-science/setlexsem-challenge/blob/HEAD/setlexsem/analyze/hypothesis_testing_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5afc0bf83778d553"}},{"code_sha256_prefix":"b063a2fc16acb1cd","entry":"invoke_bedrock","repo":"amazon-science/setlexsem-challenge","repo_kind":"official","path":"setlexsem/experiment/lmapi.py","file_url":"https://github.com/amazon-science/setlexsem-challenge/blob/HEAD/setlexsem/experiment/lmapi.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b063a2fc16acb1cd"}},{"code_sha256_prefix":"642f476d5eddbdcf","entry":"make_english_list","repo":"amazon-science/setlexsem-challenge","repo_kind":"official","path":"setlexsem/generate/prompt.py","file_url":"https://github.com/amazon-science/setlexsem-challenge/blob/HEAD/setlexsem/generate/prompt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"642f476d5eddbdcf"}},{"code_sha256_prefix":"f829eb3daa3cf327","entry":"make_hps_prompt","repo":"amazon-science/setlexsem-challenge","repo_kind":"official","path":"setlexsem/generate/generate_prompts.py","file_url":"https://github.com/amazon-science/setlexsem-challenge/blob/HEAD/setlexsem/generate/generate_prompts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f829eb3daa3cf327"}},{"code_sha256_prefix":"ec9a1efddf07bd2e","entry":"make_sampler_name_from_hps","repo":"amazon-science/setlexsem-challenge","repo_kind":"official","path":"setlexsem/generate/sample.py","file_url":"https://github.com/amazon-science/setlexsem-challenge/blob/HEAD/setlexsem/generate/sample.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ec9a1efddf07bd2e"}},{"code_sha256_prefix":"b3f84cf0a92a6cb5","entry":"make_set","repo":"amazon-science/setlexsem-challenge","repo_kind":"official","path":"setlexsem/generate/prompt.py","file_url":"https://github.com/amazon-science/setlexsem-challenge/blob/HEAD/setlexsem/generate/prompt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b3f84cf0a92a6cb5"}},{"code_sha256_prefix":"3992b00cf71e89e3","entry":"make_verb","repo":"amazon-science/setlexsem-challenge","repo_kind":"official","path":"setlexsem/generate/prompt.py","file_url":"https://github.com/amazon-science/setlexsem-challenge/blob/HEAD/setlexsem/generate/prompt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3992b00cf71e89e3"}},{"code_sha256_prefix":"493412c498a81a41","entry":"read_yaml","repo":"amazon-science/setlexsem-challenge","repo_kind":"official","path":"setlexsem/utils.py","file_url":"https://github.com/amazon-science/setlexsem-challenge/blob/HEAD/setlexsem/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"493412c498a81a41"}},{"code_sha256_prefix":"7086c5b67a013729","entry":"replace_none","repo":"amazon-science/setlexsem-challenge","repo_kind":"official","path":"setlexsem/generate/generate_prompts.py","file_url":"https://github.com/amazon-science/setlexsem-challenge/blob/HEAD/setlexsem/generate/generate_prompts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7086c5b67a013729"}},{"code_sha256_prefix":"7d0543bbfc5e3a17","entry":"viz_barplot","repo":"amazon-science/setlexsem-challenge","repo_kind":"official","path":"setlexsem/analyze/visualize.py","file_url":"https://github.com/amazon-science/setlexsem-challenge/blob/HEAD/setlexsem/analyze/visualize.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7d0543bbfc5e3a17"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}