{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/analobench-benchmarking-the-identification-of","title":"AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogies","arxiv_id":"2402.12370","date":"2024-02-19","proceeding":null,"authors":["Xiao Ye","Andrew Wang","Jacob Choi","Yining Lu","Shreya Sharma","Lingfeng Shen","Vijay Tiyyala","Nicholas Andrews","Daniel Khashabi"],"abstract":"Humans regularly engage in analogical thinking, relating personal experiences to current situations (X is analogous to Y because of Z). Analogical thinking allows humans to solve problems in creative ways, grasp difficult concepts, and articulate ideas more effectively. Can language models (LMs) do the same? To answer this question, we propose AnaloBench, a benchmark to determine analogical reasoning ability in LMs. Our benchmarking approach focuses on aspects of this ability that are common among humans: (i) recalling related experiences from a large amount of information, and (ii) applying analogical reasoning to complex and lengthy scenarios. We test a broad collection of proprietary models (e.g., GPT family, Claude V2) and open source models such as LLaMA2. As in prior results, scaling up LMs results in some performance boosts. Surprisingly, scale offers minimal gains when, (i) analogies involve lengthy scenarios, or (ii) recalling relevant scenarios from a large pool of information, a process analogous to finding a needle in a haystack. We hope these observations encourage further research in this field.","url_abs":"https://arxiv.org/abs/2402.12370v2","url_pdf":"https://arxiv.org/pdf/2402.12370v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"analobench-benchmarking-the-identification-of","repo_url":"https://github.com/jhu-clsp/analogical-reasoning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"analobench-benchmarking-the-identification-of","repo_url":"https://github.com/JHU-CLSP/AnaloBench","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"discriminative-fine-tuning","method_name":"Discriminative Fine-Tuning"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt","method_name":"GPT"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2402.12370","atlas_url":"https://app.syntology.ai/?focus=2402.12370","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.12370"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/JHU-CLSP/AnaloBench","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jhu-clsp/analogical-reasoning","reach":{"status":"ok"}}],"summary":{"ran":4,"unverified":2},"by_repo_kind":{"official":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"c9e6dded4fecc811","entry":"calculate_average_precision","repo":"JHU-CLSP/AnaloBench","repo_kind":"official","path":"code/t2_evaluation.py","file_url":"https://github.com/JHU-CLSP/AnaloBench/blob/HEAD/code/t2_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c9e6dded4fecc811"}},{"code_sha256_prefix":"f8bea94a7230aaba","entry":"check_out","repo":"JHU-CLSP/AnaloBench","repo_kind":"official","path":"code/t1_evaluation.py","file_url":"https://github.com/JHU-CLSP/AnaloBench/blob/HEAD/code/t1_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f8bea94a7230aaba"}},{"code_sha256_prefix":"1ce58e28d1fcc2b4","entry":"precision_at_k","repo":"JHU-CLSP/AnaloBench","repo_kind":"official","path":"code/t2_evaluation.py","file_url":"https://github.com/JHU-CLSP/AnaloBench/blob/HEAD/code/t2_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1ce58e28d1fcc2b4"}},{"code_sha256_prefix":"53b4a5899413ad22","entry":"retrieval_at_k","repo":"JHU-CLSP/AnaloBench","repo_kind":"official","path":"code/t2_evaluation.py","file_url":"https://github.com/JHU-CLSP/AnaloBench/blob/HEAD/code/t2_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"53b4a5899413ad22"}},{"code_sha256_prefix":"171ef823a8d9efbc","entry":"generate_chat_completion","repo":"JHU-CLSP/AnaloBench","repo_kind":"official","path":"code/t2_GPT4.py","file_url":"https://github.com/JHU-CLSP/AnaloBench/blob/HEAD/code/t2_GPT4.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"171ef823a8d9efbc"}},{"code_sha256_prefix":"e95df4a98886c7a6","entry":"load_and_map_clustered_sentences","repo":"JHU-CLSP/AnaloBench","repo_kind":"official","path":"code/data_generation.py","file_url":"https://github.com/JHU-CLSP/AnaloBench/blob/HEAD/code/data_generation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e95df4a98886c7a6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}