{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/demorank-selecting-effective-demonstrations","title":"DemoRank: Selecting Effective Demonstrations for Large Language Models in Ranking Task","arxiv_id":"2406.16332","date":"2024-06-24","proceeding":null,"authors":["Wenhan Liu","Yutao Zhu","Zhicheng Dou"],"abstract":"Recently, there has been increasing interest in applying large language models (LLMs) as zero-shot passage rankers. However, few studies have explored how to select appropriate in-context demonstrations for the passage ranking task, which is the focus of this paper. Previous studies mainly use LLM's feedback to train a retriever for demonstration selection. These studies apply the LLM to score each demonstration independently, which ignores the dependencies between demonstrations (especially important in ranking task), leading to inferior performance of top-$k$ retrieved demonstrations. To mitigate this issue, we introduce a demonstration reranker to rerank the retrieved demonstrations so that top-$k$ ranked ones are more suitable for ICL. However, generating training data for such reranker is quite challenging. On the one hand, different from demonstration retriever, the training samples of reranker need to incorporate demonstration dependencies. On the other hand, obtaining the gold ranking from the retrieved demonstrations is an NP-hard problem, which is hard to implement. To overcome these challenges, we propose a method to approximate the optimal demonstration list iteratively and utilize LLM to score demonstration lists of varying lengths. By doing so, the search space is greatly reduced and demonstration dependencies are considered. Based on these scored demonstration lists, we further design a list-pairwise training approach which compares a pair of lists that only differ in the last demonstration, to teach the reranker how to select the next demonstration given a previous sequence. In this paper, we propose a demonstration selection framework DemoRank for ranking task and conduct extensive experiments to prove its strong ability.","url_abs":"https://arxiv.org/abs/2406.16332v2","url_pdf":"https://arxiv.org/pdf/2406.16332v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"demorank-selecting-effective-demonstrations","repo_url":"https://github.com/8421bcd/demorank","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"passage-ranking","task_name":"Passage Ranking"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.16332","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.16332"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/8421bcd/demorank","reach":{"status":"ok"}}],"summary":{"ran":13,"unverified":4},"by_repo_kind":{"official":{"samples":17,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":17,"samples":[{"code_sha256_prefix":"9d00cfb48e576f8d","entry":"add_prefix","repo":"8421bcd/demorank","repo_kind":"official","path":"build_index/build_dense_index_after_train.py","file_url":"https://github.com/8421bcd/demorank/blob/HEAD/build_index/build_dense_index_after_train.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9d00cfb48e576f8d"}},{"code_sha256_prefix":"b3d0e9d10e74431c","entry":"construct_flatindex_from_embeddings","repo":"8421bcd/demorank","repo_kind":"official","path":"retrieve_utils.py","file_url":"https://github.com/8421bcd/demorank/blob/HEAD/retrieve_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b3d0e9d10e74431c"}},{"code_sha256_prefix":"79e2396cd92c6647","entry":"get_few_shot_prompts","repo":"8421bcd/demorank","repo_kind":"official","path":"llm_score.py","file_url":"https://github.com/8421bcd/demorank/blob/HEAD/llm_score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"79e2396cd92c6647"}},{"code_sha256_prefix":"2d5ea04a0d0d12b6","entry":"get_few_shot_prompts","repo":"8421bcd/demorank","repo_kind":"official","path":"llm_score_dependency_aware_rerank.py","file_url":"https://github.com/8421bcd/demorank/blob/HEAD/llm_score_dependency_aware_rerank.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2d5ea04a0d0d12b6"}},{"code_sha256_prefix":"2bb876a8dec0d022","entry":"get_pointwise_prompt","repo":"8421bcd/demorank","repo_kind":"official","path":"utils.py","file_url":"https://github.com/8421bcd/demorank/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2bb876a8dec0d022"}},{"code_sha256_prefix":"54c92ed61e995cdb","entry":"get_random_prompts","repo":"8421bcd/demorank","repo_kind":"official","path":"llm_score_dependency_aware_rerank.py","file_url":"https://github.com/8421bcd/demorank/blob/HEAD/llm_score_dependency_aware_rerank.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"54c92ed61e995cdb"}},{"code_sha256_prefix":"dd988220ebbf663f","entry":"get_random_prompts","repo":"8421bcd/demorank","repo_kind":"official","path":"run_kshot.py","file_url":"https://github.com/8421bcd/demorank/blob/HEAD/run_kshot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dd988220ebbf663f"}},{"code_sha256_prefix":"62fc247c87911474","entry":"index_retrieve","repo":"8421bcd/demorank","repo_kind":"official","path":"retrieve_utils.py","file_url":"https://github.com/8421bcd/demorank/blob/HEAD/retrieve_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"62fc247c87911474"}},{"code_sha256_prefix":"595c0a53a8076613","entry":"pool","repo":"8421bcd/demorank","repo_kind":"official","path":"build_index/build_dense_index_after_train.py","file_url":"https://github.com/8421bcd/demorank/blob/HEAD/build_index/build_dense_index_after_train.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"595c0a53a8076613"}},{"code_sha256_prefix":"dec7eb98d9881f43","entry":"sample_train_data","repo":"8421bcd/demorank","repo_kind":"official","path":"train/src/data_utils.py","file_url":"https://github.com/8421bcd/demorank/blob/HEAD/train/src/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dec7eb98d9881f43"}},{"code_sha256_prefix":"15fc32dc01a4d6a2","entry":"select_demon_by_rank","repo":"8421bcd/demorank","repo_kind":"official","path":"llm_score.py","file_url":"https://github.com/8421bcd/demorank/blob/HEAD/llm_score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"15fc32dc01a4d6a2"}},{"code_sha256_prefix":"5fe38dc5d56a7bc1","entry":"select_demon_by_rank","repo":"8421bcd/demorank","repo_kind":"official","path":"llm_score_dependency_aware_rerank.py","file_url":"https://github.com/8421bcd/demorank/blob/HEAD/llm_score_dependency_aware_rerank.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5fe38dc5d56a7bc1"}},{"code_sha256_prefix":"1094917da0e69ba3","entry":"to_up_down_format","repo":"8421bcd/demorank","repo_kind":"official","path":"train/src/data_utils.py","file_url":"https://github.com/8421bcd/demorank/blob/HEAD/train/src/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1094917da0e69ba3"}},{"code_sha256_prefix":"04460628a698fdd7","entry":"convert_index_to_gpu","repo":"8421bcd/demorank","repo_kind":"official","path":"retrieve_utils.py","file_url":"https://github.com/8421bcd/demorank/blob/HEAD/retrieve_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"04460628a698fdd7"}},{"code_sha256_prefix":"55ce04e044d233b9","entry":"get_dbs_prompts","repo":"8421bcd/demorank","repo_kind":"official","path":"run_kshot.py","file_url":"https://github.com/8421bcd/demorank/blob/HEAD/run_kshot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"55ce04e044d233b9"}},{"code_sha256_prefix":"6f3c55ed5d1e9d5e","entry":"get_kmeans_prompts","repo":"8421bcd/demorank","repo_kind":"official","path":"run_kshot.py","file_url":"https://github.com/8421bcd/demorank/blob/HEAD/run_kshot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6f3c55ed5d1e9d5e"}},{"code_sha256_prefix":"f56c7d1d6b97751f","entry":"get_random_prompts","repo":"8421bcd/demorank","repo_kind":"official","path":"llm_score.py","file_url":"https://github.com/8421bcd/demorank/blob/HEAD/llm_score.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f56c7d1d6b97751f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}