{"url":"/dataset/dpr-ann","name":"DPR-ANN","full_name":null,"description_markdown":"We provide the [code](https://github.com/IntelLabs/DPR-dataset-generator/tree/main) to generate base and query vector datasets for similarity search benchmarking and evaluation on high-dimensional vectors stemming from large language models. With the dense passage retriever (DPR) [[1]](#1), we encode text snippets from the C4 dataset [[2]](#2) to generate 768-dimensional vectors:\r\n\r\n- context DPR embeddings for the base set and\r\n- question DPR embeddings for the query set.\r\n\r\nThe metric for similarity search is inner product [[1]](#1).\r\n\r\nThe number of base and query embedding vectors is parametrizable.\r\n\r\nSee the [main repository](https://github.com/IntelLabs/DPR-dataset-generator/tree/main) for details on how to generate\r\nthe DPR10M specific instance introduced in [[3]](#3).\r\n\r\n<a id=\"1\">[1]</a>\r\nKarpukhin, V.; Oguz, B.; Min, S.; Lewis, P.; Wu, L.; Edunov, S.; Chen, D.; Yih, W..: Dense Passage\r\nRetrieval for Open-Domain Question Answering. In: Proceedings of the 2020 Conference on Empirical\r\nMethods in Natural Language Processing (EMNLP). 6769–6781. (2020)\r\n\r\n<a id=\"2\">[2]</a>\r\nRaffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; Liu,\r\nP.J.: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.\r\nIn: The Journal of Machine Learning Research 21,140:1–140:67.(2020)\r\n\r\n<a id=\"3\">[3]</a>\r\nAguerrebere, C.; Bhati I.; Hildebrand M.; Tepper M.; Willke T.:Similarity search in the blink of an eye with compressed\r\nindices. In: Proceedings of the VLDB Endowment, 16, 11 (2023)","description_withheld":null,"homepage":"https://github.com/IntelLabs/DPR-dataset-generator/tree/main","introduced_date":"2023-07-24","introduced_date_note":null,"introduced_by":{"paper":"/paper/similarity-search-in-the-blink-of-an-eye-with","title":"Similarity search in the blink of an eye with compressed indices","first_author":"Cecilia Aguerrebere","url":null},"license":{"name":"MIT","url":"https://github.com/IntelLabs/DPR-dataset-generator/blob/main/LICENSE"},"modalities":[],"tasks":[],"languages":[],"variants":["DPR-ANN"],"data_loaders":[],"num_papers_in_archive":1,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[],"papers_with_a_benchmark_row":[],"syntology_totals":{"read_at":"2026-09-24T18:15:14+00:00","papers_with_samples":0,"samples_harvested":0,"samples_ran":0,"samples_unverified":0,"pointer_only_for_licence":0,"papers_with_no_sample_that_ran":0,"note":"the per-paper counts above, summed; not a rate"},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks."}