{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2602-07143","title":"Massive Sound Embedding Benchmark (MSEB)","arxiv_id":"2602.07143","date":"2026-02-06","proceeding":null,"authors":["Georg Heigold","Ehsan Variani","Tom Bagby","Cyril Allauzen","Ji Ma","Shankar Kumar","Michael Riley"],"abstract":"Audio is a critical component of multimodal perception, and any truly intelligent system must demonstrate a wide range of auditory capabilities. These capabilities include transcription, classification, retrieval, reasoning, segmentation, clustering, reranking, and reconstruction. Fundamentally, each task involves transforming a raw audio signal into a meaningful 'embedding' - be it a single vector, a sequence of continuous or discrete representations, or another structured form - which then serves as the basis for generating the task's final response. To accelerate progress towards robust machine auditory intelligence, we present the Massive Sound Embedding Benchmark (MSEB): an extensible framework designed to evaluate the auditory components of any multimodal system. In its first release, MSEB offers a comprehensive suite of eight core tasks, with more planned for the future, supported by diverse datasets, including the new, large-scale Simple Voice Questions (SVQ) dataset. Our initial experiments establish clear performance headrooms, highlighting the significant opportunity to improve real-world multimodal experiences where audio is a core signal. We encourage the research community to use MSEB to assess their algorithms and contribute to its growth. The library is publicly hosted at github.","url_abs":"https://arxiv.org/abs/2602.07143","url_pdf":"https://arxiv.org/pdf/2602.07143","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2602.07143","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2602.07143"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/google-research/mseb","reach":null}],"summary":{"unverified":8},"by_repo_kind":{"found_in_text":{"samples":8,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2039ed771f5c6e3c","entry":"compute_average_precision","repo":"google-research/mseb","repo_kind":"found_in_text","path":"mseb/metrics.py","file_url":"https://github.com/google-research/mseb/blob/HEAD/mseb/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2039ed771f5c6e3c"}},{"code_sha256_prefix":"db44f20e41014fcf","entry":"compute_exact_match","repo":"google-research/mseb","repo_kind":"found_in_text","path":"mseb/metrics.py","file_url":"https://github.com/google-research/mseb/blob/HEAD/mseb/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"db44f20e41014fcf"}},{"code_sha256_prefix":"9f778f299692103e","entry":"compute_reciprocal_rank","repo":"google-research/mseb","repo_kind":"found_in_text","path":"mseb/metrics.py","file_url":"https://github.com/google-research/mseb/blob/HEAD/mseb/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9f778f299692103e"}},{"code_sha256_prefix":"02b1383aeb11c226","entry":"get_base_path","repo":"google-research/mseb","repo_kind":"found_in_text","path":"mseb/dataset.py","file_url":"https://github.com/google-research/mseb/blob/HEAD/mseb/dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"02b1383aeb11c226"}},{"code_sha256_prefix":"c2fd7527699739bd","entry":"get_num_shards","repo":"google-research/mseb","repo_kind":"found_in_text","path":"mseb/runner.py","file_url":"https://github.com/google-research/mseb/blob/HEAD/mseb/runner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c2fd7527699739bd"}},{"code_sha256_prefix":"0b378aacda0c34a4","entry":"list_hf_files","repo":"google-research/mseb","repo_kind":"found_in_text","path":"mseb/utils.py","file_url":"https://github.com/google-research/mseb/blob/HEAD/mseb/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0b378aacda0c34a4"}},{"code_sha256_prefix":"52f384c4d6baf04b","entry":"parse_resource_hints","repo":"google-research/mseb","repo_kind":"found_in_text","path":"mseb/runner.py","file_url":"https://github.com/google-research/mseb/blob/HEAD/mseb/runner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"52f384c4d6baf04b"}},{"code_sha256_prefix":"472187cb76b518be","entry":"read_hf_parquet","repo":"google-research/mseb","repo_kind":"found_in_text","path":"mseb/utils.py","file_url":"https://github.com/google-research/mseb/blob/HEAD/mseb/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"472187cb76b518be"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,264 of the 9,581 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9581,"papers_checked":6264},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}