{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/semantic-query-by-example-speech-search-using","title":"Semantic query-by-example speech search using visual grounding","arxiv_id":"1904.07078","date":"2019-04-15","proceeding":null,"authors":["Herman Kamper","Aristotelis Anastassiou","Karen Livescu"],"abstract":"A number of recent studies have started to investigate how speech systems can\nbe trained on untranscribed speech by leveraging accompanying images at\ntraining time. Examples of tasks include keyword prediction and within- and\nacross-mode retrieval. Here we consider how such models can be used for\nquery-by-example (QbE) search, the task of retrieving utterances relevant to a\ngiven spoken query. We are particularly interested in semantic QbE, where the\ntask is not only to retrieve utterances containing exact instances of the\nquery, but also utterances whose meaning is relevant to the query. We follow a\nsegmental QbE approach where variable-duration speech segments (queries, search\nutterances) are mapped to fixed-dimensional embedding vectors. We show that a\nQbE system using an embedding function trained on visually grounded speech data\noutperforms a purely acoustic QbE system in terms of both exact and semantic\nretrieval performance.","url_abs":"http://arxiv.org/abs/1904.07078v1","url_pdf":"http://arxiv.org/pdf/1904.07078v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"semantic-query-by-example-speech-search-using","repo_url":"https://github.com/kamperh/flickr_semantic_qbe_eval","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"semantic-retrieval","task_name":"Semantic Retrieval"},{"task_slug":"visual-grounding","task_name":"Visual Grounding"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1904.07078","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1904.07078"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kamperh/flickr_semantic_qbe_eval","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":3},"by_repo_kind":{"listed":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"55b745ff3d7b3330","entry":"calculate_eer","repo":"kamperh/flickr_semantic_qbe_eval","repo_kind":"listed","path":"eval_semantic_qbe.py","file_url":"https://github.com/kamperh/flickr_semantic_qbe_eval/blob/HEAD/eval_semantic_qbe.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"55b745ff3d7b3330"}},{"code_sha256_prefix":"542e7c5d114c0d4a","entry":"eval_qbe","repo":"kamperh/flickr_semantic_qbe_eval","repo_kind":"listed","path":"eval_semantic_qbe.py","file_url":"https://github.com/kamperh/flickr_semantic_qbe_eval/blob/HEAD/eval_semantic_qbe.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"542e7c5d114c0d4a"}},{"code_sha256_prefix":"a8deef93e78a325c","entry":"get_captions_dict","repo":"kamperh/flickr_semantic_qbe_eval","repo_kind":"listed","path":"eval_semantic_qbe.py","file_url":"https://github.com/kamperh/flickr_semantic_qbe_eval/blob/HEAD/eval_semantic_qbe.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a8deef93e78a325c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}