{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/an-open-source-implementation-of-itu-t","title":"An Open source Implementation of ITU-T Recommendation P.808 with Validation","arxiv_id":"2005.08138","date":"2020-05-17","proceeding":null,"authors":["Babak Naderi","Ross Cutler"],"abstract":"The ITU-T Recommendation P.808 provides a crowdsourcing approach for conducting a subjective assessment of speech quality using the Absolute Category Rating (ACR) method. We provide an open-source implementation of the ITU-T Rec. P.808 that runs on the Amazon Mechanical Turk platform. We extended our implementation to include Degradation Category Ratings (DCR) and Comparison Category Ratings (CCR) test methods. We also significantly speed up the test process by integrating the participant qualification step into the main rating task compared to a two-stage qualification and rating solution. We provide program scripts for creating and executing the subjective test, and data cleansing and analyzing the answers to avoid operational errors. To validate the implementation, we compare the Mean Opinion Scores (MOS) collected through our implementation with MOS values from a standard laboratory experiment conducted based on the ITU-T Rec. P.800. We also evaluate the reproducibility of the result of the subjective speech quality assessment through crowdsourcing using our implementation. Finally, we quantify the impact of parts of the system designed to improve the reliability: environmental tests, gold and trapping questions, rating patterns, and a headset usage test.","url_abs":"https://arxiv.org/abs/2005.08138v1","url_pdf":"https://arxiv.org/pdf/2005.08138v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"an-open-source-implementation-of-itu-t","repo_url":"https://github.com/microsoft/P.808","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2005.08138","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2005.08138"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/P.808","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":21},"by_repo_kind":{"official":{"samples":21,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e88fec7c0329bda1","entry":"active_speech_level_dbov","repo":"microsoft/P.808","repo_kind":"official","path":"src/utils/create_bandwidth_check_clips.py","file_url":"https://github.com/microsoft/P.808/blob/HEAD/src/utils/create_bandwidth_check_clips.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e88fec7c0329bda1"}},{"code_sha256_prefix":"e7b53d1bd52460bb","entry":"add_background_noise","repo":"microsoft/P.808","repo_kind":"official","path":"src/create_gold_clips.py","file_url":"https://github.com/microsoft/P.808/blob/HEAD/src/create_gold_clips.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7b53d1bd52460bb"}},{"code_sha256_prefix":"c2e664e9c05dee61","entry":"add_clips_balanced_block","repo":"microsoft/P.808","repo_kind":"official","path":"src/create_input.py","file_url":"https://github.com/microsoft/P.808/blob/HEAD/src/create_input.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c2e664e9c05dee61"}},{"code_sha256_prefix":"75759035bd91fc72","entry":"apply_signal_distortion","repo":"microsoft/P.808","repo_kind":"official","path":"src/create_gold_clips.py","file_url":"https://github.com/microsoft/P.808/blob/HEAD/src/create_gold_clips.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"75759035bd91fc72"}},{"code_sha256_prefix":"215b698494381522","entry":"check_last_subjective_test","repo":"microsoft/P.808","repo_kind":"official","path":"src/process_qualification_ans.py","file_url":"https://github.com/microsoft/P.808/blob/HEAD/src/process_qualification_ans.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"215b698494381522"}},{"code_sha256_prefix":"ead9249c30d8a2b0","entry":"check_listening_device","repo":"microsoft/P.808","repo_kind":"official","path":"src/process_qualification_ans.py","file_url":"https://github.com/microsoft/P.808/blob/HEAD/src/process_qualification_ans.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ead9249c30d8a2b0"}},{"code_sha256_prefix":"e668d00f5d997ea8","entry":"check_mother_tongue","repo":"microsoft/P.808","repo_kind":"official","path":"src/process_qualification_ans.py","file_url":"https://github.com/microsoft/P.808/blob/HEAD/src/process_qualification_ans.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e668d00f5d997ea8"}},{"code_sha256_prefix":"9907d254ba4dd07a","entry":"conv_filename_to_condition","repo":"microsoft/P.808","repo_kind":"official","path":"src/create_input.py","file_url":"https://github.com/microsoft/P.808/blob/HEAD/src/create_input.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9907d254ba4dd07a"}},{"code_sha256_prefix":"9138fb8858a1a02d","entry":"create_input_for_acr","repo":"microsoft/P.808","repo_kind":"official","path":"src/create_input.py","file_url":"https://github.com/microsoft/P.808/blob/HEAD/src/create_input.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9138fb8858a1a02d"}},{"code_sha256_prefix":"3849af706ed5ed7c","entry":"generate_pink_noise","repo":"microsoft/P.808","repo_kind":"official","path":"src/create_gold_clips.py","file_url":"https://github.com/microsoft/P.808/blob/HEAD/src/create_gold_clips.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3849af706ed5ed7c"}},{"code_sha256_prefix":"5ff16b3827081645","entry":"get_encoded_gold_ans","repo":"microsoft/P.808","repo_kind":"official","path":"src/master_script.py","file_url":"https://github.com/microsoft/P.808/blob/HEAD/src/master_script.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5ff16b3827081645"}},{"code_sha256_prefix":"710e73f2301ec75e","entry":"normalize_to_dbfs","repo":"microsoft/P.808","repo_kind":"official","path":"src/utils/create_jnd_check_clips.py","file_url":"https://github.com/microsoft/P.808/blob/HEAD/src/utils/create_jnd_check_clips.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"710e73f2301ec75e"}},{"code_sha256_prefix":"977672f85d76ee83","entry":"outliers_modified_z_score","repo":"microsoft/P.808","repo_kind":"official","path":"src/result_parser.py","file_url":"https://github.com/microsoft/P.808/blob/HEAD/src/result_parser.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"977672f85d76ee83"}},{"code_sha256_prefix":"88aa2735cc86e2c0","entry":"outliers_z_score","repo":"microsoft/P.808","repo_kind":"official","path":"src/result_parser.py","file_url":"https://github.com/microsoft/P.808/blob/HEAD/src/result_parser.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"88aa2735cc86e2c0"}},{"code_sha256_prefix":"66305b6c700495fc","entry":"rel_path","repo":"microsoft/P.808","repo_kind":"official","path":"src/result_parser.py","file_url":"https://github.com/microsoft/P.808/blob/HEAD/src/result_parser.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"66305b6c700495fc"}},{"code_sha256_prefix":"7c90db5cb9981e1a","entry":"rms","repo":"microsoft/P.808","repo_kind":"official","path":"src/utils/create_bandwidth_check_clips.py","file_url":"https://github.com/microsoft/P.808/blob/HEAD/src/utils/create_bandwidth_check_clips.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7c90db5cb9981e1a"}},{"code_sha256_prefix":"56c0e3f3a494ebfd","entry":"scale_to_dbov","repo":"microsoft/P.808","repo_kind":"official","path":"src/utils/create_bandwidth_check_clips.py","file_url":"https://github.com/microsoft/P.808/blob/HEAD/src/utils/create_bandwidth_check_clips.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"56c0e3f3a494ebfd"}},{"code_sha256_prefix":"d5407098010eab79","entry":"snr_mixer","repo":"microsoft/P.808","repo_kind":"official","path":"src/utils/create_jnd_check_clips.py","file_url":"https://github.com/microsoft/P.808/blob/HEAD/src/utils/create_jnd_check_clips.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d5407098010eab79"}},{"code_sha256_prefix":"958cb86721b252f3","entry":"speech_stats_for_audio","repo":"microsoft/P.808","repo_kind":"official","path":"src/utils/detect_silence_vad.py","file_url":"https://github.com/microsoft/P.808/blob/HEAD/src/utils/detect_silence_vad.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"958cb86721b252f3"}},{"code_sha256_prefix":"7ba5f9e783c82e4a","entry":"update_gold_clips_for_p804","repo":"microsoft/P.808","repo_kind":"official","path":"src/master_script.py","file_url":"https://github.com/microsoft/P.808/blob/HEAD/src/master_script.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7ba5f9e783c82e4a"}},{"code_sha256_prefix":"dbd5dc9894a7d943","entry":"update_gold_clips_for_personalized","repo":"microsoft/P.808","repo_kind":"official","path":"src/master_script.py","file_url":"https://github.com/microsoft/P.808/blob/HEAD/src/master_script.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dbd5dc9894a7d943"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}