{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/analyzing-analytical-methods-the-case-of","title":"Analyzing analytical methods: The case of phonology in neural models of spoken language","arxiv_id":"2004.07070","date":"2020-04-15","proceeding":"ACL 2020 6","authors":["Grzegorz Chrupała","Bertrand Higy","Afra Alishahi"],"abstract":"Given the fast development of analysis techniques for NLP and speech processing systems, few systematic studies have been conducted to compare the strengths and weaknesses of each method. As a step in this direction we study the case of representations of phonology in neural network models of spoken language. We use two commonly applied analytical techniques, diagnostic classifiers and representational similarity analysis, to quantify to what extent neural activation patterns encode phonemes and phoneme sequences. We manipulate two factors that can affect the outcome of analysis. First, we investigate the role of learning by comparing neural activations extracted from trained versus randomly-initialized models. Second, we examine the temporal scope of the activations by probing both local activations corresponding to a few milliseconds of the speech signal, and global activations pooled over the whole utterance. We conclude that reporting analysis results with randomly initialized models is crucial, and that global-scope methods tend to yield more consistent results and we recommend their use as a complement to local-scope diagnostic methods.","url_abs":"https://arxiv.org/abs/2004.07070v2","url_pdf":"https://arxiv.org/pdf/2004.07070v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"analyzing-analytical-methods-the-case-of","repo_url":"https://github.com/gchrupala/analyzing-analytical-methods","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"diagnostic","task_name":"Diagnostic"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2004.07070","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2004.07070"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gchrupala/analyzing-analytical-methods","reach":null}],"summary":{"ran_violates":1,"ran_draft_wrong":2,"ran_honours":2,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"de1a270e914accbe","entry":"good_alignment","repo":"gchrupala/analyzing-analytical-methods","repo_kind":"official","path":"prepare.py","file_url":"https://github.com/gchrupala/analyzing-analytical-methods/blob/HEAD/prepare.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"de1a270e914accbe"}},{"code_sha256_prefix":"708c50f766cc356e","entry":"load_alignment","repo":"gchrupala/analyzing-analytical-methods","repo_kind":"official","path":"prepare.py","file_url":"https://github.com/gchrupala/analyzing-analytical-methods/blob/HEAD/prepare.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"708c50f766cc356e"}},{"code_sha256_prefix":"388d00d6f54e27cf","entry":"majority_binary","repo":"gchrupala/analyzing-analytical-methods","repo_kind":"official","path":"analyze.py","file_url":"https://github.com/gchrupala/analyzing-analytical-methods/blob/HEAD/analyze.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"388d00d6f54e27cf"}},{"code_sha256_prefix":"16cfc7c70c1d8c4f","entry":"majority_multiclass","repo":"gchrupala/analyzing-analytical-methods","repo_kind":"official","path":"analyze.py","file_url":"https://github.com/gchrupala/analyzing-analytical-methods/blob/HEAD/analyze.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"16cfc7c70c1d8c4f"}},{"code_sha256_prefix":"203eb74fe03dd23d","entry":"make_indexer","repo":"gchrupala/analyzing-analytical-methods","repo_kind":"official","path":"prepare.py","file_url":"https://github.com/gchrupala/analyzing-analytical-methods/blob/HEAD/prepare.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"203eb74fe03dd23d"}},{"code_sha256_prefix":"992f82f94d81d237","entry":"local_classifier","repo":"gchrupala/analyzing-analytical-methods","repo_kind":"official","path":"analyze.py","file_url":"https://github.com/gchrupala/analyzing-analytical-methods/blob/HEAD/analyze.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"992f82f94d81d237"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}