{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-latent-parameters-without-human","title":"Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds","arxiv_id":"1908.11421","date":"2019-08-29","proceeding":"IJCNLP 2019 11","authors":["John P. Lalor","Hao Wu","Hong Yu"],"abstract":"Incorporating Item Response Theory (IRT) into NLP tasks can provide valuable information about model performance and behavior. Traditionally, IRT models are learned using human response pattern (RP) data, presenting a significant bottleneck for large data sets like those required for training deep neural networks (DNNs). In this work we propose learning IRT models using RPs generated from artificial crowds of DNN models. We demonstrate the effectiveness of learning IRT models using DNN-generated data through quantitative and qualitative analyses for two NLP tasks. Parameters learned from human and machine RPs for natural language inference and sentiment analysis exhibit medium to large positive correlations. We demonstrate a use-case for latent difficulty item parameters, namely training set filtering, and show that using difficulty to sample training data outperforms baseline methods. Finally, we highlight cases where human expectation about item difficulty does not match difficulty as estimated from the machine RPs.","url_abs":"https://arxiv.org/abs/1908.11421v1","url_pdf":"https://arxiv.org/pdf/1908.11421v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-latent-parameters-without-human","repo_url":"https://github.com/jplalor/py-irt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"sentiment-analysis","task_name":"Sentiment Analysis"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1908.11421","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1908.11421"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jplalor/py-irt","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":5},"by_repo_kind":{"official":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8b831d008aa4e817","entry":"create_anchor_gradient_zeroer","repo":"jplalor/py-irt","repo_kind":"official","path":"py_irt/anchor_utils.py","file_url":"https://github.com/jplalor/py-irt/blob/HEAD/py_irt/anchor_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8b831d008aa4e817"}},{"code_sha256_prefix":"b8fe10303ae293f8","entry":"read_json","repo":"jplalor/py-irt","repo_kind":"official","path":"py_irt/io.py","file_url":"https://github.com/jplalor/py-irt/blob/HEAD/py_irt/io.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b8fe10303ae293f8"}},{"code_sha256_prefix":"497e934f0ffd8827","entry":"read_jsonlines","repo":"jplalor/py-irt","repo_kind":"official","path":"py_irt/io.py","file_url":"https://github.com/jplalor/py-irt/blob/HEAD/py_irt/io.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"497e934f0ffd8827"}},{"code_sha256_prefix":"06eee8f34be53db4","entry":"register","repo":"jplalor/py-irt","repo_kind":"official","path":"py_irt/initializers.py","file_url":"https://github.com/jplalor/py-irt/blob/HEAD/py_irt/initializers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"06eee8f34be53db4"}},{"code_sha256_prefix":"c34bf8c144fbba30","entry":"safe_file","repo":"jplalor/py-irt","repo_kind":"official","path":"py_irt/io.py","file_url":"https://github.com/jplalor/py-irt/blob/HEAD/py_irt/io.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c34bf8c144fbba30"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}