{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/active-learning-of-markov-decision-processes","title":"Active Learning of Markov Decision Processes using Baum-Welch algorithm (Extended)","arxiv_id":"2110.03014","date":"2021-10-06","proceeding":null,"authors":["Giovanni Bacci","Anna Ingólfsdóttir","Kim Larsen","Raphaël Reynouard"],"abstract":"Cyber-physical systems (CPSs) are naturally modelled as reactive systems with nondeterministic and probabilistic dynamics. Model-based verification techniques have proved effective in the deployment of safety-critical CPSs. Central for a successful application of such techniques is the construction of an accurate formal model for the system. Manual construction can be a resource-demanding and error-prone process, thus motivating the design of automata learning algorithms to synthesise a system model from observed system behaviours. This paper revisits and adapts the classic Baum-Welch algorithm for learning Markov decision processes and Markov chains. For the case of MDPs, which typically demand more observations, we present a model-based active learning sampling strategy that choses examples which are most informative w.r.t.\\ the current model hypothesis. We empirically compare our approach with state-of-the-art tools and demonstrate that the proposed active learning procedure can significantly reduce the number of observations required to obtain accurate models.","url_abs":"https://arxiv.org/abs/2110.03014v1","url_pdf":"https://arxiv.org/pdf/2110.03014v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"active-learning-of-markov-decision-processes","repo_url":"https://github.com/Rapfff/Learning-Probabilistic-models","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"active-learning-of-markov-decision-processes","repo_url":"https://github.com/rapfff/jajapy","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"active-learning","task_name":"Active Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2110.03014","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2110.03014"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Rapfff/Learning-Probabilistic-models","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rapfff/jajapy","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":1,"ran_draft_wrong":1,"unverified":6},"by_repo_kind":{"listed":{"samples":8,"ran":2,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"8a0ac0e2a0e0b257","entry":"computeProbas","repo":"Rapfff/Learning-Probabilistic-models","repo_kind":"listed","path":"EM_and_BW_algorithms/src/learning/Active_Learning_MDP.py","file_url":"https://github.com/Rapfff/Learning-Probabilistic-models/blob/HEAD/EM_and_BW_algorithms/src/learning/Active_Learning_MDP.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8a0ac0e2a0e0b257"}},{"code_sha256_prefix":"d6d038cff9ee747d","entry":"folderFromParameters","repo":"Rapfff/Learning-Probabilistic-models","repo_kind":"listed","path":"EM_and_BW_algorithms/experiment/active_vs_passive/huge_experiment.py","file_url":"https://github.com/Rapfff/Learning-Probabilistic-models/blob/HEAD/EM_and_BW_algorithms/experiment/active_vs_passive/huge_experiment.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d6d038cff9ee747d"}},{"code_sha256_prefix":"5b0c0c230cf18ef4","entry":"labelsForRandomModel","repo":"rapfff/jajapy","repo_kind":"listed","path":"jajapy/base/Base_MC.py","file_url":"https://github.com/rapfff/jajapy/blob/HEAD/jajapy/base/Base_MC.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5b0c0c230cf18ef4"}},{"code_sha256_prefix":"a7ff59c573d8a803","entry":"loadParametricModel","repo":"rapfff/jajapy","repo_kind":"listed","path":"jajapy/base/Parametric_Model.py","file_url":"https://github.com/rapfff/jajapy/blob/HEAD/jajapy/base/Parametric_Model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a7ff59c573d8a803"}},{"code_sha256_prefix":"990aa6763c219705","entry":"loadSet","repo":"rapfff/jajapy","repo_kind":"listed","path":"jajapy/base/Set.py","file_url":"https://github.com/rapfff/jajapy/blob/HEAD/jajapy/base/Set.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"990aa6763c219705"}},{"code_sha256_prefix":"8dbd6f57b0c99afb","entry":"normalize","repo":"rapfff/jajapy","repo_kind":"listed","path":"jajapy/base/tools.py","file_url":"https://github.com/rapfff/jajapy/blob/HEAD/jajapy/base/tools.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8dbd6f57b0c99afb"}},{"code_sha256_prefix":"eb6cb2c5ef1ede24","entry":"normpdf","repo":"rapfff/jajapy","repo_kind":"listed","path":"jajapy/base/tools.py","file_url":"https://github.com/rapfff/jajapy/blob/HEAD/jajapy/base/tools.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eb6cb2c5ef1ede24"}},{"code_sha256_prefix":"765918ddb958d369","entry":"resolveRandom","repo":"rapfff/jajapy","repo_kind":"listed","path":"jajapy/base/tools.py","file_url":"https://github.com/rapfff/jajapy/blob/HEAD/jajapy/base/tools.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"765918ddb958d369"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}