{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/prego-online-mistake-detection-in-procedural","title":"PREGO: online mistake detection in PRocedural EGOcentric videos","arxiv_id":"2404.01933","date":"2024-04-02","proceeding":"CVPR 2024 1","authors":["Alessandro Flaborea","Guido Maria D'Amely di Melendugno","Leonardo Plini","Luca Scofano","Edoardo De Matteis","Antonino Furnari","Giovanni Maria Farinella","Fabio Galasso"],"abstract":"Promptly identifying procedural errors from egocentric videos in an online setting is highly challenging and valuable for detecting mistakes as soon as they happen. This capability has a wide range of applications across various fields, such as manufacturing and healthcare. The nature of procedural mistakes is open-set since novel types of failures might occur, which calls for one-class classifiers trained on correctly executed procedures. However, no technique can currently detect open-set procedural mistakes online. We propose PREGO, the first online one-class classification model for mistake detection in PRocedural EGOcentric videos. PREGO is based on an online action recognition component to model the current action, and a symbolic reasoning module to predict the next actions. Mistake detection is performed by comparing the recognized current action with the expected future one. We evaluate PREGO on two procedural egocentric video datasets, Assembly101 and Epic-tent, which we adapt for online benchmarking of procedural mistake detection to establish suitable benchmarks, thus defining the Assembly101-O and Epic-tent-O datasets, respectively.","url_abs":"https://arxiv.org/abs/2404.01933v2","url_pdf":"https://arxiv.org/pdf/2404.01933v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"prego-online-mistake-detection-in-procedural","repo_url":"https://github.com/aleflabo/prego","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"mistake-detection","task_name":"Mistake Detection"},{"task_slug":"one-class-classification","task_name":"One-Class Classification"},{"task_slug":"online-mistake-detection","task_name":"Online Mistake Detection"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"llama","method_name":"LLaMA"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2404.01933","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.01933"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aleflabo/prego","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/aleflabo/PREGO","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6,"ran_fixture":3,"ran_violates":1,"unverified":1},"by_repo_kind":{"official":{"samples":11,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"21b59364b1cadbc8","entry":"apply_rotary_emb","repo":"aleflabo/PREGO","repo_kind":"official","path":"step_anticipation/llama/model.py","file_url":"https://github.com/aleflabo/PREGO/blob/HEAD/step_anticipation/llama/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"21b59364b1cadbc8"}},{"code_sha256_prefix":"d003debd2df6cd68","entry":"eliminate_consecutive_duplicates","repo":"aleflabo/PREGO","repo_kind":"official","path":"utils/aggregate.py","file_url":"https://github.com/aleflabo/PREGO/blob/HEAD/utils/aggregate.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d003debd2df6cd68"}},{"code_sha256_prefix":"481e169e4642ae46","entry":"find_changes","repo":"aleflabo/PREGO","repo_kind":"official","path":"utils/aggregate.py","file_url":"https://github.com/aleflabo/PREGO/blob/HEAD/utils/aggregate.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"481e169e4642ae46"}},{"code_sha256_prefix":"ded3bc4e3cdab571","entry":"get_metrics","repo":"aleflabo/PREGO","repo_kind":"official","path":"step_anticipation/src/models/llama_meta.py","file_url":"https://github.com/aleflabo/PREGO/blob/HEAD/step_anticipation/src/models/llama_meta.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ded3bc4e3cdab571"}},{"code_sha256_prefix":"cc516edf16453625","entry":"get_toy","repo":"aleflabo/PREGO","repo_kind":"official","path":"step_anticipation/src/models/llama_meta.py","file_url":"https://github.com/aleflabo/PREGO/blob/HEAD/step_anticipation/src/models/llama_meta.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cc516edf16453625"}},{"code_sha256_prefix":"5c2877e8bd1ddf2e","entry":"get_toy","repo":"aleflabo/PREGO","repo_kind":"official","path":"step_anticipation/src/models/llm_hf.py","file_url":"https://github.com/aleflabo/PREGO/blob/HEAD/step_anticipation/src/models/llm_hf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5c2877e8bd1ddf2e"}},{"code_sha256_prefix":"5c04ac9160c4312f","entry":"load_data","repo":"aleflabo/PREGO","repo_kind":"official","path":"step_anticipation/src/models/llama_meta.py","file_url":"https://github.com/aleflabo/PREGO/blob/HEAD/step_anticipation/src/models/llama_meta.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5c04ac9160c4312f"}},{"code_sha256_prefix":"2fcc0c590f063c0c","entry":"load_data","repo":"aleflabo/PREGO","repo_kind":"official","path":"step_anticipation/src/models/llm_hf.py","file_url":"https://github.com/aleflabo/PREGO/blob/HEAD/step_anticipation/src/models/llm_hf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2fcc0c590f063c0c"}},{"code_sha256_prefix":"9aef5cccb42aa073","entry":"precompute_freqs_cis","repo":"aleflabo/PREGO","repo_kind":"official","path":"step_anticipation/llama/model.py","file_url":"https://github.com/aleflabo/PREGO/blob/HEAD/step_anticipation/llama/model.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9aef5cccb42aa073"}},{"code_sha256_prefix":"5a639d78ada17fee","entry":"reshape_for_broadcast","repo":"aleflabo/PREGO","repo_kind":"official","path":"step_anticipation/llama/model.py","file_url":"https://github.com/aleflabo/PREGO/blob/HEAD/step_anticipation/llama/model.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5a639d78ada17fee"}},{"code_sha256_prefix":"e28848558038eee4","entry":"sample_top_p","repo":"aleflabo/PREGO","repo_kind":"official","path":"step_anticipation/llama/generation.py","file_url":"https://github.com/aleflabo/PREGO/blob/HEAD/step_anticipation/llama/generation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e28848558038eee4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}