{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/anomaly-detection-using-autoencoders-in-high","title":"Anomaly Detection using Autoencoders in High Performance Computing Systems","arxiv_id":"1811.05269","date":"2018-11-13","proceeding":null,"authors":["Andrea Borghesi","Andrea Bartolini","Michele Lombardi","Michela Milano","Luca Benini"],"abstract":"Anomaly detection in supercomputers is a very difficult problem due to the\nbig scale of the systems and the high number of components. The current state\nof the art for automated anomaly detection employs Machine Learning methods or\nstatistical regression models in a supervised fashion, meaning that the\ndetection tool is trained to distinguish among a fixed set of behaviour classes\n(healthy and unhealthy states).\n  We propose a novel approach for anomaly detection in High Performance\nComputing systems based on a Machine (Deep) Learning technique, namely a type\nof neural network called autoencoder. The key idea is to train a set of\nautoencoders to learn the normal (healthy) behaviour of the supercomputer nodes\nand, after training, use them to identify abnormal conditions. This is\ndifferent from previous approaches which where based on learning the abnormal\ncondition, for which there are much smaller datasets (since it is very hard to\nidentify them to begin with).\n  We test our approach on a real supercomputer equipped with a fine-grained,\nscalable monitoring infrastructure that can provide large amount of data to\ncharacterize the system behaviour. The results are extremely promising: after\nthe training phase to learn the normal system behaviour, our method is capable\nof detecting anomalies that have never been seen before with a very good\naccuracy (values ranging between 88% and 96%).","url_abs":"http://arxiv.org/abs/1811.05269v1","url_pdf":"http://arxiv.org/pdf/1811.05269v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"anomaly-detection-using-autoencoders-in-high","repo_url":"https://github.com/AndreaBorghesi/anomaly_detection_HPC","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"anomaly-detection-using-autoencoders-in-high","repo_url":"https://github.com/LogAnalysisTeam/loglizer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"anomaly-detection-using-autoencoders-in-high","repo_url":"https://github.com/Young-in/ANM-Assignment2-loglizer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"anomaly-detection-using-autoencoders-in-high","repo_url":"https://github.com/logpai/loglizer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"anomaly-detection-using-autoencoders-in-high","repo_url":"https://github.com/sgamezrdo/anomaly-aoutoencoding","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"anomaly-detection","task_name":"Anomaly Detection"},{"task_slug":"high","task_name":"Vocal Bursts Intensity Prediction"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}