{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dreem-open-datasets-multi-scored-sleep","title":"Dreem Open Datasets: Multi-Scored Sleep Datasets to compare Human and Automated sleep staging","arxiv_id":"1911.03221","date":"2019-10-31","proceeding":null,"authors":["Antoine Guillot","Fabien Sauvet","Emmanuel H. During","Valentin Thorey"],"abstract":"Sleep stage classification constitutes an important element of sleep disorder diagnosis. It relies on the visual inspection of polysomnography records by trained sleep technologists. Automated approaches have been designed to alleviate this resource-intensive task. However, such approaches are usually compared to a single human scorer annotation despite an inter-rater agreement of about 85 % only. The present study introduces two publicly-available datasets, DOD-H including 25 healthy volunteers and DOD-O including 55 patients suffering from obstructive sleep apnea (OSA). Both datasets have been scored by 5 sleep technologists from different sleep centers. We developed a framework to compare automated approaches to a consensus of multiple human scorers. Using this framework, we benchmarked and compared the main literature approaches. We also developed and benchmarked a new deep learning method, SimpleSleepNet, inspired by current state-of-the-art. We demonstrated that many methods can reach human-level performance on both datasets. SimpleSleepNet achieved an F1 of 89.9 % vs 86.8 % on average for human scorers on DOD-H, and an F1 of 88.3 % vs 84.8 % on DOD-O. Our study highlights that using state-of-the-art automated sleep staging outperforms human scorers performance for healthy volunteers and patients suffering from OSA. Consideration could be made to use automated approaches in the clinical setting.","url_abs":"https://arxiv.org/abs/1911.03221v4","url_pdf":"https://arxiv.org/pdf/1911.03221v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dreem-open-datasets-multi-scored-sleep","repo_url":"https://github.com/Dreem-Organization/dreem-learning-evaluation","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null},{"paper_slug":"dreem-open-datasets-multi-scored-sleep","repo_url":"https://github.com/Dreem-Organization/dreem-learning-open","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"automatic-sleep-stage-classification","task_name":"Automatic Sleep Stage Classification"},{"task_slug":"multimodal-sleep-stage-detection","task_name":"Multimodal Sleep Stage Detection"},{"task_slug":"sleep-stage-detection","task_name":"Sleep Stage Detection"},{"task_slug":"sleep-staging","task_name":"Sleep Staging"}],"methods":[{"method_slug":"gru","method_name":"GRU"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/sleep-stage-detection-on-dodh","task":"Sleep Stage Detection","dataset":"DODH","model":"SimpleSleepNet","rank_in_archive_order":1,"of":2,"metrics":{"Accuracy":"89.9","Kappa":"84.6"},"uses_additional_data":false},{"leaderboard":"/sota/sleep-stage-detection-on-dodh","task":"Sleep Stage Detection","dataset":"DODH","model":"DeepSleepNet","rank_in_archive_order":2,"of":2,"metrics":{"Accuracy":"89.6","Kappa":"84.3"},"uses_additional_data":false},{"leaderboard":"/sota/sleep-stage-detection-on-dodo","task":"Sleep Stage Detection","dataset":"DODO","model":"SimpleSleepNet","rank_in_archive_order":1,"of":3,"metrics":{"Accuracy":"88.7","Kappa":"82.3"},"uses_additional_data":false},{"leaderboard":"/sota/sleep-stage-detection-on-dodo","task":"Sleep Stage Detection","dataset":"DODO","model":"DeepSleepNet","rank_in_archive_order":2,"of":3,"metrics":{"Accuracy":"87.5","Kappa":"80.4"},"uses_additional_data":false},{"leaderboard":"/sota/sleep-stage-detection-on-dodo","task":"Sleep Stage Detection","dataset":"DODO","model":"SeqSleepNet","rank_in_archive_order":3,"of":3,"metrics":{"Accuracy":"85.5","Kappa":"77.2"},"uses_additional_data":false},{"leaderboard":"/sota/sleep-stage-detection-on-mass-ss3","task":"Sleep Stage Detection","dataset":"MASS SS3","model":"Deep Sleep Net","rank_in_archive_order":1,"of":6,"metrics":{"Accuracy":"89.1%"},"uses_additional_data":false},{"leaderboard":"/sota/sleep-stage-detection-on-mass-ss3","task":"Sleep Stage Detection","dataset":"MASS SS3","model":"Simple Sleep Net","rank_in_archive_order":2,"of":6,"metrics":{"Accuracy":"88.8%"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/1911.03221","atlas_url":"https://app.syntology.ai/?focus=1911.03221","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1911.03221"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Dreem-Organization/dreem-learning-evaluation","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Dreem-Organization/dreem-learning-open","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":3,"unverified":19},"by_repo_kind":{"official":{"samples":22,"ran":3,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"918adff8add965aa","entry":"build_consensus_hypnogram","repo":"Dreem-Organization/dreem-learning-evaluation","repo_kind":"official","path":"evaluation.py","file_url":"https://github.com/Dreem-Organization/dreem-learning-evaluation/blob/HEAD/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"918adff8add965aa"}},{"code_sha256_prefix":"302691ee73a2e048","entry":"compute_night_metrics","repo":"Dreem-Organization/dreem-learning-evaluation","repo_kind":"official","path":"evaluation.py","file_url":"https://github.com/Dreem-Organization/dreem-learning-evaluation/blob/HEAD/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"302691ee73a2e048"}},{"code_sha256_prefix":"f92cfbe72d68e6b8","entry":"compute_soft_agreement","repo":"Dreem-Organization/dreem-learning-evaluation","repo_kind":"official","path":"evaluation.py","file_url":"https://github.com/Dreem-Organization/dreem-learning-evaluation/blob/HEAD/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f92cfbe72d68e6b8"}},{"code_sha256_prefix":"48b2a11244c8eec9","entry":"affine","repo":"Dreem-Organization/dreem-learning-open","repo_kind":"official","path":"dreem_learning_open/models/modulo_net/normalization.py","file_url":"https://github.com/Dreem-Organization/dreem-learning-open/blob/HEAD/dreem_learning_open/models/modulo_net/normalization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"48b2a11244c8eec9"}},{"code_sha256_prefix":"8ae689bec5854251","entry":"augment_data","repo":"Dreem-Organization/dreem-learning-open","repo_kind":"official","path":"dreem_learning_open/datasets/data_augmentation.py","file_url":"https://github.com/Dreem-Organization/dreem-learning-open/blob/HEAD/dreem_learning_open/datasets/data_augmentation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8ae689bec5854251"}},{"code_sha256_prefix":"867d2c4a9b3665e3","entry":"clip","repo":"Dreem-Organization/dreem-learning-open","repo_kind":"official","path":"dreem_learning_open/models/modulo_net/normalization.py","file_url":"https://github.com/Dreem-Organization/dreem-learning-open/blob/HEAD/dreem_learning_open/models/modulo_net/normalization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"867d2c4a9b3665e3"}},{"code_sha256_prefix":"7e53985f41c0c2e4","entry":"cohen_kappa_custom","repo":"Dreem-Organization/dreem-learning-open","repo_kind":"official","path":"dreem_learning_open/utils/scores.py","file_url":"https://github.com/Dreem-Organization/dreem-learning-open/blob/HEAD/dreem_learning_open/utils/scores.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e53985f41c0c2e4"}},{"code_sha256_prefix":"1161728228a99af7","entry":"compute_epoch_features","repo":"Dreem-Organization/dreem-learning-open","repo_kind":"official","path":"dreem_learning_open/preprocessings/h5_to_memmap.py","file_url":"https://github.com/Dreem-Organization/dreem-learning-open/blob/HEAD/dreem_learning_open/preprocessings/h5_to_memmap.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1161728228a99af7"}},{"code_sha256_prefix":"856fb590eba3e24d","entry":"compute_memmap_hash","repo":"Dreem-Organization/dreem-learning-open","repo_kind":"official","path":"dreem_learning_open/preprocessings/h5_to_memmap.py","file_url":"https://github.com/Dreem-Organization/dreem-learning-open/blob/HEAD/dreem_learning_open/preprocessings/h5_to_memmap.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"856fb590eba3e24d"}},{"code_sha256_prefix":"248d2cceb9938a91","entry":"cyclic_index_window","repo":"Dreem-Organization/dreem-learning-open","repo_kind":"official","path":"dreem_learning_open/preprocessings/epoch_features_processing.py","file_url":"https://github.com/Dreem-Organization/dreem-learning-open/blob/HEAD/dreem_learning_open/preprocessings/epoch_features_processing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"248d2cceb9938a91"}},{"code_sha256_prefix":"a377531334afcb0a","entry":"f1_custom","repo":"Dreem-Organization/dreem-learning-open","repo_kind":"official","path":"dreem_learning_open/utils/scores.py","file_url":"https://github.com/Dreem-Organization/dreem-learning-open/blob/HEAD/dreem_learning_open/utils/scores.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a377531334afcb0a"}},{"code_sha256_prefix":"17f77ead9e021fe4","entry":"fft","repo":"Dreem-Organization/dreem-learning-open","repo_kind":"official","path":"dreem_learning_open/preprocessings/epoch_features_processing.py","file_url":"https://github.com/Dreem-Organization/dreem-learning-open/blob/HEAD/dreem_learning_open/preprocessings/epoch_features_processing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"17f77ead9e021fe4"}},{"code_sha256_prefix":"9ccdc341527e4e3d","entry":"filter","repo":"Dreem-Organization/dreem-learning-open","repo_kind":"official","path":"dreem_learning_open/preprocessings/signal_processing.py","file_url":"https://github.com/Dreem-Organization/dreem-learning-open/blob/HEAD/dreem_learning_open/preprocessings/signal_processing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9ccdc341527e4e3d"}},{"code_sha256_prefix":"02e50ac0d452d271","entry":"filter_base","repo":"Dreem-Organization/dreem-learning-open","repo_kind":"official","path":"dreem_learning_open/preprocessings/signal_processing.py","file_url":"https://github.com/Dreem-Organization/dreem-learning-open/blob/HEAD/dreem_learning_open/preprocessings/signal_processing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"02e50ac0d452d271"}},{"code_sha256_prefix":"282ac4f0e343a8bf","entry":"iir_bandpass_filter","repo":"Dreem-Organization/dreem-learning-open","repo_kind":"official","path":"dreem_learning_open/preprocessings/signal_processing.py","file_url":"https://github.com/Dreem-Organization/dreem-learning-open/blob/HEAD/dreem_learning_open/preprocessings/signal_processing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"282ac4f0e343a8bf"}},{"code_sha256_prefix":"be0b88d3dbb9fec8","entry":"index_window","repo":"Dreem-Organization/dreem-learning-open","repo_kind":"official","path":"dreem_learning_open/preprocessings/epoch_features_processing.py","file_url":"https://github.com/Dreem-Organization/dreem-learning-open/blob/HEAD/dreem_learning_open/preprocessings/epoch_features_processing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"be0b88d3dbb9fec8"}},{"code_sha256_prefix":"c7091685d495bb47","entry":"inference_on_dataset","repo":"Dreem-Organization/dreem-learning-open","repo_kind":"official","path":"dreem_learning_open/logger/logger.py","file_url":"https://github.com/Dreem-Organization/dreem-learning-open/blob/HEAD/dreem_learning_open/logger/logger.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c7091685d495bb47"}},{"code_sha256_prefix":"b7851f924aab2ec7","entry":"kill_channel","repo":"Dreem-Organization/dreem-learning-open","repo_kind":"official","path":"dreem_learning_open/datasets/data_augmentation.py","file_url":"https://github.com/Dreem-Organization/dreem-learning-open/blob/HEAD/dreem_learning_open/datasets/data_augmentation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b7851f924aab2ec7"}},{"code_sha256_prefix":"efd2486d369320eb","entry":"mask_non_scored_values","repo":"Dreem-Organization/dreem-learning-open","repo_kind":"official","path":"dreem_learning_open/utils/scores.py","file_url":"https://github.com/Dreem-Organization/dreem-learning-open/blob/HEAD/dreem_learning_open/utils/scores.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"efd2486d369320eb"}},{"code_sha256_prefix":"548eb2a6d247f18e","entry":"memmap_hash","repo":"Dreem-Organization/dreem-learning-open","repo_kind":"official","path":"dreem_learning_open/logger/logger.py","file_url":"https://github.com/Dreem-Organization/dreem-learning-open/blob/HEAD/dreem_learning_open/logger/logger.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"548eb2a6d247f18e"}},{"code_sha256_prefix":"87ffbd79b27cf280","entry":"processs_group","repo":"Dreem-Organization/dreem-learning-open","repo_kind":"official","path":"dreem_learning_open/preprocessings/h5_to_memmap.py","file_url":"https://github.com/Dreem-Organization/dreem-learning-open/blob/HEAD/dreem_learning_open/preprocessings/h5_to_memmap.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"87ffbd79b27cf280"}},{"code_sha256_prefix":"a31dce1f7ac6d818","entry":"torch_spectrogram","repo":"Dreem-Organization/dreem-learning-open","repo_kind":"official","path":"dreem_learning_open/models/modulo_net/normalization.py","file_url":"https://github.com/Dreem-Organization/dreem-learning-open/blob/HEAD/dreem_learning_open/models/modulo_net/normalization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a31dce1f7ac6d818"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}