{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/effective-benchmarks-for-optical-turbulence","title":"Effective Benchmarks for Optical Turbulence Modeling","arxiv_id":"2401.03573","date":"2024-01-07","proceeding":null,"authors":["Christopher Jellen","Charles Nelson","Cody Brownell","John Burkhardt"],"abstract":"Optical turbulence presents a significant challenge for communication, directed energy, and imaging systems, especially in the atmospheric boundary layer. Effective modeling of optical turbulence strength is critical for the development and deployment of these systems. The lack of standard evaluation tools, especially long-term data sets, modeling tasks, metrics, and baseline models, prevent effective comparisons between approaches and models. This reduces the ease of reproducing results and contributes to over-fitting on local micro-climates. Performance characterized using evaluation metrics provides some insight into the applicability of a model for predicting the strength of optical turbulence. However, these metrics are not sufficient for understanding the relative quality of a model. We introduce the \\texttt{otbench} package, a Python package for rigorous development and evaluation of optical turbulence strength prediction models. The package provides a consistent interface for evaluating optical turbulence models on a variety of benchmark tasks and data sets. The \\texttt{otbench} package includes a range of baseline models, including statistical, data-driven, and deep learning models, to provide a sense of relative model quality. \\texttt{otbench} also provides support for adding new data sets, tasks, and evaluation metrics. The package is available at \\url{https://github.com/cdjellen/otbench}.","url_abs":"https://arxiv.org/abs/2401.03573v1","url_pdf":"https://arxiv.org/pdf/2401.03573v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"effective-benchmarks-for-optical-turbulence","repo_url":"https://github.com/cdjellen/otbench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"time-series-forecasting","task_name":"Time Series Forecasting"},{"task_slug":"time-series-regression","task_name":"Time Series Regression"}],"methods":[{"method_slug":"hybrid-awt","method_name":"Hybrid AWT"}],"datasets_introduced":[{"slug":"mlo-cn2","name":"MLO-Cn2","full_name":"Mauna Loa Seeing Study"},{"slug":"usna-cn2-long-term","name":"USNA-Cn2 (long-term)","full_name":"Unites States Naval Academy Long-term Scintillation Study"},{"slug":"usna-cn2-short-duration","name":"USNA-Cn2 (short-duration)","full_name":"Unites States Naval Academy Short-duration Optical Turbulence Dataset"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/time-series-forecasting-on-mlo-cn2","task":"Time Series Forecasting","dataset":"MLO-Cn2","model":"GBRT","rank_in_archive_order":1,"of":7,"metrics":{"RMSE":"0.428"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-forecasting-on-mlo-cn2","task":"Time Series Forecasting","dataset":"MLO-Cn2","model":"Mean Window Forecast","rank_in_archive_order":2,"of":7,"metrics":{"RMSE":"0.481"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-forecasting-on-mlo-cn2","task":"Time Series Forecasting","dataset":"MLO-Cn2","model":"Minute Climatology","rank_in_archive_order":3,"of":7,"metrics":{"RMSE":"0.551"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-forecasting-on-mlo-cn2","task":"Time Series Forecasting","dataset":"MLO-Cn2","model":"RNN","rank_in_archive_order":4,"of":7,"metrics":{"RMSE":"0.581"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-forecasting-on-mlo-cn2","task":"Time Series Forecasting","dataset":"MLO-Cn2","model":"Climatology","rank_in_archive_order":5,"of":7,"metrics":{"RMSE":"0.658"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-forecasting-on-mlo-cn2","task":"Time Series Forecasting","dataset":"MLO-Cn2","model":"Linear Forecast","rank_in_archive_order":6,"of":7,"metrics":{"RMSE":"0.930"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-forecasting-on-mlo-cn2","task":"Time Series Forecasting","dataset":"MLO-Cn2","model":"Persistence","rank_in_archive_order":7,"of":7,"metrics":{"RMSE":"1.227"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-forecasting-on-usna-cn2-short","task":"Time Series Forecasting","dataset":"USNA-Cn2 (short-duration)","model":"GBRT","rank_in_archive_order":1,"of":5,"metrics":{"RMSE":"0.160"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-forecasting-on-usna-cn2-short","task":"Time Series Forecasting","dataset":"USNA-Cn2 (short-duration)","model":"Mean Window Forecast","rank_in_archive_order":2,"of":5,"metrics":{"RMSE":"0.182"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-forecasting-on-usna-cn2-short","task":"Time Series Forecasting","dataset":"USNA-Cn2 (short-duration)","model":"RNN","rank_in_archive_order":3,"of":5,"metrics":{"RMSE":"0.187"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-forecasting-on-usna-cn2-short","task":"Time Series Forecasting","dataset":"USNA-Cn2 (short-duration)","model":"Minute Climatology","rank_in_archive_order":4,"of":5,"metrics":{"RMSE":"0.453"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-forecasting-on-usna-cn2-short","task":"Time Series Forecasting","dataset":"USNA-Cn2 (short-duration)","model":"Persistence","rank_in_archive_order":5,"of":5,"metrics":{"RMSE":"0.821"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-regression-on-mlo-cn2","task":"Time Series Regression","dataset":"MLO-Cn2","model":"GBRT","rank_in_archive_order":1,"of":5,"metrics":{"RMSE":"0.212"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-regression-on-mlo-cn2","task":"Time Series Regression","dataset":"MLO-Cn2","model":"RNN","rank_in_archive_order":2,"of":5,"metrics":{"RMSE":"0.336"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-regression-on-mlo-cn2","task":"Time Series Regression","dataset":"MLO-Cn2","model":"Minute Climatology","rank_in_archive_order":3,"of":5,"metrics":{"RMSE":"0.504"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-regression-on-mlo-cn2","task":"Time Series Regression","dataset":"MLO-Cn2","model":"Climatology","rank_in_archive_order":4,"of":5,"metrics":{"RMSE":"0.661"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-regression-on-mlo-cn2","task":"Time Series Regression","dataset":"MLO-Cn2","model":"Persistence","rank_in_archive_order":5,"of":5,"metrics":{"RMSE":"1.209"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-regression-on-usna-cn2-long-term","task":"Time Series Regression","dataset":"USNA-Cn2 (long-term)","model":"Hybrid Air-Water Temperature Difference","rank_in_archive_order":1,"of":9,"metrics":{"RMSE":"0.458"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-regression-on-usna-cn2-long-term","task":"Time Series Regression","dataset":"USNA-Cn2 (long-term)","model":"RNN","rank_in_archive_order":2,"of":9,"metrics":{"RMSE":"0.530"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-regression-on-usna-cn2-long-term","task":"Time Series Regression","dataset":"USNA-Cn2 (long-term)","model":"Minute Climatology","rank_in_archive_order":3,"of":9,"metrics":{"RMSE":"0.625"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-regression-on-usna-cn2-long-term","task":"Time Series Regression","dataset":"USNA-Cn2 (long-term)","model":"Climatology","rank_in_archive_order":4,"of":9,"metrics":{"RMSE":"0.632"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-regression-on-usna-cn2-long-term","task":"Time Series Regression","dataset":"USNA-Cn2 (long-term)","model":"Offshore Macro Meteorological","rank_in_archive_order":5,"of":9,"metrics":{"RMSE":"0.675"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-regression-on-usna-cn2-long-term","task":"Time Series Regression","dataset":"USNA-Cn2 (long-term)","model":"Air-Water Temperature Difference","rank_in_archive_order":6,"of":9,"metrics":{"RMSE":"1.046"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-regression-on-usna-cn2-long-term","task":"Time Series Regression","dataset":"USNA-Cn2 (long-term)","model":"Persistence","rank_in_archive_order":7,"of":9,"metrics":{"RMSE":"1.208"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-regression-on-usna-cn2-long-term","task":"Time Series Regression","dataset":"USNA-Cn2 (long-term)","model":"Macro Meteorological","rank_in_archive_order":8,"of":9,"metrics":{"RMSE":"1.217"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-regression-on-usna-cn2-long-term","task":"Time Series Regression","dataset":"USNA-Cn2 (long-term)","model":"GBRT","rank_in_archive_order":9,"of":9,"metrics":{"RMSE":"1.340"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-regression-on-usna-cn2-short","task":"Time Series Regression","dataset":"USNA-Cn2 (short-duration)","model":"Offshore Macro Meteorological","rank_in_archive_order":1,"of":10,"metrics":{"RMSE":"0.178"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-regression-on-usna-cn2-short","task":"Time Series Regression","dataset":"USNA-Cn2 (short-duration)","model":"GBRT","rank_in_archive_order":2,"of":10,"metrics":{"RMSE":"0.299"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-regression-on-usna-cn2-short","task":"Time Series Regression","dataset":"USNA-Cn2 (short-duration)","model":"Hybrid Air-Water Temperature Difference","rank_in_archive_order":3,"of":10,"metrics":{"RMSE":"0.303"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-regression-on-usna-cn2-short","task":"Time Series Regression","dataset":"USNA-Cn2 (short-duration)","model":"Linear Forecast","rank_in_archive_order":4,"of":10,"metrics":{"RMSE":"0.358"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-regression-on-usna-cn2-short","task":"Time Series Regression","dataset":"USNA-Cn2 (short-duration)","model":"RNN","rank_in_archive_order":5,"of":10,"metrics":{"RMSE":"0.375"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-regression-on-usna-cn2-short","task":"Time Series Regression","dataset":"USNA-Cn2 (short-duration)","model":"Minute Climatology","rank_in_archive_order":6,"of":10,"metrics":{"RMSE":"0.452"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-regression-on-usna-cn2-short","task":"Time Series Regression","dataset":"USNA-Cn2 (short-duration)","model":"Climatology","rank_in_archive_order":7,"of":10,"metrics":{"RMSE":"0.480"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-regression-on-usna-cn2-short","task":"Time Series Regression","dataset":"USNA-Cn2 (short-duration)","model":"Persistence","rank_in_archive_order":8,"of":10,"metrics":{"RMSE":"0.758"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-regression-on-usna-cn2-short","task":"Time Series Regression","dataset":"USNA-Cn2 (short-duration)","model":"Macro Meteorological","rank_in_archive_order":9,"of":10,"metrics":{"RMSE":"0.864"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-regression-on-usna-cn2-short","task":"Time Series Regression","dataset":"USNA-Cn2 (short-duration)","model":"Air-Water Temperature Difference","rank_in_archive_order":10,"of":10,"metrics":{"RMSE":"0.910"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}