DOI: https://doi.org/10.18517/ijods.2.1.38-46.2021
PY-METEO-NUM: Dockerized Python Notebook Environment for Portable Data Analysis Workflows in Indonesian Atmospheric Science Communities
Abstract
Reproducibility and replicability in analyzing data is one of the main requirements for the advance-ment of scientific fields that rely heavily on computational data analysis, such as atmospheric science. However, there are very few research activities that field in Indonesia that emphasize the principle of transparency of codes and data in the dissemination of the results. This issue is a major challenge for the Indonesian scientific community to verify the output of research activities from their peers. One common obstacle to the reproducibility of data-driven research is the portability issue of the computing environment used to reproduce the results. Therefore, in this article, we would like to offer a solution through Debian-based dockerized Jupyter Notebook that have been installed with several Python libraries that are often used in atmospheric science research. Through this containerized computing environment, we expect to overcome the portability and dependency constraints that often faced by atmospheric scientists and also to encourage the growth of research ecosystem in Indonesia through an open and replicable environment.
Article Details
How to Cite
[1]
S. Herho and I. Dasapta Erwin, “PY-METEO-NUM: Dockerized Python Notebook Environment for Portable Data Analysis Workflows in Indonesian Atmospheric Science Communities”, Int. J. Data. Science., vol. 2, no. 1, pp. 38-46, Sep. 2020.
Issue
Section
Articles
References
J. M. Wallace and P. P. Hobbs, Atmospheric Science An Introductory Survey, vol. 92 of International Geophysics Series. Academic Press, 2 ed., 2006.
S. Nurcahyo, F. Nhita, and Adiwijaya, “Rainfall prediction in kemayoran jakarta using hybrid genetic algorithm (ga) and partially connected feedforward neural network (pcfnn),” in 2014 2nd International Conference on Information and Communication Technology (ICoICT), pp. 166–171, 2014.
F. Nhita, D. Saepudin, Adiwijaya, and U. N. Wisesty, “Comparative study of moving average on rainfall time series data for rainfall forecasting based on evolving neural network classifier,” in 2015 3rd International Symposium on Computational and Business Intelligence (ISCBI), pp. 112–116, 2015.
Gunawansyah, T. H. Liong, and Adiwijaya, “Prediction and anomaly detection of rainfall using evolving neural network to support planting calender in soreang (bandung),” in 2017 5th International Conference on Information and Communication Technology (ICoIC7), pp. 1–6, 2017.
J. A. Suyatno, F. Nhita, and A. A. Rohmawati, “Rainfall forecasting in bandung regency using c4.5 algorithm,” in 2018 6th International Conference on Information and Communication Technology (ICoICT), pp. 324–328, 2018.
D. E. Irawan, C. N. Rachmi, H. Irawan, J. Abraham, K. Kusno, M. T. Multazam, K. K. Rosada, S. H. Nugroho, G. Kusumah, D. Holidin, and N. A. Aziz, “Penerapan Open Science di Indonesia agar riset lebih terbuka, mudah Diakses, dan Meningkatkan Dampak Saintifik. (Indonesia) [The application of Open Science in Indonesia so that research is more open, easily accessible, and increases the scientific impact],” Berkala Ilmu Perpustakaan dan Informasi, vol. 13, no. 1, pp. 25–36, 2017.
J. W.-B. Lin, “Why python is the next wave in earth sciences computing,” Bulletin of the American Meteorological Society, vol. 93, no. 12, pp. 1823–1824, 2012.
R. Almugbel, L.-H. Hung, J. Hu, A. Almutairy, N. Ortogero, Y. Tamta, and K. Y. Yeung, “Reproducible bioconductor workflows using browser-based interactive notebooks and containers,” Journal of American Medical Informatics Association, vol. 25, no. 1, pp. 4–12, 2017.
C. Boettiger, “An introduction to docker for reproducible research,” SIGOPS Oper. Syst. Rev., vol. 49, p. 71–79, Jan. 2015.
J. P. Hacker, J. Exby, D. Gill, I. Jimenez, C. Maltzahn, T. See, G. Mullendore, and K. Fossell, “A containerized mesoscale model and analysis toolkit to accelerate classroom learning, collaborative research, and uncertainty quantification,” Bulletin of the American Meteorological Society, vol. 98, no. 6, pp. 1129–1138, 2017.
N. H. D. Morris, S. Voutsinas and R. Mann, “Use of docker for deployment and testing of astronomy software,” arXiv preprint arXiv:1707.03341, 2017.
U. Team, Unidata Python Training, accessed June 9, 2020. https://unidata.github.io/ python-training/.
J. W.-B. Lin, A Hands-On Introduction to Using Python in the Atmospheric and Oceanic Sciences. 2012.
D. Irving, “Python for atmosphere and ocean scientists,” Journal of Open Source Education, vol. 2, no. 16, p. 37, 2019.
J. D. Hunter, “Matplotlib: A 2d graphics environment,” Computing in Science Engineering, vol. 9, no. 3, pp. 90–95, 2007.
Met Office, Cartopy: a cartographic python library with a matplotlib interface. Exeter, Devon, 2010 - 2015.
Oscar, “oscarbranson/cbsyst: beta,” Aug. 2018.
C. S. Zender, “Analysis of self-describing gridded geoscience data with netcdf operators (nco),” Environmental Modelling Software, vol. 23, no. 10, pp. 1338 – 1342, 2008.
P. Petrelli, “coecms/era5: python base codes to interface the CDS api and automate ERA5 download: first release v0.1,” Nov. 2019.
B. E. j. Rose, “Climlab: a python toolkit for interactive, process-oriented climate modeling,” Journal of Open Source Software, vol. 3, no. 24, p. 659, 2018.
K. M. Thyng, C. A. Greene, R. D. Hetland, H. M. Zimmerle, and S. F. DiMarco, “True colors of oceanography: Guidelines for effective and accurate colormap selection,” Oceanography, vol. 293, September 2016.
Fernandes, “python-ctd v0.2.1,” Aug. 2014.
S. J. Taylor and B. Letham, “Forecasting at scale,” PeerJ Preprints, vol. 5, p. e3190v2, Sept. 2017.
Filipe, “python-gsw v3.0.3,” Aug. 2014.
Met Office, Iris: A Python library for analysing and visualising meteorological and oceanographic data sets. Exeter, Devon, v1.2 ed., 2010 - 2013.
R. M. May, S. C. Arms, P. Marsh, E. Bruning, J. R. Leeman, K. Goebbert, J. E. Thielen, and Z. S. Bruick, “Metpy: A Python package for meteorological data,” 2008 - 2020.
J. Whitaker, C. Khrulev, D. Huard, C. Paulik, S. Hoyer, Filipe, L. Pastewka, A. Mohr, C. Marquardt, B. Couwen-berg, M. Taves, J. Whitaker, M. Cuntz, M. Bohnet, M. Brett, R. Hetland, M. Korenciak,ˇ barronh, K. Onu, J. J. Helmus, J. Hamman, A. Barna, fredrik 1, B. Koziol, T. Kluyver, R. May, J. Smrekar, C. Barker, C. Gohlke, and
B. P. Kinoshita, “Unidata/netcdf4-python: Version 1.5.3 release,” Oct. 2019.
G. Bradski, “The OpenCV Library,” Dr. Dobb’s Journal of Software Tools, 2000.
T. Kralidis, “geopython/owslib: v0.20.0,” 2020.
K. Wilcox, A. Crosby, and B. McKenna, “https://github.com/asascience-open/paegan,” 2018.
P. Kershaw, R. Ananthakrishnan, L. Cinquini, B. Lawrence, S. Pascoe, and F. Siebenlist, “A flexible component based access control architecture for opendap services,” 05 2010.
J. Salvatier, T. V. Wiecki, and C. Fonnesbeck, “Probabilistic programming in python using pymc3,” PeerJ Computer Science, vol. 2, p. e55, Apr. 2016.
R. May, S. Arms, J. Leeman, and J. Chastang, “Siphon: A collection of Python utilities for accessing remote atmospheric and oceanic datasets,” 2014 - 2017.
M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, M. Kudlur, J. Levenberg, R. Monga, S. Moore, D. G. Murray, B. Steiner, P. Tucker, V. Vasudevan, P. Warden, M. Wicke, Y. Yu, and X. Zheng, “Tensorflow: A system for large-scale machine learning,” in 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), (Savannah, GA), pp. 265–283, USENIX Association, Nov. 2016.
A. Dawson, “Windspharm: A high-level library for global wind field computations using spherical harmonics,” Journal of Open Research Software, vol. 4, no. 1, 2016.
W. Ladwig, “Wrf-python (version 1.3.2),” 2020.
S. Hoyer and J. J. Hamman, “xarray: N-d labeled arrays and datasets in python,” Journal of Open Research Software, vol. 5, no. 1, p. 10, 2017.
D. Huard, T. J. Smith, P. Bourgault, T. Logan, sbiner, P. Roy, D. Caron, jwenfai, RondeauG, C. Whelan, and A. Stephens, “Ouranosinc/xclim: v0.17.0,” May 2020.
M. Collier and P. Uhe, “CMIP5 datasets from the ACCESS1.0 and ACCESS1.3 coupled climate models,” tech. rep., The Centre for Australian Weather and Climate Research, 12 2012.
C. S. RAMAGE, “Role of a tropical “maritime continent” in the atmospheric circulation,” Monthly Weather Review, vol. 96, no. 6, pp. 365–370, 1968.
“Gnu general public license.”
S. Nurcahyo, F. Nhita, and Adiwijaya, “Rainfall prediction in kemayoran jakarta using hybrid genetic algorithm (ga) and partially connected feedforward neural network (pcfnn),” in 2014 2nd International Conference on Information and Communication Technology (ICoICT), pp. 166–171, 2014.
F. Nhita, D. Saepudin, Adiwijaya, and U. N. Wisesty, “Comparative study of moving average on rainfall time series data for rainfall forecasting based on evolving neural network classifier,” in 2015 3rd International Symposium on Computational and Business Intelligence (ISCBI), pp. 112–116, 2015.
Gunawansyah, T. H. Liong, and Adiwijaya, “Prediction and anomaly detection of rainfall using evolving neural network to support planting calender in soreang (bandung),” in 2017 5th International Conference on Information and Communication Technology (ICoIC7), pp. 1–6, 2017.
J. A. Suyatno, F. Nhita, and A. A. Rohmawati, “Rainfall forecasting in bandung regency using c4.5 algorithm,” in 2018 6th International Conference on Information and Communication Technology (ICoICT), pp. 324–328, 2018.
D. E. Irawan, C. N. Rachmi, H. Irawan, J. Abraham, K. Kusno, M. T. Multazam, K. K. Rosada, S. H. Nugroho, G. Kusumah, D. Holidin, and N. A. Aziz, “Penerapan Open Science di Indonesia agar riset lebih terbuka, mudah Diakses, dan Meningkatkan Dampak Saintifik. (Indonesia) [The application of Open Science in Indonesia so that research is more open, easily accessible, and increases the scientific impact],” Berkala Ilmu Perpustakaan dan Informasi, vol. 13, no. 1, pp. 25–36, 2017.
J. W.-B. Lin, “Why python is the next wave in earth sciences computing,” Bulletin of the American Meteorological Society, vol. 93, no. 12, pp. 1823–1824, 2012.
R. Almugbel, L.-H. Hung, J. Hu, A. Almutairy, N. Ortogero, Y. Tamta, and K. Y. Yeung, “Reproducible bioconductor workflows using browser-based interactive notebooks and containers,” Journal of American Medical Informatics Association, vol. 25, no. 1, pp. 4–12, 2017.
C. Boettiger, “An introduction to docker for reproducible research,” SIGOPS Oper. Syst. Rev., vol. 49, p. 71–79, Jan. 2015.
J. P. Hacker, J. Exby, D. Gill, I. Jimenez, C. Maltzahn, T. See, G. Mullendore, and K. Fossell, “A containerized mesoscale model and analysis toolkit to accelerate classroom learning, collaborative research, and uncertainty quantification,” Bulletin of the American Meteorological Society, vol. 98, no. 6, pp. 1129–1138, 2017.
N. H. D. Morris, S. Voutsinas and R. Mann, “Use of docker for deployment and testing of astronomy software,” arXiv preprint arXiv:1707.03341, 2017.
U. Team, Unidata Python Training, accessed June 9, 2020. https://unidata.github.io/ python-training/.
J. W.-B. Lin, A Hands-On Introduction to Using Python in the Atmospheric and Oceanic Sciences. 2012.
D. Irving, “Python for atmosphere and ocean scientists,” Journal of Open Source Education, vol. 2, no. 16, p. 37, 2019.
J. D. Hunter, “Matplotlib: A 2d graphics environment,” Computing in Science Engineering, vol. 9, no. 3, pp. 90–95, 2007.
Met Office, Cartopy: a cartographic python library with a matplotlib interface. Exeter, Devon, 2010 - 2015.
Oscar, “oscarbranson/cbsyst: beta,” Aug. 2018.
C. S. Zender, “Analysis of self-describing gridded geoscience data with netcdf operators (nco),” Environmental Modelling Software, vol. 23, no. 10, pp. 1338 – 1342, 2008.
P. Petrelli, “coecms/era5: python base codes to interface the CDS api and automate ERA5 download: first release v0.1,” Nov. 2019.
B. E. j. Rose, “Climlab: a python toolkit for interactive, process-oriented climate modeling,” Journal of Open Source Software, vol. 3, no. 24, p. 659, 2018.
K. M. Thyng, C. A. Greene, R. D. Hetland, H. M. Zimmerle, and S. F. DiMarco, “True colors of oceanography: Guidelines for effective and accurate colormap selection,” Oceanography, vol. 293, September 2016.
Fernandes, “python-ctd v0.2.1,” Aug. 2014.
S. J. Taylor and B. Letham, “Forecasting at scale,” PeerJ Preprints, vol. 5, p. e3190v2, Sept. 2017.
Filipe, “python-gsw v3.0.3,” Aug. 2014.
Met Office, Iris: A Python library for analysing and visualising meteorological and oceanographic data sets. Exeter, Devon, v1.2 ed., 2010 - 2013.
R. M. May, S. C. Arms, P. Marsh, E. Bruning, J. R. Leeman, K. Goebbert, J. E. Thielen, and Z. S. Bruick, “Metpy: A Python package for meteorological data,” 2008 - 2020.
J. Whitaker, C. Khrulev, D. Huard, C. Paulik, S. Hoyer, Filipe, L. Pastewka, A. Mohr, C. Marquardt, B. Couwen-berg, M. Taves, J. Whitaker, M. Cuntz, M. Bohnet, M. Brett, R. Hetland, M. Korenciak,ˇ barronh, K. Onu, J. J. Helmus, J. Hamman, A. Barna, fredrik 1, B. Koziol, T. Kluyver, R. May, J. Smrekar, C. Barker, C. Gohlke, and
B. P. Kinoshita, “Unidata/netcdf4-python: Version 1.5.3 release,” Oct. 2019.
G. Bradski, “The OpenCV Library,” Dr. Dobb’s Journal of Software Tools, 2000.
T. Kralidis, “geopython/owslib: v0.20.0,” 2020.
K. Wilcox, A. Crosby, and B. McKenna, “https://github.com/asascience-open/paegan,” 2018.
P. Kershaw, R. Ananthakrishnan, L. Cinquini, B. Lawrence, S. Pascoe, and F. Siebenlist, “A flexible component based access control architecture for opendap services,” 05 2010.
J. Salvatier, T. V. Wiecki, and C. Fonnesbeck, “Probabilistic programming in python using pymc3,” PeerJ Computer Science, vol. 2, p. e55, Apr. 2016.
R. May, S. Arms, J. Leeman, and J. Chastang, “Siphon: A collection of Python utilities for accessing remote atmospheric and oceanic datasets,” 2014 - 2017.
M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, M. Kudlur, J. Levenberg, R. Monga, S. Moore, D. G. Murray, B. Steiner, P. Tucker, V. Vasudevan, P. Warden, M. Wicke, Y. Yu, and X. Zheng, “Tensorflow: A system for large-scale machine learning,” in 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), (Savannah, GA), pp. 265–283, USENIX Association, Nov. 2016.
A. Dawson, “Windspharm: A high-level library for global wind field computations using spherical harmonics,” Journal of Open Research Software, vol. 4, no. 1, 2016.
W. Ladwig, “Wrf-python (version 1.3.2),” 2020.
S. Hoyer and J. J. Hamman, “xarray: N-d labeled arrays and datasets in python,” Journal of Open Research Software, vol. 5, no. 1, p. 10, 2017.
D. Huard, T. J. Smith, P. Bourgault, T. Logan, sbiner, P. Roy, D. Caron, jwenfai, RondeauG, C. Whelan, and A. Stephens, “Ouranosinc/xclim: v0.17.0,” May 2020.
M. Collier and P. Uhe, “CMIP5 datasets from the ACCESS1.0 and ACCESS1.3 coupled climate models,” tech. rep., The Centre for Australian Weather and Climate Research, 12 2012.
C. S. RAMAGE, “Role of a tropical “maritime continent” in the atmospheric circulation,” Monthly Weather Review, vol. 96, no. 6, pp. 365–370, 1968.
“Gnu general public license.”