/awesome-helio

A curated list of datasets, tools and papers for machine learning in heliophysics.

Primary LanguageJupyter Notebook

Awesome Helio

A curated list of datasets, tools and papers for machine learning in heliophysics.

About

The primary goal of awesome-helio is to provide a compiled list for all the great resources contributing to the state of the art in heliophysics that are out there but are often hard to find. awesome-helio is mainly focussed on SDO and IRIS data and targets machine learning groups but can also provide important insights for other users.

Contributing

Please take a quick look at the contribution guidelines first.

If you find any datasets, tools and papers that is missing or is not a good fit, please submit a pull request to improve this file. Thank you!

Contents

Missions

SDO and IRIS are part of a larger fleet of spacecraft strategically placed throughout our heliosphere:

Heliophysics Mission Fleet Heliophysics Mission Fleet, Source NASA

A comprehensive list of different missions can be found here.

Books

Datasets

For datasets at FHNW refer to this link.

Data formats

JPEG vs. Fits (Flexible Image Transport System - commonly used digital file format in astronomy)

e-Callisto

  • e-Callisto Data access - The CALLISTO spectrometer is a programmable heterodyne receiver used for observation of solar radio bursts for astronomical science, education, outreach and citizen science as well as rfi-monitoring.

IRIS

  • LMSAL data search - Useful search interface by Lockheed Martin with links to various resources for each observation. Use our internal mirror for data download or work directly on one of the dedicated machines.
  • IRISdataset (Currently only for internal access) - Dataset with selected representative observations for different observation classes.

RHESSI

TODO

SDO

Before working with SDO data make sure to checkout the Guide to SDO Data Analysis, which contains a set of instructions on how to browse, find, download, and analyze SDO data.

  • Curated Image Parameter Dataset - Massive image parameter dataset extracted from the Solar Dynamics Observatory (SDO) mission's AIA instrument, for the period of January 2011 through the current date, with the cadence of six minutes, for nine wavelength channels.
  • JSOC - Data products from the Solar Dynamics Observatory, as well as certain other missions and instruments, are available from the JSOC database.
  • DeepSDO Event dataset - Dataset curated by experts containing three solar event categories (coronal holes, sunspots, and prominences). Suitable for object detection using deep learning-based models.
  • LSDO - A Large-scale Solar Dynamics Observatory image dataset for computer vision applications Dataverse.
  • Machine Learning Data Set for NASA's Solar Dynamics Observatory - A curated dataset from the NASA Solar Dynamics Observatory (SDO) mission in a format suitable for machine learning research.
  • SDOBenchmark - Machine learning image dataset for the prediction of solar flares.
  • SOHO/MDI & SDO/HMI line-of-sight Magnetogram Dataset - Curated dataset consisting of co-aligned, co-temporal observations of the same physical structures as observed by HMI and MDI (rotated to the corresponding the HMI frame), ideal for learning-based super-resolution techniques.
  • SOHO/SDO ML Ready Dataset - Code to generate and temporally sync SoHO and/or SDO Mission image products to make a standardized machine-learning-ready dataset.
  • Solar Flare Prediction from Time Series of Solar Magnetic Field Parameters - Processed dataset provided for the IEEE Big Data 2019 Big Data Cup consisting of a set of magnetic field parameters calculated from individual SHARPs.
  • SWAN-SF Multivariate time series (MVTS) data extracted from HMI Active Region Patch (SHARP) series.

SOHO

  • Shneider, Carl, et al. "A Machine-Learning-Ready Dataset Prepared from the Solar and Heliospheric Observatory Mission." arXiv preprint arXiv:2108.06394 (2021).

STIX

  • STIX Data Center - Spectrometer Telescope for Imaging X-rays (STIX) on Solar Orbiter is a hard X-ray imaging spectrometer covering the energy range from 4 to 150 keV

Links

Papers

Datasets

  • Mahajan, Sushant S., et al. "Improved Measurements of the Sun's Meridional Flow and Torsional Oscillation from Correlation Tracking on MDI and HMI Magnetograms." The Astrophysical Journal 917.2 (2021): 100. link data
  • Kucuk, Ahmet, Juan M. Banda, and Rafal A. Angryk. "A large-scale solar dynamics observatory image dataset for computer vision applications." Scientific data 4.1 (2017): 1-9. link
  • Ahmadzadeh, Azim, Dustin J. Kempton, and Rafal A. Angryk. "A Curated Image Parameter Data Set from the Solar Dynamics Observatory Mission." The Astrophysical Journal Supplement Series 243.1 (2019): 18. link
  • Galvez, Richard, et al. "A machine-learning data set prepared from the NASA solar dynamics observatory mission." The Astrophysical Journal Supplement Series 242.1 (2019): 7. link
  • Angryk, Rafal A., et al. "Multivariate time series dataset for space weather data analytics." Scientific data 7.1 (2020): 1-13. link code
  • Shneider, Carl, et al. "A Machine-Learning-Ready Dataset Prepared from the Solar and Heliospheric Observatory Mission." arXiv preprint arXiv:2108.06394 (2021). link
  • Baek, Ji-Hye, et al. "Solar Event Detection Using Deep-Learning-Based Object Detection Methods." Solar Physics 296.11 (2021): 1-15. link

Event detection, classification and tracking

  • Martens, P. C. H., et al. "Computer vision for the solar dynamics observatory (SDO)." Solar Physics 275.1 (2012): 79-113. link
  • Schuh, Michael A., Dustin Kempton, and Rafal A. Angryk. "A Region-Based Retrieval System for Heliophysics Imagery." FLAIRS Conference. 2017. link
  • Kucuk, Ahmet, Juan M. Banda, and Rafal A. Angryk. "Solar event classification using deep convolutional neural networks." International Conference on Artificial Intelligence and Soft Computing. Springer, Cham, 2017. link
  • Illarionov, Egor A., and Andrey G. Tlatov. "Segmentation of coronal holes in solar disc images with a convolutional neural network." Monthly Notices of the Royal Astronomical Society 481.4 (2018): 5014-5021. link
  • Kempton, Dustin J., Michael A. Schuh, and Rafal A. Angryk. "Tracking solar phenomena from the sdo." The Astrophysical Journal 869.1 (2018): 54. link
  • Armstrong, John A., and Lyndsay Fletcher. "Fast solar image classification using deep learning and its importance for automation in solar physics." Solar Physics 294.6 (2019): 1-23. link
  • Love, Teri, Thomas Neukirch, and Clare E. Parnell. "Analyzing AIA Flare Observations Using Convolutional Neural Networks." Frontiers in Astronomy and Space Sciences 7 (2020): 34. link

Missions

  • Pesnell, W. Dean, B. J. Thompson, and P. C. Chamberlin. "The solar dynamics observatory (SDO)." The Solar Dynamics Observatory. Springer, New York, NY, 2011. 3-15. link
  • Lemen, James R., et al. "The atmospheric imaging assembly (AIA) on the solar dynamics observatory (SDO)." The solar dynamics observatory. Springer, New York, NY, 2011. 17-40. link

Space weather prediction

  • Bobra, Monica G., and Sebastien Couvidat. "Solar flare prediction using SDO/HMI vector magnetic field data with a machine-learning algorithm." The Astrophysical Journal 798.2 (2015): 135. link
  • McGregor, Sean, et al. "Flarenet: A deep learning framework for solar phenomena prediction." Neural Information Processing Systems (NIPS) 2017 workshop on Deep Learning for Physical Sciences (DLPS), Long Beach, CA, US. 2017. link
  • Nagem, Tarek AM, et al. "Deep learning technology for predicting solar flares from (Geostationary Operational Environmental Satellite) data." (2018) link
  • Panos, Brandon, et al. "Identifying typical Mg II flare spectra using machine learning." The Astrophysical Journal 861.1 (2018): 62. link
  • Jonas, Eric, et al. "Flare prediction using photospheric and coronal image data." Solar Physics 293.3 (2018): 1-22. link
  • Chen, Yang, et al. "Identifying solar flare precursors using time series of SDO/HMI Images and SHARP Parameters." Space Weather 17.10 (2019): 1404-1426. link
  • Panos, Brandon, and Lucia Kleint. "Real-time flare prediction based on distinctions between flaring and non-flaring active region spectra." The Astrophysical Journal 891.1 (2020): 17. link
  • Ivanov, Sergey, et al. "Solar activity classification based on Mg II spectra: towards classification on compressed data." arXiv preprint arXiv:2009.07156 (2020). link
  • Wang, Jingjing, et al. "Solar Flare Predictive Features Derived from Polarity Inversion Line Masks in Active Regions Using an Unsupervised Machine Learning Algorithm." The Astrophysical Journal 892.2 (2020): 140. link
  • Ahmadzadeh, Azim, et al. "How to Train Your Flare Prediction Model: Revisiting Robust Sampling of Rare Events." The Astrophysical Journal Supplement Series 254.2 (2021): 23. link

Other Applications

  • Wright, Paul J., et al. "DeepEM: Demonstrating a Deep Learning Approach to DEM Inversion." Zenodo. link
  • Guedes dos Santos, L. F., et al. "Multi-Channel Auto-Calibration for the Atmospheric Imaging Assembly instrument with Deep Learning." AGU Fall Meeting Abstracts. Vol. 2020. 2020. link

Tools

  • Aiapy - Package for analyzing calibrated (level 1) EUV imaging data from AIA.
  • IRISreader - Python library that allows for efficient browsing through IRIS satellite data in order to simplify machine learning applications.
  • Global High-Resolution Hα network
  • Helioviewer - Solar and heliospheric image visualization tool. code
  • Integrated Solar Database - Solar and heliospheric image visualization tool including image parameters with Extended Spatiotemporal Querying Capabilities.
  • integrated Space Weather Analysis (iSWA) system
  • SolarMonitor - Provides realtime information about the solar activity. The most recent images of the sun are shown, together with the description of the different NOAA AR of the day and which flares has been associated to them.
  • SpaceML - A Machine Learning toolbox and developer community building the next generation AI applications for space science and exploration containing a set of code examples.
  • Sunpy - Set of tools for solar data analysis with Python.
  • SWPC CME Analysis Tool (SWPC_CAT) - Primary tool being used by NOAA SWPC in measuring key parameters of a Coronal Mass Ejection (CME) code

Videos