/spark_at_nersc

Collections of scripts and notebooks using Spark to be used at NERSC

Primary LanguageJupyter NotebookApache License 2.0Apache-2.0

Apache Spark @ NERSC

Collections of scripts and notebooks using Spark to be used at NERSC.

Scripts

  • Benchmarking Apache Spark FITS connector.
    • Test I/O performances.
  • Re-partitioning data.
    • Test communication performances.

Notebooks

  • Manipulating cosmological data using Apache Spark.
    • Test user-defined functions.