/pyspark-utils

Reusable Python classes that extend open source PySpark capabilities. Examples of implementation is available under notebooks of repo https://github.com/bennyaustin/synapse-dataplatform

Primary LanguagePythonApache License 2.0Apache-2.0

pyspark-utils

This repo has a collection of reusable Python classes that extend out of box PySpark capabilities. This collection has the following classes.

1. CommonTransforms

CommonTransforms is a Python class that uses PySpark libraries to apply common transformations to a Spark dataframe. More information about this class and it's usage is available here - CommonTransforms/README.md