Lighter

Lighter is an opensource application for interacting with Apache Spark on Kubernetes or Apache Hadoop YARN. It is hevily inspired by Apache Livy and has some overlaping features.

Lighter supports:

Interactive Python Sessions through Sparkmagic kernel
Batch application submissions through the REST API

⚠️ If you are using interactive sessions: While we have tested batch applications quite extensively, there might be some problems with interactive sessions, consider current release of Lighter as alpha.

You can read a breaf description on how Lighter works here.

Using Lighter

Quickstart with Minikube
Installation on Kubernetes
Installation on Docker
Configuration Properties
Configuring Sparkmagic
Using REST API

Developing Lighter

Building

To build Docker image, containing application, you need to run

docker build -t lighter .

Spark versions

Lighter will always try to support the latest patch version for all officially recommended Apache Spark releases i.e.: 3.2.4, 3.3.2 and 3.4.1.

Contributing

See Contribution guide

License

Lighter is MIT licensed.