The code under this repository has been moved under connectors module in https://github.com/delta-io/delta repository. No changes are allowed to this repository. Any issues or changes, please propose them on the https://github.com/delta-io/delta repository. More details on deprecation are found at #556.
We are building connectors to bring Delta Lake to popular big-data engines outside Apache Spark (e.g., Apache Hive, Presto, Apache Flink) and also to common reporting tools like Microsoft Power BI.
This is the repository for Delta Lake Connectors. It includes
- Delta Standalone: a native library for reading and writing Delta Lake metadata.
- Connectors to popular big-data engines (e.g., Apache Hive, Presto, Apache Flink) and to common reporting tools like Microsoft Power BI.
Please refer to the main Delta Lake repository if you want to learn more about the Delta Lake project.
- Delta Standalone Java API docs
- Flink/Delta Connector Java API docs
Delta Standalone, formerly known as the Delta Standalone Reader (DSR), is a JVM library to read and write Delta tables. Unlike https://github.com/delta-io/delta, this project doesn't use Spark to read or write tables and it has only a few transitive dependencies. It can be used by any application that cannot use a Spark cluster.
- To compile the project, run
build/sbt standalone/compile - To test the project, run
build/sbt standalone/test - To publish the JAR, run
build/sbt standaloneCosmetic/publishM2
See Delta Standalone for detailed documentation.
Read Delta tables directly from Apache Hive using the Hive Connector. See the dedicated README.md for more details.
Use the Flink/Delta Connector to read and write Delta tables from Apache Flink applications. The connector includes a sink for writing to Delta tables from Apache Flink, and a source for reading Delta tables using Apache Flink (still in progress.) See the dedicated README.md for more details.
sql-delta-import allows for importing data from a JDBC source into a Delta table.
The connector for Microsoft Power BI is basically just a custom Power Query function that allows you to read a Delta table from any file-based data source supported by Microsoft Power BI. Details can be found in the dedicated README.md.
We use GitHub Issues to track community reported issues. You can also contact the community for getting answers.
We welcome contributions to Delta Lake Connectors repository. We use GitHub Pull Requests for accepting changes.
There are two mediums of communication within the Delta Lake community.
-
Public Slack Channel
-
Public Mailing list
- Before local debugging of
standalonetests in IntelliJ, run allstandalonetests using SBT. This helps IntelliJ recognize the golden tables as class resources.