Vector Data Source for Apache Spark
A library to integrate Vector with Spark, allowing you to load data from Spark into Vector in parallel and to consume the results of Vector based computations in Spark(SQL). This connector works with both Vector SMP and VectorH MPP. It is built with sbt.
Documentation
Spark-Vector Data Source
sbt doc
Spark-Vector Loader
sbt loader/doc
Spark-Vector Provider
sbt provider/doc
Requirements
This library has different versions for Spark 1.5+, 2.1+ and 3.1.1.
Spark Version | Compatible version of Vector Data Source for Spark |
---|---|
1.5 - 1.6.3 |
1.0 |
2.1 - 2.3 |
2.0 |
2.2+ |
2.1 |
3.1.1 |
3.0 |
Spark-Vector Data Source
Building
sbt assembly
Using with Spark shell/submit
This module can be added to Spark using the --driver-class-path
command line option. Spark shell example (assuming $SPARK_VECTOR
is the root directory of spark-vector):
spark-shell --driver-class-path $SPARK_VECTOR/target/spark-vector-assembly-master.jar
Assuming that there is a Vector installation on node vectorhost
, instance VI
and database databasename
SparkSQL
spark.sqlContext.sql("""CREATE TEMPORARY VIEW vector_table
USING com.actian.spark_vector.sql.DefaultSource
OPTIONS (
host "vectorhost",
instance "VI",
database "databasename",
table "vector_table",
user "user",
password "password"
)""")
and then to load data into Vector:
spark.sqlContext.sql("insert into vector_table select * from spark_table")
... or to read Vector data into Spark:
spark.sqlContext.sql("select * from vector_table")
Options
The OPTIONS
clause of the SparkSQL statement can contain:
Parameter | Required | Default | Notes |
---|---|---|---|
host | Yes | none | Host name of where Vector is located |
instance | Yes | none | Vector database instance identifier (two letters) |
database | Yes | none | Vector database name |
user | Yes | None | User name to use when connecting to Vector |
password | Yes | None | Password to use when connecting to Vector |
table | Yes | None | Vector target table |
loadpreSQL* | No | None | Query to execute before a load, in the same transaction. Multiple queries can be specified using different suffixes, e.g. loadpreSQL0, loadpreSQL1, etc. In this case, the query execution order is determined by the lexicographic order |
loadpostSQL* | No | None | Query to execute after a load, in the same transaction. Multiple queries can be specified using different suffixes, e.g. loadpostSQL0, loadpostSQL1, etc. In this case, the query execution order is determined by the lexicographic order |
Spark-Vector Loader
The Spark-Vector loader is a command line client utility that provides the ability to load CSV, Parquet and ORC files through Spark into Vector, using the Spark-Vector connector.
Building
sbt loader/assembly
Usage: CSV
Loading CSV files into Vector with Spark:
spark-submit --class com.actian.spark_vector.loader.Main $SPARK_VECTOR/loader/target/spark_vector_loader-assembly-master.jar load csv -sf hdfs://namenode:port/tmp/file.csv
-vh vectorhost -vi VI -vd databasename -tt vector_table -sc " " -vu <user> -vp <password>
Usage: Parquet
Loading Parquet files into Vector with Spark:
spark-submit --class com.actian.spark_vector.loader.Main $SPARK_VECTOR/loader/target/spark_vector_loader-assembly-master.jar load parquet -sf hdfs://namenode:port/tmp/file.parquet
-vh vectorhost -vi VI -vd databasename -tt vector_table -vu <user> -vp <password>
Usage: ORC
Loading ORC files into Vector with Spark:
spark-submit --class com.actian.spark_vector.loader.Main $SPARK_VECTOR/loader/target/spark_vector_loader-assembly-master.jar load orc -sf hdfs://namenode:port/tmp/file.orc
-vh vectorhost -vi VI -vd databasename -tt vector_table -vu <user> -vp <password>
List of options
A full list of of options can be retrieved with:
spark-submit --class com.actian.spark_vector.loader.Main $SPARK_VECTOR/loader/target/spark_vector_loader-assembly-master.jar load --help
Spark-Vector provider
The Spark-Vector provider is a Spark application which serves Vector requests for external data sources. Documentation can be found here.
Building
sbt provider/assembly
Unit testing
Spark-Vector Data Source
sbt '; set javaOptions ++= "-Dvector.host=<host> -Dvector.instance=<instance ID> -Dvector.database=<database> -Dvector.user=<user> -Dvector.password=<password> -Dprovider.sparkHome=<path to spark distribution> -Dprovider.jar=<path to spark_vector_provider assembly jar> -Dprovider.sparkInfoFile=<path to spark_info_file shared with Vector>".split(" ").toSeq; test'
Spark-Vector Loader
sbt loader/test
Spark-Vector Provider
sbt provider/test
License
Copyright 2021 Actian Corporation.
Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0