ClickHouse/spark-clickhouse-connector

支持dataframe api 方式读写数据吗?

melin opened this issue · 2 comments

melin commented

支持dataframe api 方式读写数据吗? 提供一个实例

val df = spark.table("test_db.tbl")
df.writeTo("test_db.tbl").append

https://housepower.github.io/spark-clickhouse-connector/quick_start/03_play_with_spark_shell/

melin commented
val df = spark.table("test_db.tbl")
df.writeTo("test_db.tbl").append

https://housepower.github.io/spark-clickhouse-connector/quick_start/03_play_with_spark_shell/

这需要注册这些配置,是静态,
$SPARK_HOME/bin/spark-shell
--conf spark.sql.catalog.clickhouse=xenon.clickhouse.ClickHouseCatalog
--conf spark.sql.catalog.clickhouse.host=${CLICKHOUSE_HOST:-127.0.0.1}
--conf spark.sql.catalog.clickhouse.protocol=grpc
--conf spark.sql.catalog.clickhouse.grpc_port=${CLICKHOUSE_GRPC_PORT:-9100}
--conf spark.sql.catalog.clickhouse.user=${CLICKHOUSE_USER:-default}
--conf spark.sql.catalog.clickhouse.password=${CLICKHOUSE_PASSWORD:-}
--conf spark.sql.catalog.clickhouse.database=default
--jars /path/clickhouse-spark-runtime-3.3_2.12:0.5.0.jar,/path/clickhouse-jdbc-0.3.2-patch11-

我希望动态的,从数据库获取连接信息,读取数据:

val jdbcDF = spark.read
.format("jdbc")
.option("url", "jdbc:postgresql:dbserver")
.option("dbtable", "schema.tablename")
.option("user", "username")
.option("password", "password")
.load()