piccolbo/dplyr.spark.hive

add back possibility to specify schema in load_to in spark

Closed this issue · 1 comments

Given the limits in schema inference (takes a character for integer, understandably, no support for date, less understandably) maybe it's better to have this back, even if it is less convenient. Maybe also use in tests for safety. Maybe tests will never pass without this.

CREATE TABLE cars (yearMade double, carMake string, carModel string, comments string, blank string)
USING com.databricks.spark.csv
OPTIONS (path "cars.csv", header "true")

This clashes with some suspected bug in spark csv, see https://github.com/databricks/spark-csv/issues/184