cordon-thiago/airflow-spark

Error when try to save spark dataframe to .csv file

leacostae opened this issue · 1 comments

try 1:
df.write.format('csv').mode('overwrite').save(.'/home/my.csv', header=True, index=False, sep='|')
try 2:
df.coalesce(1).write.option('mode', 'append').option('header', 'true').csv("./home/my.csv")
try 3:
spark_df.repartition(1).write.format('com.databricks.spark.csv').save("./tmp/data.csv", header='true', index=False, sep='|')

Don't save any dataframe to a csv file, always show the same error (see below)
imagen

I use another solutions, converting the spark dataframe to a pandas dataframe and save it later ;)