Redshift query abort without throwing exception in spark job

[hotstar-xia]

Hi, we are using the following command to unload some query results from redshift to s3.

      .option("url", s"${config.redshiftUrl}?user=${config.redshiftUser}&password=${config.redshiftPassword}")
      .option("query", sql)
      .option("tempdir", s"s3://${config.redshiftTempS3Bucket}/redshift/temp_data/download/$segId")
      .option("forward_spark_s3_credentials", "true")
      .load()

But everytime when redshift abort our query, the spark job is still running and didn't receive any exceptions or signals. Do you have any suggestions on exception handling?

[88manpreet]

That is most likely due to spark session not being closed.
I am assuming you created or using a spark session in the above code.
Could you try adding sc.stop() in your exception handling?