Please support spark 3.0

Question

Please support spark 3.0

AceHack opened this issue 4 years ago · 10 comments

AceHack commented 4 years ago

When I try to write to dynamo from a different source other than dynamo on spark 3.0, I get the following error:

DefaultSource does not allow create table as select

AceHack commented 4 years ago

Yes

Answer 1 · 2020-06-23T06:34:00.000Z

Does this work on Spark 2.X.X?

Answer 2 · 2020-07-14T07:13:40.000Z

There is an expectation of support in Spark 3?

Answer 3 · 2020-07-15T23:43:27.000Z

Yes please support spark 3, this is the last package I need to upgrade.

Answer 4 · 2020-07-16T05:56:58.000Z

We'll try to prfioritize this, since there seems to be so much demand.

Answer 5 · 2020-07-23T09:17:18.000Z

Release v1.1.0 is on it's way to Maven Central. I have upgraded to Spark 3, and as a result future releases will be on Scala 2.12 only and incompatible with Spark 2.x.x.
Closing this issue. Please open new ones if there are bugs in the integration to Spark 3.

Answer 6 · 2020-07-23T15:16:11.000Z

Spark 2.4.x supports scala 2.12

Answer 7 · 2020-07-23T16:33:12.000Z

How long before we should see the new version on maven?

Answer 8 · 2020-07-23T16:34:51.000Z

Or do you have a snapshot repo I could point to in the meantime?

Answer 9 · 2020-07-24T07:09:18.000Z

The package is up here now:
https://search.maven.org/search?q=spark-dynamodb

You are right that Spark 2.4.x supports Scala 2.12, but the Data Source API has diverged significantly in Spark 3 from how it looked in Spark 2, so the code is no longer backwards compatible.