Pulsar - Spark adapter for scala 2.11

Question

Pulsar - Spark adapter for scala 2.11

aditiwari01 opened this issue 3 years ago · 8 comments

The pulsar-spark adapter available seems to be using spark-streaming-2_10.
Is there any plan for adapter for spark_2_11?

Answer 1 · 2021-12-10T08:43:27.000Z

There is no much activity on the Spark adapter.

Would you like to send a patch ?

Answer 2 · 2021-12-10T08:55:58.000Z

I'm exploring options as of now. If I end up writing a patch, would definitely send it.

Answer 3 · 2021-12-20T08:43:08.000Z

The current available spark pulsar adapter seems to be unreliable. I've written a reliable connector with addition consideration for back-pressure and rate limit. Currently testing around it. Can raise a PR if that's fine.

Answer 4 · 2021-12-20T15:43:54.000Z

Can raise a PR if that's fine

Sure! thanks

Answer 5 · 2021-12-26T14:08:56.000Z

@eolivelli
I have raised a PR with a new receiver. I'm yet to add examples/documentation for the same, but would appreciate if you can go through it and provide the review.

Answer 6 · 2021-12-26T14:11:20.000Z

Also can you approve running workflows so I can check if there are any issues.

Answer 7 · 2021-12-26T18:09:54.000Z

@eolivelli Need a small help. In local I've been building with skip License. Flow is failing because of
Some files do not have the expected license header

How can I add license to the new files that I have added?

Answer 8 · 2021-12-28T18:47:30.000Z

@aditiwari01 - try running mvn initialize license:format from the base directory of the project. Let me know if that doesn't work.