streamnative/pulsar-spark

[Bug] Same named durable subscription created when using custom subscription name via readerconfig

grssam opened this issue · 1 comments

Describe the bug
Consider a scenario:

predefinedSubscription option is not provided.
pulsar.reader.subscriptionName option is provided

Additionally, assume #107 is fixed.

In this case, the metadatareader will use the subscription name from either readerConfig or create a subscription based on the logic here .

In case it reuses the subscription name from reader config, the code logic in PulsarProvider::createSource is as following:

  • Create a metadata reader
  • Setup cursor
    • Create a new durable subscription as provided in readerConfig
    • Get offset info and note down the latest offsets
  • Create PulsarSource object
    • Setup a reader using same subscription name as mentioned in readerConfig

This fails as during reader creation, it would observe that a durable subscription already exists (created by the metadatareader)

In case metadata reader uses a different logic to figure out the subscription name, then doing any sort of custom authorization logic is tricky.

** Proposal **

Use a different/new config to inpput a custom subscription name that would be used by the metada reader class only.

OR

use a non-durable subsription in metadata reader here

nlu90 commented

@grssam After several recent PRs, now the PulsarHelper (previously PulsarMetadataReader) won't use the subscription name provided inside the reader config to create the subscription.

It will either use the predefinedSubscription or derive it from the driverGroupIdPrefix to create a meta subscription to manage the progress of the spark job.