[Bug] Same named durable subscription created when using custom subscription name via readerconfig
grssam opened this issue · 1 comments
Describe the bug
Consider a scenario:
predefinedSubscription
option is not provided.
pulsar.reader.subscriptionName
option is provided
Additionally, assume #107 is fixed.
In this case, the metadatareader will use the subscription name from either readerConfig or create a subscription based on the logic here .
In case it reuses the subscription name from reader config, the code logic in PulsarProvider::createSource
is as following:
- Create a metadata reader
- Setup cursor
- Create a new durable subscription as provided in readerConfig
- Get offset info and note down the latest offsets
- Create PulsarSource object
- Setup a reader using same subscription name as mentioned in readerConfig
This fails as during reader creation, it would observe that a durable subscription already exists (created by the metadatareader)
In case metadata reader uses a different logic to figure out the subscription name, then doing any sort of custom authorization logic is tricky.
** Proposal **
Use a different/new config to inpput a custom subscription name that would be used by the metada reader class only.
OR
use a non-durable subsription in metadata reader here
@grssam After several recent PRs, now the PulsarHelper
(previously PulsarMetadataReader) won't use the subscription name provided inside the reader config to create the subscription.
It will either use the predefinedSubscription
or derive it from the driverGroupIdPrefix
to create a meta subscription to manage the progress of the spark job.