Global Store Adding change log topic

[saggarsunil]

I have a situation where i want to maintain my own global table structure. Essentially, Global Store is linked to a processor and i use some calculations to create key and then store it to key-value store.

Since global state store doesn’t have change log topic, it uses original topic as change log. And in case of state restore, it just loads the topic data to global table (which will be wrong) since we constructed our own key & value.

Topic Data

Key, value
1234,abcd
4567,defg

StateStore
Key, value
12ab, abcd
45de, defg

My requirement is to create a global store with my custom keys ( not the one which comes directly from topic). Any solution ?

[mjsax]

What you want to do is not supported. A global store is a read-only materialized view of the data from the corresponding input topic. The only way to "write" into the store is by writing into the topic.

Does writing into the topic (instead of directly into the store) work for you?

[saggarsunil]

We are able to write to global state store. So, it is not read only. Also, does that mean any updates to the records will not be processed ?

[mjsax]

It's a flaw in the API. You can "write" but you need to write the data from the source topic unmodified. Hence, you cannot write anything "arbitrary" or anything based on some computation. The data in the store must be the exact some data as in the input topic atm.

To remove this limitation we need to address those two tickets:

• https://issues.apache.org/jira/browse/KAFKA-7663
• https://issues.apache.org/jira/browse/KAFKA-8037

[saggarsunil]

Thanks Matthias. We created our own changelog topic to workaround this limitation.

…

On Wed, Mar 11, 2020 at 3:04 PM Matthias J. Sax ***@***.***> wrote: It's a flaw in the API. You can "write" but you need to write the data from the source topic unmodified. Hence, you cannot write anything "arbitrary" or anything based on some computation. The data in the store *must* be the exact some data as in the input topic atm. To remove this limitation we need to address those two tickets: - https://issues.apache.org/jira/browse/KAFKA-7663 - https://issues.apache.org/jira/browse/KAFKA-8037 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#322?email_source=notifications&email_token=ABPTVNTYJGXLTXENALJ6K5TRG5LJ7A5CNFSM4K476PJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOOZYPA#issuecomment-597531708>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABPTVNTRCVERQM3L6AARLDLRG5LJ7ANCNFSM4K476PJA> .

-- ------------------- Thanks Sunil Saggar

[mjsax]

Can you elaborate how you do this? Curious to learn about it.

[saggarsunil]

Work around was mentioned in the following link https://stackoverflow.com/questions/59029964/kafka-streams-use-cases-for-add-global-store ```By the way, if you want the latter behavior, you can approximate it right now by applying your transformations and then using to(my-global-changelog) to manufacture a "changelog" topic. Then, you'd create the global store to read from your my-global-changelog instead of the input.```

…

On Wed, Mar 11, 2020 at 9:22 PM Matthias J. Sax ***@***.***> wrote: Can you elaborate how you do this? Curious to learn about it. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#322 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABPTVNWZ7ALJHZISB6657F3RG6XSHANCNFSM4K476PJA> .

-- ------------------- Thanks Sunil Saggar

[mjsax]

I see -- you don't really create a changelog topic, but follow my suggestion from above:

Does writing into the topic (instead of directly into the store) work for you?

Maybe this statement was too brief to be clear....