Timestamp RSS and Twitter

Question

Timestamp RSS and Twitter

DerAlexmeister opened this issue 3 years ago · 3 comments

Add a timestamp in the RSS-Feed and Twitter data which will be put into kafka topics.

Answer 1 · 2021-07-12T15:06:26.000Z

If I am appending an event with the timestamp, when it got scraped, there is a good chance, that we are getting the same event twice.
e.g. if we are scraping the last thirty tweets of a twitter account every day, the chances are high, that we are getting the same tweet on both days, but with different timestamps.
Maybe we can check by the name, if there is already such an event and get rid of the duplicate, if we scrape one?

Answer 2 · 2021-07-13T08:26:58.000Z

We should discuss this.

Answer 3 · 2021-07-13T08:30:12.000Z

Event sourcing. Might this be an option.