Add a historical event log that can be used to query and re-hydrate Kafka streams from Neo4j using Spring Data
Opened this issue · 1 comments
One of the critical problems that often goes unaddressed in a CQRS/ES microservices architecture is "how do I query the history of events?"
Secondly, people often ask how they can query the history of events to re-hydrate a new aggregate service with historical domain data.
While I think that Kafka streams provide a nice abstraction for event sourcing, I do think that it has a few deficiencies when it comes to querying streams. Here, I propose an idea that uses Neo4j to store a linked list that replicates topics from Kafka, and preserves ordering and indexing. For this purpose, I will design a "streams" graph using a new Neo4j database and a Spring Boot application. An example of the graph can be found below.
The goal here is to provide easy access to teams who are looking to ingest event data from Neo4j as a collection of new events. Neo4j provides a connector to Kafka[0] which will stream event projections back into a Kafka topic, which can then be used to create an event sourced set of aggregates connected to a user.
Going to implement this with Apache Pinot