Replication
Opened this issue · 3 comments
In your README you mention you are planning on doing master-master replication. Any thoughts on what approach you will take?
For strfry I'm doing it all at the nostr websocket level. Relays will stream events to each-other. To fix any inconsistencies, relays can periodically sync with each other using negentropy:
https://github.com/hoytech/negentropy
https://github.com/hoytech/strfry/blob/next/docs/negentropy.md
There's a reference implementation in C++ too.
At this time, I have only outbound replication in preparation. It uses system, how events are stored in an internal documents-oriented database (which is also being in development/testing for this project). Each event has DocID (unsigned) number which is increased with each new event or an update. So the approach is easy, each replication task remembers DocID of last posted event and continues to posting new events in the order as the were received and stored in the database. Once whole database is replicated, replication tasks are subscribed to an internal publisher and they replicate everything they receiving.
Posting duplicated event is not error.
The outbound replication can be configured as master-to-master when one node replicates to other and the other node replicates to the first node. The problem is to identify the connection to prevent replication of replicated events back to the original node. The replicator should be somehow identified. As there is no way to identify clients, i plan to use AUTH command to identify replicators and source node (node's url will be included in AUTH response). Response to AUTH command won't be mandatory (unless this will be required by configuration option). But replicators must response to AUTH request.
For inbound replication, I will probably use timestamps to find new events. The inbound replication is intended for replication from foreign nodes. I also plan to create custom version of REQ command which includes DocID with the same meaning as in case of outbound replication. This will be used when the inbound replication use my server as a source. An information about this possibility can be carried also by AUTH command.
Thank you for the response -- very cool!
If I understand it right, you will be streaming events to each end-point and tracking the last DocID sent for each one so you know where to resume sending from in case they get disconnected. Makes sense.
My goal with negentropy is a little different. I want to make it so that two nodes that have never talked before can quickly compare their (possibly very large) DBs and determine which events need to be exchanged in either direction. Nostr is a very chaotic and "eventually consistent" system and it's quite common that relays will have overlapping subsets of events gathered independently.
Anyway, just wanted to say I like your project. Us C++ folks should stick together since there aren't too many of us building on nostr!
I was inspired by how CouchDB has it. Unfortunately, within the framework of the NOSTR protocol, this would mean defining new commands. CouchDB sends a manifest to the neighboring node with a list of documents (IDs) it has, and the other node responds with a list of IDs it has in addition, or a list of IDs it is interested in (depending on the direction chosen). And then the missing documents are transferred. Once state is synchronized, live "replay" of received documents to the other node continues (if enabled)
Perhaps this could be done with ephemeral events that would include these lists in the "e" tag. But both nodes have to "know" this feature