ask_sync equivalent in ipfs branch
buhrmi opened this issue · 15 comments
Hi,
I'm quite intrigued by your implementation using IPFS. But I don't understand how another node on the Network is notified about new events when using IPFS. In the master branch it seems that the ask_sync method takes care of this. How would this work with IPFS? Maybe I just don't see it.
Stefan
Hi, the idea with the ipfs branch, is that nodes publish their current head to a mutable reference (currently, /ipns does this job but badly, work is going on to improve this). Thus, all the magic happens in get_head(remote_id) and the remote node doesn't need to do anything (only his ipfs daemon works). Events are actually downloaded (or read from disk) on-the-fly with parents() and get_event() using ipfs.
Note: I have partly abandoned working on this implementation since I discovered the paper from david mazières about the stellar consensus protocol. I personally think it's superior to swirlds in many points (mainly the open participation model and the fact that anyone can choose the safety/liveness tradeoff). Moreover I think an SCP based key-value store could be used to implement properly the mutable reference system for ipfs (replace ipns/iprs).
Yes, I've seen that the remote node is checking for new events using get_head. However, in the "reference" implementation, a node actively notifies the randomly choose remote node about a new event (gossip). In the ipfs version, there is no gossipping since the originating node is not contacting any other nodes. It's just publishing the event and is hopeing that some random remote node randomly picks up the event. So I was thinking if there is a way to still keep using IPFS but have the originating node actively notify a remote node instead of only publishing a reference to the event.
Actually, the master branch has the same behavior: syncing is one sided and nodes are waiting for someone to ask them for updates. In the sync method, the local node gets an update from the remote one, he updates his tree with a new head and doesn't notify anyone. Double gossiping wouldn't add anything because it would require half the amount of syncs to propagate some event but would take twice as long to do a sync. It is strictly equivalent but single sided syncing is simpler.
Hmm. I can't see it. Where in the IPFS branch code is the local node notified?
Notified of what? The idea of swirld is to pick a random node, get his head event and all the events we don't know about and make a new head whose parents are our old head and the head of the remote node. The remote events we don't know about (including the head of the remote node) can be queried by using an rpc (ask_sync) or by using the ipns references (get_head) but it is exactely the same behavior. A node sends his local events only if someone asks for them, he never actively notifies the other nodes.
If this is true then maybe there is something really counter-intuitive going on with Swirlds. Assuming I'm a local node and an event occurred (for example, a new transaction on the system). I would want to propagate this event as fast a possible so I can come to a consensus really fast on this transaction. I guess that should be desired. But If I just wait for some other nodes to ask me for the last head, time is lost. Therefore I assumed a node would want actively start a sync process (notify another node by telling it about this event) because it wants to build a consensus over its latest events rather than because it wants to know other's events.
Yeah, this part may be a bit counter-intuitive... My personal feeling about this was that the single-side sync was cleaner because you can have a single process mutating the hashgraph (the asynchronous rpc handler has only read access). That way people can't (or at least can less) DoS you by asking you to work on a sync and I don't bother with all the locking/threading stuff! But actually, even in the case you mentioned (settling on a transaction as fast as possible) you are going to need people to cooperate at some point, so it boils down to you waiting for other nodes to talk to each others. Anyway, if everyone has the same syncing frequency as you, the result will be the same (one remotely initiated sync for every locally initiated sync on average at a given node), if they go faster then you're happy and if they go slower, spamming them is surely not a good idea.
Random notes that may help you to dig into my code:
- Maybe you noticed that the code is a bit better organized and simpler in the ipfs branch, I did some refactoring that I should merge back.
- There is a small error in the
can_seeinvariant, it should read stores for each event ev and for each member m the latest event from m that ev can see (because computing strongly seen events from lower rounds is useful for the votes).
i see, thanks... btw, can I ask how you have been testing the ipfs branch? multiple PCs? I'm trying to write a javascript implementation and I'm trying to make it compatible with yours (same data structure for objects and links in the mdag). but i can only run one ipfs daemon on one pc, and since it's providing the crypto infrastructure, I can only run one hashgraph node at a time.
hum shame on me, actually I didn't really test the ipfs branch (in the sense that I tested most new functions like querying the HTTP API but I didn't test a small network with interacting nodes). I was looking to test it extensively using iptb but as I'm not really used to Go I didn't manage to make it work in the small amount of time I had (it shouldn't be too difficult however, you only need to dig a bit in the source code to get the API url for each test-node).
I will try to make this repository clean as soon as I have the time (these tests scripts and maybe a bit more comments in the code).
ah, no shame. i'm pushing lots of untested code all the time lol ... would be cool. maybe we can get to a state where the python implementation and javascript implementation will actually be able to maintain a consensus together. that would be rather beautiful I think. I'm trying to extend the hashgraph algo to allow it to run in a network where nodes can be dynamically added (not pre-configured). But it's really difficult to find a way to deal with nodes that leave the network and never come back. I was talking to Leemon and he said that famous witnesses could be used for a mechanism that goes something like "if the famous witnesses didn't see any events from a node since X amount of time, disregard that node from future consensus rounds and declare it 'dead'". That's something that SCP can not do afaik.
Yeah, actually, one can extend the swirld protocol with either quorum (node set) changes or even stake changes by embedding them in a transaction, once the transaction is settled, the new settings are in applied (I think that's what he meant in the second whitepaper about extentions). One should just react in time to have enough nodes still alive to settle on the transaction and not get stuck.
My about SCP is quite the opposite: SCP is designed for open membership, there is no need for a global member list as one can change it's quorum set more or less whenever he wants. The quorum sets are embedded in every protocol message (it's done efficiently but that's the result). On the other hand, swirld has a centralized consensus model (using the stellar vocabulary) because there is this member list. More specifically, swirlds could be modified slightly to have open-membership, but then you have to come up with a registration mechanism (distributed or not) that is not vulnerable to sybil attacks (either with an invitation system, a consensus based system as I said earlier or with a proof-of-whatever protocol) and that's not easy given that I don't like the unfair side of invitations or insider-consensus and that I don't like that ugly proof-of-work hack either.
For me, the only real problem with SCP, is the quorum-set management. This must be done by the real admin behind each node because if you automate anything then you would surely be vulnerable to sybil attacks. But doing it by hand means you have to take care that you are strongly enough connected to the rest of the network. One solution might be to detect nodes that we mostly agree with since a long time (or any other condition like that) and suggest to the user to add them to the quorum set. I'm not fond of the tier-ed approach suggested in the paper (and promoted by stellar) because on the long term it induces a power disparity between people and that's exactly what distributed consensus is fighting (imho).
I don't think it's a good idea to put protocol-level functionality into the actual transactions. The transaction payload should be reserved exclusively for application-level usage imho.
When you say that in SCP there is no global ("forced") member list, we come back to my original point about the non-existent ask_sync method. In an IPFS-based implementation of hashgraph, one could also build his own little consensus-"bubble" (similar to a Quorum Slice) by only adding the IPFS nodeId to his list of known nodes (multiaddrs) that he personally decides and wants to maintain consensus with. Actually, I now think an invitation system would work really well for this scenario.
I do agree that protocol-level things should not be in transactions but we could set a new field in the transaction specifying if they are user-payload some protocol message. It makes sense if you see changing membership as a second-level feature of the protocol.
Hmm that implies that when peoples talk about being convinced by 2/3 of the nodes they don't talk about the same set... I don't think it would simply work™ because all the proofs from the paper assume the members set is the same for every one. They would most likely be false if we change that (and you can see how much more complicated the proofs/protocol design of SCP is to work in that case). What definitely works is to sync with only some nodes (so that nodes don't need to know everybody) but that wouldn't even help to scale well because I don't think we can really run the protocol with algorithms in less then O(number of nodes) (time and space) (we do have to cast the vote of every witness and every node has a witness for every round).
I think that fundamentally swirlds can't scale well because things aren't sub-linear: you must in some way or another know everybody in the network. That's not intrinsically bad as you could want to run a distributed db in a moderately sized set of nodes but in that case you most surely can manage a external stake/membership system so it's maybe not so relevant to care about advanced management with swirlds.
I'm sorry to maybe shut down some dreams of a simple algorithm for strongly consistent highly distributed database! Since I wrote that implementation I convinced myself it couldn't work well but writing that last post made my ideas a bit more clear... Maybe you have seen my project of a database agnostic SCP implementation in Rust (still at very early stage). However this is surely not gonna take a month of work to get working as it is much more complex and I have some other things to do right now :/