Hot swapping PseudoKV contracts

[MeijeSibbel]

Say we have a contract-set with 40 hosts, of which we only need 20 at a time (e.g. 5-of-20), would it be possible to temporarily "suspend" a host/contract from this set in order to carry out operations on the contract?

A specific use case of this would be when contracts are running out of funds. Currently we wait until 20 contracts are out of funds, we disable and swap the contract-set for a new set and renew the old contracts with more funds. After renewal is successful we place the set back on the waiting list. This does however require a contract-set swap which might be unnecessary if we are able to individually suspend and renew contracts during a short period of time. When renewed the contract can be reactivated by PseudoKV and business as usual. This way we only have to swap a contract set when more than 20 hosts are unresponsive.

Is this over-complicating things or is there some viability to this approach?

Thanks.

[lukechampine]

Interesting. HostSet could definitely be extended to accommodate this, although it would need to gain a mutex. I think we'd only need to add one method, RemoveHost, which would close the session and remove the host from the set. Then, you'd renew the contract, and finally you'd call AddHost using the renewed contract. If that API sounds good, I can implement this pretty quickly.

[MeijeSibbel]

cc @jkawamoto to confirm API^^

[jkawamoto]

Looks good to me.

[lukechampine]

I've pushed a RemoveHost method to kv, but it doesn't have locking. Adding locks will have far-reaching consequences, so I only want to do it if necessary. @jkawamoto, can you see if you can integrate RemoveHost without requiring locks? I think it's possible if you make sure you only do the remove->renew->add process between other uses of PseudoKV.

[MeijeSibbel]

@lukechampine so we can't call RemoveHost while kv is uploading data?

[lukechampine]

Correct, you'd have to wait for the current upload to finish. Is that a non-starter?

[MeijeSibbel]

With current upload do you mean all uploads or just the file that is uploading at that specific moment? can we do the following; complete file upload --> remove host --> renew in background --> while contract is renewing --> upload new file ?

If we can't upload any further data until the contract is renewed and we call AddHost again then this won't work as it forces the user to stop uploading and will require a contract set swap (what we are doing now), which is exactly what we are trying to avoid with this technique.

[lukechampine]

you would have to do something like the following:

1. (multiple uploads currently running...)
2. Pause new uploads and wait for existing uploads to finish
3. RemoveHost on the contract(s) requiring renewal, and begin renewing
4. With renewals occurring in the background, resume uploads
5. When renewals complete, pause new uploads and wait for existing uploads to finish
6. AddHost on the renewed contract(s)
7. Resume uploads

So the actual amount of downtime (time spent uploading nothing at all) will be quite small; just long enough to call RemoveHost. However, throughput will still be reduced, because at each "synchronization point," you won't be able to begin new uploads until the existing uploads have finished.

[MeijeSibbel]

I'd like to confirm with @jkawamoto but i think this shouldn't be a problem as the amount of downtime is similar to what occurs now when a contract set has to be swapped.

[jkawamoto]

Since we currently close and recreate the host set after uploading files finishes, I'm wondering how RemoveHost could improve the current process... I also think "Pause new uploads and wait for existing uploads to finish" is one of the trickiest points. We now swap the contract set to pause new uploads. If RemoveHost requires pausing new uploads, we still need to swap contract sets, and the goal of this issue isn't achieved.

[MeijeSibbel]

We now swap the contract set to pause new uploads. If RemoveHost requires pausing new uploads, we still need to swap contract sets, and the goal of this issue isn't achieved.

@jkawamoto I don't see why this is true. We can pause the upload process, RemoveHost the host in question and continue uploading to the existing contract set (assuming there are enough responsive hosts left). When the contract is renewed we just pause the uploads again and AddHost. There is no need to swap the contract set. To make this more efficient we might want to do this with say 2-3 contracts at the same time to not pause the upload process too frequently.

[MeijeSibbel]

Since this has been implemented i think we can close it.