New context type that can dispatch to N actual contexts
pietern opened this issue · 0 comments
pietern commented
This is what we do in PyTorch upstream today but it would be good to move the functionality into Gloo. This would be a new type of context that wraps N contexts and load balances over them. If we were to create these wrapped contexts for multiple devices, this would be a natural place to apply load balancing as well.