edumeet/edumeet-media-node

Auto Scaling Question

Closed this issue · 2 comments

Dear Team,

We are using AWS infra for edumeet services. we are working on autoscaling feature for which we need to load several media-node instances automatically based on connections or CPU usage.

Now challenge here is how can we provide medianode URLs on edumeet-roomserver with out disconnection.

Thanks

havfo commented

There is actually a very crude way to do that right now. This will be a lot simpler when the media nodes are managed through the management server, but for now, here goes:

  1. Connect to the room server interactive server by running yarn connect
  2. Type t and press Enter. You should now be in terminal mode, and have raw access to the node process (BE CAREFUL!)
  3. Do the following commands, and change the appropriate values:
const MediaNode = serverManager.mediaService.mediaNodes.items[0].constructor
const KDPoint = serverManager.mediaService.mediaNodes.items[0].kdPoint.constructor

let newNode = new MediaNode({ id: 'some-random-string', hostname: 'some.host.here', port: 3002, secret: 'lsdkfmlk', kdPoint: new KDPoint([ 10.394, 63.430 ]) })

serverManager.mediaService.mediaNodes.add(newNode)
serverManager.mediaService.loadBalancer.kdTree.addNode(new KDPoint([ 10.394, 63.430 ], { newNode }))
serverManager.mediaService.loadBalancer.kdTree.rebalance()
  1. Make sure you update the config file so that the media node is also initialized on the next restart of the server

The client now has graceful fail-over to a new media-node if the current one terminates/fails in some way.

A media-node can now be instructed to drain on a timer. Any existing rooms on the media-node will be allowed to remain on the node, but no new rooms are allowed. To drain, connect to the interactive console on the media-node and issue the drain command with an optional number of seconds as a parameter, ex: drain 3600 for 1 hour (default is 1 hour). When the timer expires the media-node will terminate itself.

The client will gracefully fail-over to a new media-node if it is still on a media-node that terminates after draining.

The room-server now properly takes into account media-node health and health checking. This also includes getting information about when a media-node is draining. The room-server will also do an initial health check of all media-nodes when it starts. Whenever a media node comes back up after dying/draining/terminating, it will go back into use as usual.

The room-server now has a way to add a media-node at runtime by connecting to the interactive console, going to the terminal by typing t and hitting enter. Then do serverManager.mediaService.addMediaNode({ hostname, port, secret, longitude, latitude }) with the appropriate settings.