[FEATURE REQUEST] Run Tests on Vast.AI with a GitHub Action

[movchan74]

Feature Summary

Right now the tests are run on GitHub Actions with a CPU only. All the GPU tests are run locally. It would be beneficial to run the GPU tests on Vast.AI with a GitHub Action.

Justification/Rationale

• Running the GPU tests on Vast.AI with a GitHub Action will allow us to run GPU tests from the CI/CD pipeline.
• Potentially it will allow us to run the tests on different GPUs and configurations.
• Save developers time by not having to run the tests locally.

Proposed Implementation

Vast.AI provides a REST API that allows us to create and manage virtual machines. We can use the API to create a virtual machine with a GPU, run the tests, and destroy the virtual machine.

The implementation can be done in the following steps:

• Create a virtual machine with a GPU using the Vast.AI API. We already have a template for creating an instance for SDK development.
• Log in to the virtual machine, git clone the repository, install the dependencies, and run the tests.
• Destroy the virtual machine after the tests are done.

The implementation can be done in a GitHub Action. I would suggest to trigger the action manually (at least at the beginning) to avoid unnecessary costs.

Also, we can keep the virtual machine in the stopped state that only charges for the storage instead of destroying it after the tests are done. This will allow us to run the tests faster and probably even cheaper because we won't have to create and destroy the virtual machine every time we run the tests (which takes time and costs money).

As a stretch goal, we can run the tests on different GPUs and configurations to test the SDK on different hardware.