Support running Opt on non default cudaStream

[gerwang]

I want to launch another CPU thread (call it CPU thread B) and use Opt on that thread, and I don't want functions like Opt_ProblemSolve to block GPU execution invoked on CPU thread A.

Does Opt support running Opt_ProblemSolve on different cudaStream? If Opt can only run on default stream, then if I schedule all GPU tasks invoked by thread A on a non-default cudaStream, can two CPU threads run without blocking each other?

By the way, does Opt call cudaDeviceSynchronize internally? If so, I'm afraid those two threads will block each other.

[Mx7f]

Opt does not currently support non-default cudaStreams. There are three levels of support we could offer:

1. Internally create and use a non-default stream internally for all operations.
2. Expose the stream parameter for reading via the C API,
3. Expose the stream parameter for writing via the C API (probably in Opt_InitParams).

2 and 3 worry me because we'd need a portable way to expose the stream in the C API, which currently includes no headers, especially not cuda headers. We could probably be fine passing streams as (void*) though.

The first option is easier, and likely sufficient for your purposes. This will require non-trivial, but straightforward changes to API/src/solverGPUGaussNewton.t and API/src/util.t to use a stream when possible. This includes making many cudaMemcpy's into async to avoid unneccessary synchronization.