Sharing D3D12 device with HIP by LUID unsupported
Nielsbishere opened this issue · 9 comments
Unfortunately I seem to be unable to explicitly create the same device that's being used for D3D12. D3D12 only supports exposing LUID, while vulkan supports both. Is it possible to add support for creating device from LUID or does HIP prevent this? It would also be very handy to have a debug message when querying by LUID if no devices support LUID.
As I only found this out when manually looping over physical devices and their properties.
OIDN supports creating a device from LUID. oidnNewDeviceByLUID does exactly this but this does not guarantee that all GPUs in the system actually report the LUID. HIP unfortunately does not support querying the LUID, and we can't do anything about this. Perhaps you could try requesting this feature from AMD but HIP has very limited support on Windows.
Why should a debug message be printed if a LUID is not found? You should call oidnNewDeviceByLUID and then check the return value and/or the error status. If the specified LUID was not found, you will get null and an error. There are no guarantees that the specified LUID will be found, so such a failure should be expected by the application. In this case, buffer sharing should not be used, and the application should fall back to copying through the host. This is described in the documentation in more detail.
Fair enough. I will be filing an issue with amd then. Does arc (e.g. sycl) support LUID or not? And does this mean sharing between d3d12 and amd is completely impossible? Because afaik the uuid and pcie of the device are both inaccessible from dxgi (at least the ones returned from oidn's helper functions don't match it). Maybe I can try hacking something in there by using vulkan to query the uuid + LUID combo from the LUID acquired from dx.
Yes, Intel GPUs and SYCL support LUID (just make sure to have a recent enough driver), and buffer sharing is fully functional. Only HIP has such issues.
The lack of LUID support in HIP does not necessarily mean that sharing is impossible, but it still might not work due to other reasons. Actually, the LUID is needed only for selecting the GPU, nothing else. You can still share buffers if you create the OIDN device on the right GPU in some other way. If you have a single AMD GPU, this should be trivial: just create a device with OIDN_DEVICE_TYPE_HIP. This should not be done in production code but it's perfectly fine for an experiment.
Regarding UUID with HIP: I don't think this would help either because my experience is that HIP returns a different UUID than Vulkan for the same GPU. This might depend on the driver version and OS though, I'm not sure.
Thanks for your help. I know that just new device should work, guess I'll test it for R&D only and will raise this with our AMD rep. Will of course need that before production.
As an update: our AMD rep has replied and will be passing this along to the HIP team (more likely the ROCm team, since HIP supports querying it but only for cuda backend).
Thanks for the update. If/when ROCm will add support for querying the LUID, we will add it to OIDN.
Hi @atafra I got a response from someone else saying it might be possible to query LUID from the PCIE bus which should be available to us. https://github.com/scarsty/mlcc/blob/master/vramusage.h this shows how he does it.
I'm glad to say that AMD got back to us; #230. They've added support for LUID querying into HIP, so it should be possible to add now 🎉