Jitify doesn't forward `-I` options to NVRTC.
Robadob opened this issue · 2 comments
We have a Linux box running Ubuntu 16.04.
This has the Ubuntu packaged version of CUDA installed (very old 7.5 or similar), which leaves some directories on path.
It also has all of the modern CUDA versions which must be enabled via module load.
When testing our RTC code on this machine (using CUDA 10.1), all runtime compilation failed with.
---------------------------------------------------
--- JIT compile log for rtc_test_func_program ---
---------------------------------------------------
/usr/include/cuda.h(229): error: invalid redeclaration of type name "CUuuid_st"
__nv_nvrtc_builtin_header.h(1549): here
/usr/include/cuda.h(231): error: invalid redeclaration of type name "CUuuid"
__nv_nvrtc_builtin_header.h(1552): here
2 errors detected in the compilation of "rtc_test_func_program".
After some debugging, it appears that NVRTC loads cuda.h, not Jitify.
When Jitify is processing options, to detect passed include directories it removes them from the options.
https://github.com/NVIDIA/jitify/blob/master/jitify.hpp#L2579
As such when NVRTC tries to locate cuda.h, and it can only look on path, hence finds the really old one, rather than the one in the provided include dir.
The workaround for this is to pass the CUDA include dir to Jitify with --include-path= rather than -I, so Jitify doesn't intercept and remove it.
Is there a reason that all include directories are not forwarded to NVRTC?
It's unclear to me, how NVRTC finds cuda.h on Windows, as this file isn't on the path.
Jitify intentionally hijacks the -I flags so that it can handle all header loading and provide features that NVRTC doesn't (such as only warning about missing headers, loading headers through different mechanisms etc.). If we pass the flag to NVRTC then some of these features no longer work.
There is a known issue in NVRTC on Linux where "/usr/include" is always used as an include path (this will be fixed in a future release). I think this is what's causing your problem.
We work around this issue for standard system headers by using a list of preincluded builtin headers here:
Line 2193 in de1797c
But this won't help you because we don't have a builtin version of cuda.h.
I can't immediately think of a better workaround than the trick you found with using --include-path (which is a bug in Jitify that we will have to fix at some point!). Actually you could probably use --pre-include=/usr/local/cuda/include/cuda.h as a better alternative.
Thanks, I've just tested it and --pre-include= also works (we're deciding the cuda location based on an env var though, so we're not hardcoding it to that path).