Reliance on llvm-config for clang's path doesn't work with nix

Question

Reliance on llvm-config for clang's path doesn't work with nix

acowley opened this issue 7 years ago · 22 comments

I've had to incorporate a patch in the nix derivation I use for cquery to force the build to rely on the clang found on PATH rather than using llvm-config --bindir output. The latter gives a useful path for various LLVM executables (e.g. llc, opt, etc.), but, at least with nix, clang is found elsewhere.

What I'm not sure of is how to support this configuration. Should there be a nix variant of the build that we key off of to set such things, or perhaps just a flag to specify which clang executable?

Answer 1 · 2018-01-05T19:53:25.000Z

waf has an option --check-cxx-compiler=/usr/bin/clang++ for specifying the compiler driver.
But yeah this may should be documented more clearly. Appreciated if you can contribute to README.md or https://github.com/jacobdufault/cquery/wiki/Build

Answer 2 · 2018-01-05T20:05:32.000Z

This is not working for me:

[nix-shell:~/temp/cquery/source]$ which clang++
/nix/store/61plv8rx9jvjzk8awp7cxv42gy4g41a6-clang-wrapper-4.0.1/bin/clang++

[nix-shell:~/temp/cquery/source]$ echo $configurePhase
./waf configure --use-system-clang --prefix=$out --check-cxx-compiler=/nix/store/61plv8rx9jvjzk8awp7cxv42gy4g41a6-clang-wrapper-4.0.1/bin/clang++

[nix-shell:~/temp/cquery/source]$ eval $configurePhase
Setting top to                           : /Users/acowley/temp/cquery/source 
Setting out to                           : /Users/acowley/temp/cquery/source/build 
Checking for '/nix/store/61plv8rx9jvjzk8awp7cxv42gy4g41a6-clang-wrapper-4.0.1/bin/clang++' (C++ compiler) : not found 
could not configure a C++ compiler!
(complete log in /Users/acowley/temp/cquery/source/build/config.log)

I should note that configure worked before my patch, finding the desired clang++. The build phase is where things failed.

[nix-shell:~/temp/cquery/source]$ ./waf configure --use-system-clang --prefix=$out
Setting top to                           : /Users/acowley/temp/cquery/source 
Setting out to                           : /Users/acowley/temp/cquery/source/build 
Checking for 'clang++' (C++ compiler)    : clang++ 
Checking for header stdio.h              : yes 
checking for llvm-config                 : /nix/store/imk7znwmgcg2rs4rljiz6cvlfw9jc6bk-llvm-4.0.1/bin/llvm-config 
Checking for clang flags                 : yes 
Checking for library clang               : yes 
Clang includes                           : ['/nix/store/imk7znwmgcg2rs4rljiz6cvlfw9jc6bk-llvm-4.0.1/include'] 
Clang library dir                        : ['/nix/store/8121g71248csyy5c5d9n3ga0fa02xs91-llvm-4.0.1-lib/lib'] 
'configure' finished successfully (1.493s)

Answer 3 · 2018-01-06T04:44:43.000Z

Checking for '/nix/store/61plv8rx9jvjzk8awp7cxv42gy4g41a6-clang-wrapper-4.0.1/bin/clang++' (C++ compiler) : not found

Does the compiler driver path exist?

Answer 4 · 2018-01-06T04:49:00.000Z

Yes, that’s what I was showing with the initial which line.

Answer 5 · 2018-01-07T19:13:46.000Z

wscript is a Python file. You can from IPython import embed; embed() to set breakpoints and inspect variables. Would you like to figure it out by yourself?
BTW, I sometimes play with Haskell but am not persuaded to use Nix. Arch Linux is good enough to me.
BTW 2, recently I checked https://github.com/haskell/haskell-ide-engine and it still did not support find-references.
BTW 3, https://github.com/google/haskell-indexer the author may be willing to cooperate with other language server contributors.

Answer 6 · 2018-01-07T20:04:17.000Z

I don’t think this is a debugging issue. It’s that llvm-config is available, but is not a good source for a path to clang (or libclang). You can see the patch I linked that it is just skipping the use of llvm-config to obtain the path for clang during the build step.

I’m not trying to advocate for nix usage, it is just an example of llvm and clang being installed at different paths.

I appreciate that for many people llvm and clang are installed to the same location, so I do not want to break things for anyone. The question is just if a new flag that could change this behavior in the build phase would be acceptable.

I am using cquery for C++.

Edit: To be clear, I can work around this for nix packaging. I opened the issue because I suspect the independence of llvm and clang will increase as llvm backends for compilers become more common. This will make the use of llvm-config for finding clang less and less reliable.

Answer 7 · 2018-01-07T23:14:02.000Z

From your stdout/stderr dump:

$ which clang++
/nix/store/61plv8rx9jvjzk8awp7cxv42gy4g41a6-clang-wrapper-4.0.1/bin/clang++

Checking for '/nix/store/61plv8rx9jvjzk8awp7cxv42gy4g41a6-clang-wrapper-4.0.1/bin/clang++' (C++ compiler) : not found

And I suppose you used the correct --check-cxx-compiler. This looks very weird to me. Why is the compiler driver not found?

It is nice to have Nix package for cquery, and necessary change in wscript is welcomed if not too complicated. We should also expand https://github.com/jacobdufault/cquery/wiki/Build to include build instructions for some non standard distributions.

Do you have some patch to make correct clang picked on Nix (ideally via --check-cxx-compiler so no extra option is introduced) without deleting the current llvm-config code (which works fine on OpenIndiana and FreeBSD)

Answer 8 · 2018-01-07T23:50:13.000Z

Yes, that is weird. I can look into it. That arises during the configure step, though. Without the —check-cxx-compiler flag the configure step picks up the clang++ on PATH as wanted, but the build step constructs its own path to clang++ using llvm-config.

So there is the question of why my attempt to use that flag isn’t working, and whether or not the build step should do what it’s doing with llvm-config to find clang++ even though the configure step already found a clang++.

The latter is the one that I’m more immediately concerned by, but they’re both concerns. I’ll put some more time into it and update.

Answer 9 · 2018-01-08T01:34:23.000Z

Okay, here's what I've worked out:

I think that if bld.env['CXX'] has a value, we should not compute a path for clang in the build function. That is, rather than if bld.env['llvm_config']: on this line, we should also check that bld.env['CXX'] is not set. The configure step is setting CXX, so why not use it?
I can get llvm-config out of the build environment when building with nix to avoid the problem as long as I specify the --clang-prefix-path configure option, which works out fine. However, bld.env['llvm_config'] defaults to 'llvm-config'. So even if llvm-config is not found by the configure step, the if bld.env['llvm_config']: conditional still evaluates to true! I can fix this by explicitly passing --llvm-config= to the configure step.
I'm not sure I tracked down the --check-cxx-compiler flag you suggested, but it seems to me that it wants one of a set of options defined here. If I set it to clang++, configure completes successfully. Doing that doesn't do anything different than if I leave it unset, but it shows a valid value for that flag. I think perhaps passing an absolute path might not be the right way to use this flag.

I'm not sure if this is a situation that actually matters to anyone, and I can apparently find a path through the configure options to make things work for nix users, so I can close this issue if there's no interest in the above details. I think there is a problem with wcript's design assuming the connection between clang++ and llvm-config, but nix might be a worst-case scenario for this assumption, so if it's workable there perhaps it can be made to work in any configuration.

Answer 10 · 2018-01-08T06:11:33.000Z

Thanks for triage! I hope it is fixed now. If not, feel free to reopen!
Appreciated if you could add Nix build instruction on https://github.com/jacobdufault/cquery/wiki/Build

Answer 11 · 2018-01-09T02:09:09.000Z

I just wanted to take a moment to thank and commend @jacobdufault and @MaskRay for some really remarkable work on this project. Your effort is very much appreciated!

Answer 12 · 2018-01-11T01:26:00.000Z

so this is coming up for me. the company i work at has clang installed globally, b/c they've patched it such that it works w/ our internal build tools that are quite extensive. the clang install they have going does not provide llvm-config.

i first got the configure step through w/o setting the --check-cxx-compiler flag:

$ ./waf configure --use-system-clang --clang-prefix /src/head/depot//third_party/crosstool/v18/stable/toolchain --prefix=$HOME/.local/cquery Setting top to : /usr/local/home/razamatan/src/cquery Setting out to : /usr/local/home/razamatan/src/cquery/build Checking for 'clang++' (C++ compiler) : not found Checking for 'g++' (C++ compiler) : /usr/bin/g++ Checking for header stdio.h : yes checking for llvm-config : not found Checking for clang prefix : /src/head/depot/third_party/crosstool/v18/stable/toolchain Checking for library clang : yes Clang includes : ['/src/head/depot//third_party/crosstool/v18/stable/toolchain/include'] Clang library dir : ['/src/head/depot/third_party/crosstool/v18/stable/toolchain/lib'] 'configure' finished successfully (1.155s)

sure enough, on ./waf build, things get screwy based on acowley's observation that the build step thinks that llvm-config exists b/c it's been defaulted (and hasn't really been detected in my case).

trying to hint the configure step to fully specify the clang++ binary using the --check-cxx-compiler results in the described configuration error:

$ ./waf configure --use-system-clang --clang-prefix /src/head/depot/third_party/crosstool/v18/stable/toolchain --check-cxx-compiler /src/head/depot/third_party/crosstool/v18/stable/toolchain/bin/clang++ --prefix=$HOME/.local/cquery Setting top to : /usr/local/home/razamatan/src/cquery Setting out to : /usr/local/home/razamatan/src/cquery/build Checking for '/src/head/depot/third_party/crosstool/v18/stable/toolchain/bin/clang++' (C++ compiler) : not found could not configure a C++ compiler! (complete log in /usr/local/home/razamatan/src/cquery/build/config.log)

the configured clang++ most certainly exists (it's a symlink to clang, but i changed to to directly clang in case the symlink was the issue, but it still errors the same).

how do i go about using this llvm install w/o having llvm-config?

Answer 13 · 2018-01-11T02:23:46.000Z

@razamatan Something that looks odd to me about that output is in your first log, there,

Checking for 'clang++' (C++ compiler) : not found
Checking for 'g++' (C++ compiler) : /usr/bin/g++

Is /src/head/depot/third_party/crosstool/v18/stable/toolchain/bin not on $PATH? I think the issue is that wscript does not do anything very clever for finding clang. The way it is now relies on it being found either on $PATH or in relation to llvm-config --bindir. The --clang-prefix flag is not used for finding the compiler executable, and --check-cxx-compiler does not take a path to a compiler, but is instead used for manually picking between clang++ and g++ in most cases.

Answer 14 · 2018-01-11T03:42:52.000Z

thanks. setting up my path correctly does fix the configure output to find clang++, but it has no effect on the build.

Answer 15 · 2018-01-11T04:02:10.000Z

@razamatan Just to confirm, that is with a very recent commit?

Answer 16 · 2018-01-11T04:58:48.000Z

@razamatan Oh, and I just realized that I don't see the --llvm-config= flag in your configure invocations. I think you want something like,

./waf configure --use-system-clang --clang-prefix /src/head/depot//third_party/crosstool/v18/stable/toolchain --prefix=$HOME/.local/cquery --llvm-config=

Answer 17 · 2018-01-11T05:27:04.000Z

thanks! that worked

Answer 18 · 2018-01-31T16:02:40.000Z

I'm still encountering this bug with nix and clang-5.0.1

[nix-shell:~/hacking/cquery]$ ./waf configure --use-system-clang
Setting top to                           : /home/mike/hacking/cquery
Setting out to                           : /home/mike/hacking/cquery/build
Checking for 'clang++' (C++ compiler)    : not found
Checking for 'g++' (C++ compiler)        : g++
Checking for header stdio.h              : not found
The configuration failed
(complete log in /home/mike/hacking/cquery/build/config.log)

[nix-shell:~/hacking/cquery]$ head -15 build/config.log
# project cquery (0.0.1) configured on Wed Jan 31 11:02:18 2018 by
# waf 2.0.2 (abi 20, python 20705f0 on linux2)
# using ./waf configure --use-system-clang
#
----------------------------------------
Setting top to
/home/mike/hacking/cquery
----------------------------------------
Setting out to
/home/mike/hacking/cquery/build
----------------------------------------
Checking for 'clang++' (C++ compiler)
find program=['clang++'] paths=['/nix/store/yay7zpv8g65md88l60waa47wamcp8b28-patchelf-0.9/bin', '/nix/store/qd4dsi9qsnfyisiqznhl6h2zdyr2lg0z-paxctl-0.9/bin', '/nix/store/6s3dl13ingr21z93dy6zxcc652wdlcrh-gcc-wrapper-6.4.0/bin', '/nix/store/zg2rbsm898sxgz27ps8vg006gywbcggc-gcc-6.4.0/bin', '/nix/store/07sm9955r98psvdzqzq18nikmvlm06ii-glibc-2.26-131-bin/bin', '/nix/store/qd55j183ym04y43aam889xkimq704rdx-coreutils-8.29/bin', '/nix/store/i8fxs7ppvkayaczq1vwzw9vcqkqrrj02-binutils-wrapper-2.28.1/bin', '/nix/store/mdyy001q67hiks0g24ra53z7ckm4jfr4-binutils-2.28.1/bin', '/nix/store/07sm9955r98psvdzqzq18nikmvlm06ii-glibc-2.26-131-bin/bin', '/nix/store/qd55j183ym04y43aam889xkimq704rdx-coreutils-8.29/bin', '/nix/store/6s3dl13ingr21z93dy6zxcc652wdlcrh-gcc-wrapper-6.4.0/bin', '/nix/store/zg2rbsm898sxgz27ps8vg006gywbcggc-gcc-6.4.0/bin', '/nix/store/07sm9955r98psvdzqzq18nikmvlm06ii-glibc-2.26-131-bin/bin', '/nix/store/qd55j183ym04y43aam889xkimq704rdx-coreutils-8.29/bin', '/nix/store/i8fxs7ppvkayaczq1vwzw9vcqkqrrj02-binutils-wrapper-2.28.1/bin', '/nix/store/mdyy001q67hiks0g24ra53z7ckm4jfr4-binutils-2.28.1/bin', '/nix/store/07sm9955r98psvdzqzq18nikmvlm06ii-glibc-2.26-131-bin/bin', '/nix/store/qd55j183ym04y43aam889xkimq704rdx-coreutils-8.29/bin', '/nix/store/vj00y007jsq4l7z69hml84x2njmfh88x-llvm-5.0.1/bin', '/nix/store/piy10ra6rmszg2hhafb37vryl20p62z7-ncurses-6.0-20171125-dev/bin', '/nix/store/hk3s7q3486lkkipv291m7y498q6rj0pp-ncurses-6.0-20171125/bin', '/nix/store/8fb6nbawi1n7vm6wgc70dilysq6a68hg-clang-wrapper-5.0.1/bin', '/nix/store/5zvimsqh3dwbdmb0sar2r674i14vf4zg-clang-5.0.1/bin', '/nix/store/07sm9955r98psvdzqzq18nikmvlm06ii-glibc-2.26-131-bin/bin', '/nix/store/qd55j183ym04y43aam889xkimq704rdx-coreutils-8.29/bin', '/nix/store/qd55j183ym04y43aam889xkimq704rdx-coreutils-8.29/bin', '/nix/store/8x05myb28xqwixakpz0xhqyb8735kvy7-findutils-4.6.0/bin', '/nix/store/18ak9x2mqr2inprnr4biad04n6pi4bc5-diffutils-3.6/bin', '/nix/store/zgr9zbw7p8gihn9h0nj8ggkcplrd9g7f-gnused-4.4/bin', '/nix/store/rhbm7javg5n5zmhhld7qbi23n5qc0srf-gnugrep-3.1/bin', '/nix/store/va1vrgbmbrr6k80r63cfy1121ypdlrrz-gawk-4.2.0/bin', '/nix/store/zr5z057g5ilp885mp2mlnq2x9djp386w-gnutar-1.30/bin', '/nix/store/l2qzm4i6kfraiwwvr6xvjs8a3qq4iz0i-gzip-1.8/bin', '/nix/store/n16brhwmh08as707294s3b9q6m74jy47-bzip2-1.0.6.0.1-bin/bin', '/nix/store/pi3cnkd4v5d4ncgg1c4m3avmvdin953s-gnumake-4.2.1/bin', '/nix/store/i0ay05pqkbnvpfijm52mmlrp6kmkl80c-bash-4.4-p12/bin', '/nix/store/4s37ffm568sykkjffjjyc7skhampbmi6-patch-2.7.5/bin', '/nix/store/f7jfl4ql2302g07r6hym9dgna2db7772-xz-5.2.3-bin/bin', '/home/mike/.nix-profile/bin', '/home/mike/.nix-profile/sbin', '/home/mike/.nix-profile/bin', '/home/mike/.nix-profile/sbin', '/home/mike/.nix-profile/bin', '/home/mike/.nix-profile/sbin', '/home/mike/bin', '/usr/local/bin', '/usr/bin', '/bin', '/home/mike/bin', '/usr/local/sbin', '/usr/sbin', '/home/mike/bin'] var='CXX' -> ['g++']
['g++', '-dM', '-E', '-']
out: #define __SSP_STRONG__ 3
[nix-shell:~/hacking/cquery]$ which clang++
/nix/store/8fb6nbawi1n7vm6wgc70dilysq6a68hg-clang-wrapper-5.0.1/bin/clang++

Answer 19 · 2018-01-31T17:37:23.000Z

Hey @mikeandmore, I guess you're using nix on linux or nixos, so $CXX is set to g++? If you want the configure phase to pick up clang instead of gcc, I think this is where you'd use the --check-cxx-compiler=clang++ flag mentioned earlier. I don't know if that will fix everything for you as I've not used it. You can also try my nix derivation that is updated fairly often and includes a bit of patching to let configure find more of the things it's looking for, but I've only used it with nix on darwin, so my $CXX is already clang++. If that proves to be an issue, then I'd like to fix it in the derivation.

Answer 20 · 2018-01-31T17:55:01.000Z

Yes, I think that's the issue. After I unset CXX everything works well! Thanks

Answer 21 · 2018-01-31T17:58:50.000Z

@mikeandmore Can you please add a section for Nix in https://github.com/cquery-project/cquery/wiki/Build

Answer 22 · 2018-01-31T18:32:03.000Z

@MaskRay Done.