PCH breaking between calls, rebuild_cmdstan() errors out
bob-carpenter opened this issue · 9 comments
Summary:
Update: It turns out what happened is that in the first report, the first run where it appeared to work, failed in the parser, so it never tried to run the compiler. I had updated Xcode before running everything below, but the problem didn't appear until I finally got around to compiling. So I did pull the rug out from under CmdStan.
This was not at all obvious to me and I was essentially blocked because the usual ways to fix didn't work. I had to reinstall cmdstan (which I did through CmdStanPy). Then everything worked. So I'm just going to close this issue.
Description:
I'm going to give you two session reports. The first one is where the problem crops up. The second is where I fail to fix it.
~/github/bob-carpenter/case-studies/ngrams (master)$ python3 tokenizer.py
f{np.shape(tokens) = }
Traceback (most recent call last):
File "/Users/bcarpenter/github/bob-carpenter/case-studies/ngrams/tokenizer.py", line 53, in <module>
model = csp.CmdStanModel(stan_file = 'bigram.stan')
File "/usr/local/lib/python3.9/site-packages/cmdstanpy/model.py", line 191, in __init__
model_info = self.src_info()
File "/usr/local/lib/python3.9/site-packages/cmdstanpy/model.py", line 305, in src_info
raise ValueError(
ValueError: Failed to get source info for Stan model '/Users/bcarpenter/github/bob-carpenter/case-studies/ngrams/bigram.stan'. Console:
Semantic error in '/Users/bcarpenter/github/bob-carpenter/case-studies/ngrams/bigram.stan', line 22, column 23 to column 35:
-------------------------------------------------
20: y[1] ~ categorical(theta_uni);
21: for (n in 2:N)
22: y[n] ~ categorical(theta_bigram[y[n - 1]]);
^
23: }
-------------------------------------------------
Identifier 'theta_bigram' not in scope.
When I try again I get the dreaded PCH error.
~/github/bob-carpenter/case-studies/ngrams (master)$ python3 tokenizer.py
f{np.shape(tokens) = }
13:46:46 - cmdstanpy - INFO - compiling stan file /Users/bcarpenter/github/bob-carpenter/case-studies/ngrams/bigram.stan to exe file /Users/bcarpenter/github/bob-carpenter/case-studies/ngrams/bigram
13:47:05 - cmdstanpy - WARNING - CmdStan's precompiled header (PCH) files may need to be rebuilt.Please run cmdstanpy.rebuild_cmdstan().
If the issue persists please open a bug report
Traceback (most recent call last):
File "/Users/bcarpenter/github/bob-carpenter/case-studies/ngrams/tokenizer.py", line 53, in <module>
model = csp.CmdStanModel(stan_file = 'bigram.stan')
File "/usr/local/lib/python3.9/site-packages/cmdstanpy/model.py", line 236, in __init__
self.compile(force=str(compile).lower() == 'force')
File "/usr/local/lib/python3.9/site-packages/cmdstanpy/model.py", line 563, in compile
raise ValueError(
ValueError: Failed to compile Stan model '/Users/bcarpenter/github/bob-carpenter/case-studies/ngrams/bigram.stan'. Console:
--- Translating Stan model to C++ code ---
bin/stanc --o=/Users/bcarpenter/github/bob-carpenter/case-studies/ngrams/bigram.hpp /Users/bcarpenter/github/bob-carpenter/case-studies/ngrams/bigram.stan
--- Compiling, linking C++ code ---
clang++ -std=c++1y -Wno-unknown-warning-option -Wno-tautological-compare -Wno-sign-compare -D_REENTRANT -Wno-ignored-attributes -I stan/lib/stan_math/lib/tbb_2020.3/include -O3 -I src -I stan/src -I stan/lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.78.0 -I stan/lib/stan_math/lib/sundials_6.1.1/include -I stan/lib/stan_math/lib/sundials_6.1.1/src/sundials -DBOOST_DISABLE_ASSERTS -c -include-pch stan/src/stan/model/model_header.hpp.gch -x c++ -o /Users/bcarpenter/github/bob-carpenter/case-studies/ngrams/bigram.o /Users/bcarpenter/github/bob-carpenter/case-studies/ngrams/bigram.hpp
error: PCH file uses an older PCH format that is no longer supported
1 error generated.
make: *** [/Users/bcarpenter/github/bob-carpenter/case-studies/ngrams/bigram] Error 1
Command ['make', '/Users/bcarpenter/github/bob-carpenter/case-studies/ngrams/bigram']
error during processing No such file or directory
~/github/bob-carpenter/case-studies/ngrams (master)$
This is my call to rebuild_cmdstan() failing.
>>> import cmdstanpy as csp
>>> csp.rebuild_cmdstan()
Compiling: (00:58) | | ... Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cmdstanpy/install_cmdstan.py", line 280, in build
do_command(cmd, fd_out=None, pbar=progress_hook)
File "/usr/local/lib/python3.9/site-packages/cmdstanpy/utils/command.py", line 76, in do_command
raise RuntimeError(msg)
RuntimeError: Command ['make', 'build', '-j1']
error during processing No such file or directory
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.9/site-packages/cmdstanpy/install_cmdstan.py", line 386, in rebuild_cmdstan
build(verbose, progress, cores)
File "/usr/local/lib/python3.9/site-packages/cmdstanpy/install_cmdstan.py", line 286, in build
raise CmdStanInstallError(f'Command "make build" failed\n{str(e)}')
cmdstanpy.install_cmdstan.CmdStanInstallError: Command "make build" failed
Command ['make', 'build', '-j1']
error during processing No such file or directory
Compiling: (01:10) | | ...
>>> csp.rebuild_cmdstan()
Compiling: (08:07) | | ...
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cmdstanpy/install_cmdstan.py", line 280, in build
do_command(cmd, fd_out=None, pbar=progress_hook)
File "/usr/local/lib/python3.9/site-packages/cmdstanpy/utils/command.py", line 76, in do_command
raise RuntimeError(msg)
RuntimeError: Command ['make', 'build', '-j1']
error during processing No such file or directory
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.9/site-packages/cmdstanpy/install_cmdstan.py", line 386, in rebuild_cmdstan
build(verbose, progress, cores)
File "/usr/local/lib/python3.9/site-packages/cmdstanpy/install_cmdstan.py", line 286, in build
raise CmdStanInstallError(f'Command "make build" failed\n{str(e)}')
cmdstanpy.install_cmdstan.CmdStanInstallError: Command "make build" failed
Command ['make', 'build', '-j1']
error during processing No such file or directory
Compiling: (01:06) | | ...
Compiling: (03:51) | | ...
KeyboardInterrupt
>>>
Current Version:
>>> cmdstanpy.show_versions()
INSTALLED VERSIONS
---------------------
python: 3.9.4 (default, Apr 5 2021, 01:47:16)
[Clang 11.0.0 (clang-1100.0.33.17)]
python-bits: 64
OS: Darwin
OS-release: 22.6.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
cmdstan_folder: /Users/bcarpenter/.cmdstan/cmdstan-2.31.0
cmdstan: (2, 31)
cmdstanpy: 1.1.0
pandas: 2.0.3
xarray: 2023.1.0
tqdm: 4.64.0
numpy: 1.25.0
' '
we've discussed getting rid of pch headers altogether. time to revisit this issue?
Disabling the precompiled headers roughly doubles the amount of time required to compile a model (see roualdes/bridgestan#151). This problem also seems to be essentially exclusive to MacOS, I've never encountered it on Linux
We tried to fix issues like this in stan-dev/cmdstan#1171 which was included in the 2.33 release
This problem also seems to be essentially exclusive to MacOS, I've never encountered it on Linux
I thought it was only implemented for MacOS?
update: defined in CmdStan makefile here: https://github.com/stan-dev/cmdstan/blob/e99e0f2ac9a920624c36899817856183a13d6e3e/makefile#L121-L133
We use it on all three major platforms since 2.28
@bob-carpenter can you share the output of csp.rebuild_cmdstan(verbose=True)?
Disabling the precompiled headers roughly doubles the amount of time required to compile a model
Thanks for linking the issue!
I'm always going to favor things that are more robust to configuration error.
Errors like this make the 2x time per compile worth it for me. Whenever I run into it and I have to spend half an hour rebuilding cmdstan to get everything back into a consistent state, it's tough. It's a hard context switch from where I'm normally at. I don't know how someone that doesn't know the infrastructure as well navigates past this problem.
Btw, I'd be happy if there were clear instructions that appear when this error occurs. I just don't think we have that yet.
Rebuilding cmdstan usually works (I'm curious what exactly is going wrong for Bob during that).
We used to try to show instructions to do so, but the message the compiler shows when the PCH is bad seems to change often and we ended up writing such a broad net that we were showing that suggestion when the PCH wasn't even the real issue.
Adding PRECOMPILED_HEADERS=false into a make/local file will make it never be used on your individual system, which might be a good idea if you're in a situation where this is happening frequently
@bob-carpenter can you share the output of csp.rebuild_cmdstan(verbose=True)?
Afraid not. I upgraded CmdStan, updated CmdStanPy, and rebooted. The problem turned out to be because I really had updated in between. I'll update the issue text to reflect that.
I wouldn't want to remove precompiled headers altogether. I'll let you know if I get this problem going forward. If I keep running into the problem, I'll lobby for turning it off by default.
Closing because we expect PCH to break and require manual fixes when we update the underlying C++ compiler, at least for the version I was using.