`libvex_guest_offsets.h` not found when pip install -e .\pyvex

[yifanwww]

Environment: Windows 10 1909, Python 3.7.7, Visual Studio 2019 (MSVC 14.25.28610).

To repeat this problem, you need a clean vex local repository with no compiled files existing. Maybe you need to delete the existing vex and reclone one.

Then execute pip install -e .\pyvex. The following errors will be outputed:

(.test) D:\Projects\Git\YF\angr\angr-dev>pip install -e .\pyvex
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple/, http://mirrors.aliyun.com/pypi/simple/
Obtaining file:///D:/Projects/Git/YF/angr/angr-dev/pyvex
Requirement already satisfied: pycparser in d:\projects\git\yf\angr\.test\lib\site-packages (from pyvex==8.20.1.7) (2.20)
Requirement already satisfied: cffi>=1.0.3 in d:\projects\git\yf\angr\.test\lib\site-packages (from pyvex==8.20.1.7) (1.14.0)
Requirement already satisfied: archinfo==8.20.1.7 in d:\projects\git\yf\angr\angr-dev\archinfo (from pyvex==8.20.1.7) (8.20.1.7)
Processing c:\users\yf\appdata\local\pip\cache\wheels\2c\b6\ef\b7dfcb733e707e3fbe2004bc082ab81423d885bf4ca10f082b\bitstring-3.1.7-py3-none-any.whl
Processing c:\users\yf\appdata\local\pip\cache\wheels\bc\6f\cd\f72c82ed44590b04c6deb82b177f0d260aec8621f6568e0368\future-0.18.2-py3-none-any.whl
Installing collected packages: bitstring, future, pyvex
  Running setup.py develop for pyvex
    ERROR: Command errored out with exit status 1:
     command: 'D:\Projects\Git\YF\angr\.test\Scripts\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'D:\\Projects\\Git\\YF\\angr\\angr-dev\\pyvex\\setup.py'"'"'; __file__='"'"'D:\\Projects\\Git\\YF\\angr\\angr-dev\\pyvex\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
         cwd: D:\Projects\Git\YF\angr\angr-dev\pyvex\
    Complete output (29 lines):
    running develop
    Building libVEX

    Microsoft (R) Program Maintenance Utility Version 14.25.28614.0
    Copyright (C) Microsoft Corporation.  All rights reserved.

        cl /Fo:priv/ir_defs.o -c priv/ir_defs.c /Ipub /Ipriv  /O2 /wd4715 /DPYVEX
    Microsoft (R) C/C++ Optimizing Compiler Version 19.25.28614 for x86
    Copyright (C) Microsoft Corporation.  All rights reserved.

    ir_defs.c
        cl /Fo:priv/ir_match.o -c priv/ir_match.c /Ipub /Ipriv  /O2 /wd4715 /DPYVEX
    Microsoft (R) C/C++ Optimizing Compiler Version 19.25.28614 for x86
    Copyright (C) Microsoft Corporation.  All rights reserved.

    ir_match.c
        cl /Fo:priv/ir_opt.o -c priv/ir_opt.c /Ipub /Ipriv  /O2 /wd4715 /DPYVEX
    Microsoft (R) C/C++ Optimizing Compiler Version 19.25.28614 for x86
    Copyright (C) Microsoft Corporation.  All rights reserved.

    ir_opt.c
    priv/ir_opt.c(40): fatal error C1083: Cannot open include file: 'libvex_guest_offsets.h': No such file or directory
    NMAKE : fatal error U1077: '"C:\Programs\Visual Studio 2019\Visual Studio 2019\VC\Tools\MSVC\14.25.28610\bin\HostX86\x86\cl.EXE"' : return code '0x2'
    Stop.
    cc -Ipub -Ipriv -Wall -Wmissing-prototypes -Wstrict-prototypes -Wshadow -Wpointer-arith -Wbad-function-cast -Wcast-qual -Wcast-align -Wmissing-declarations -Wwrite-strings -Wformat -Wformat-security -std=gnu99 -fstrict-aliasing -fPIC -DPYVEX  -g -o auxprogs/genoffsets auxprogs/genoffsets.c
    process_begin: CreateProcess(NULL, cc -Ipub -Ipriv -Wall -Wmissing-prototypes -Wstrict-prototypes -Wshadow -Wpointer-arith -Wbad-function-cast -Wcast-qual -Wcast-align -Wmissing-declarations -Wwrite-strings -Wformat -Wformat-security -std=gnu99 -fstrict-aliasing -fPIC -DPYVEX -g -o auxprogs/genoffsets auxprogs/genoffsets.c, ...) failed.
    make (e=2): The system cannot find the file specified.
    make: *** [Makefile-gcc:74: pub/libvex_guest_offsets.h] Error 2
    error: Unable to build libVEX.
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'D:\Projects\Git\YF\angr\.test\Scripts\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'D:\\Projects\\Git\\YF\\angr\\angr-dev\\pyvex\\setup.py'"'"'; __file__='"'"'D:\\Projects\\Git\\YF\\angr\\angr-dev\\pyvex\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.

This problem doesn't happen if you are not on the last commit in vex, Opt: Support disabling cross-instruction optimizations. In this commit, libvex_guest_offsets.h was included in ir_opt.c, but this file was not generated when compiling vex.

I find a way to solve this problem. You need to execute nmake /f Makefile-msvc pub\libvex_guest_offsets.h in vex to get libvex_guest_offsets.h first. Then you can install pyvex successfully. Once you succeed, you can always reinstall pyvex successfully whether libvex_guest_offsets.h exists or not, which can be generated somehow.

But it should be executed automatically, not manually. Please fix this problem.

[ltfish]

Should be fixed. Thanks for reporting!