peterjc123/pytorch-scripts

MSBuild can't find the project file for ATEN

franchenstein opened this issue · 13 comments

I am trying to install the latest version of PyTorch for Windows 10 by using cpu.bat, as my computer doesn't have a CUDA-compatible GPU. While running the script using VS prompt, I get the following error when it tries to build ATEN:

-- Build files have been written to: C:/.../pytorch-scripts/pytorch/aten/build

C:\...\pytorch-scripts\pytorch\aten\build>msbuild INSTALL.vcxproj /p:Configuration=Release
Microsoft (R) Build Engine version 15.6.85.37198 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

MSBUILD : error MSB1009: Project file does not exist.
Switch: INSTALL.vcxproj

C:\...\pytorch-scripts\pytorch\aten\build>IF NOT 1 == 0 exit 1

but I can't find this file anywhere in the directory. Where is it supposed to be?

This is not the actual error. It must be occurred before. Could you find it?

You're right. I'm sorry, I didn't notice the error before that. I had the following message:

-- Configuring done
CMake Error at src/ATen/cpu/tbb/CMakeLists.txt:239 (add_library):
  Cannot find source file:

    C:/.../pytorch-scripts/pytorch/aten/src/ATen/cpu/tbb/tbb_remote/src/tbb/ia32-masm/itsx.asm src/tbb/ia32-masm/lock_byte.asm

  Tried extensions .c .C .c++ .cc .cpp .cxx .cu .m .M .mm .h .hh .h++ .hm
  .hpp .hxx .in .txx


CMake Error at src/ATen/cpu/tbb/CMakeLists.txt:239 (add_library):
  No SOURCES given to target: tbb_static

Shouldn't the latest versions of CMake 'compile' assembly out of the box?

Well, I see. It is a mis-typing in the CMake scripts. Since we use Ninja in CI, it seems to be ignored. Would you please change the code in pytorch/aten/src/ATen/cpu/tbb/CMakeLists.txt:L144-145 and see whether it works.

      list(APPEND tbb_src "${TBB_ROOT_DIR}/src/tbb/ia32-masm/atomic_support.asm"
        "${TBB_ROOT_DIR}/src/tbb/ia32-masm/itsx.asm src/tbb/ia32-masm/lock_byte.asm")
# -> to the following
      list(APPEND tbb_src "${TBB_ROOT_DIR}/src/tbb/ia32-masm/atomic_support.asm"
        "${TBB_ROOT_DIR}/src/tbb/ia32-masm/itsx.asm"
        "${TBB_ROOT_DIR}/src/tbb/ia32-masm/lock_byte.asm")

@franchenstein But it should not happen. I have tested the MSVC build using CI only a few time ago. What version of CI are you using? And can you located the assembler found using CMAKE?

 -- Using python found in C:\Jenkins\Miniconda3\python.exe
 -- The ASM_MASM compiler identification is MSVC
 -- Found assembler: C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/VC/Tools/MSVC/14.11.25503/bin/Hostx86/x64/ml64.exe   <- this one
 -- Performing Test SUPPORTS_VOLATILE_FLAG

I made the proposed change and the previous error did not occur anymore. On the other hand, after more than 40 minutes, the compilation failed with the following message:

c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(129): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(130): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(131): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(132): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(134): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(135): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(136): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(137): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
  ReduceOpsKernel.cpp.AVX2.cpp
c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(129): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(130): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(131): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(132): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(134): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(135): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(136): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(137): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
  Generating Code...
Done Building Project "C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj" (default ta
rgets) -- FAILED.

Done Building Project "C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\ALL_BUILD.vcxproj" (default target
s) -- FAILED.

Done Building Project "C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\INSTALL.vcxproj" (default targets)
 -- FAILED.

I used Cl version 19.11.25548.2 for x64. I couldn't find the assembler being used. I had the following message:

-- Using python found in C:\Users\dfranch\AppData\Local\Continuum\anaconda3\python.exe
-- Configuring done
-- Generating done
-- Build files have been written to: C:/Users/dfranch/Downloads/pytorch-scripts/pytorch/aten/build

How could _mm256_extract_epi64 be not found in MSVC 2017? It should the issue only for 2015. Can you give me your full build log?

build_log.txt

The log is quite long because of lots repeated messages. I am attaching it to this message.

Sorry, a flag is missing there. I just solved it. Could you please try again?

I added the flag but I still get the same 16 errors.

Would you please clean the build first? And make sure you call it in a simple CMD window, not a dev shell.

DId that and now everything works great! Thank you very much for the support!

Which flag is missing? I get the same error compiling v0.4.1 from source.

I found a commit on April 23, 2018 which had this added:
set CMAKE_GENERATOR=Visual Studio 15 2017 Win64

I ran that command and the errors have gone away. I don't know which file in pytorch should be updated. I had to compile the pytorch code from source, because my gpu is a GTX 960. I'm not sure if cuda 8 from these scripts is too advanced for my gpu.