pyg-team/pyg-lib

Conda forge package

hadim opened this issue · 7 comments

hadim commented

😵 Describe the installation problem

I am building a conda package for pyg-lib at conda-forge/staged-recipes#21566

For now, it's only for Linux cpu and cuda but I guess OSX should build as well.

Are you willing to be added as a maintainer of the package as well?

Currently the conda recipe relies on git submodule for parallel-hashmap and cutlass. But those two deps are available on conda-forge and so it's preferred for the conda package to rely on those. But it requires modifying the CMake file for this to work.

Environment

  • pyg-lib version:
  • PyTorch version:
  • OS:
  • Python version:
  • CUDA/cuDNN version:
  • How you installed PyTorch and pyg-lib (conda, pip, source):
  • Any other relevant information:
rusty1s commented

Hey @hadim, thanks for reaching out and for your efforts. Appreciate it. I am happy to accept a PR that changes the CMake. Please feel free to add me as a maintainer, but I am not totally sure I have the expertise in conda to contribute :)

hadim commented

Sounds good.

Is pyg-lib a mandatory dep now for pyg or just optional? Also, does it replace the other libs sparse and scatter?

rusty1s commented

It is currently optional, and will probably be made mandatory in PyG 2.3 (once I figured out how to enable Windows installation). torch-sparse and torch-scatter will be deprecated soon.

  • 👍1
hadim commented

This is what the patch looks like at the moment for the recipe to work:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 3e88c65..fb34cd5 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -21,10 +21,15 @@ if(WITH_CUDA)
   add_definitions(-DWITH_CUDA)
   set(CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} --expt-relaxed-constexpr")
 
-  set(CUTLASS_DIR third_party/cutlass/include)
-  include_directories(${CUTLASS_DIR})
-  set(CUTLASS_UTIL_DIR third_party/cutlass/tools/util/include)
-  include_directories(${CUTLASS_UTIL_DIR})
+  if (NOT "$ENV{EXTERNAL_CUTLASS_INCLUDE_DIR}" STREQUAL "")
+    include_directories($ENV{EXTERNAL_CUTLASS_INCLUDE_DIR})
+  else()
+    set(CUTLASS_DIR third_party/cutlass/include)
+    include_directories(${CUTLASS_DIR})
+    set(CUTLASS_UTIL_DIR third_party/cutlass/tools/util/include)
+    include_directories(${CUTLASS_UTIL_DIR})
+  endif()
+
 endif()
 
 set(CSRC pyg_lib/csrc)
@@ -36,8 +41,12 @@ file(GLOB_RECURSE ALL_HEADERS ${CSRC}/*.h)
 add_library(${PROJECT_NAME} SHARED ${ALL_SOURCES})
 target_include_directories(${PROJECT_NAME} PUBLIC "${CMAKE_CURRENT_SOURCE_DIR}")
 
-set(PHMAP_DIR third_party/parallel-hashmap)
-target_include_directories(${PROJECT_NAME} PRIVATE ${PHMAP_DIR})
+if (NOT "$ENV{EXTERNAL_PHMAP_INCLUDE_DIR}" STREQUAL "")
+  include_directories($ENV{EXTERNAL_PHMAP_INCLUDE_DIR})
+else()
+  set(PHMAP_DIR third_party/parallel-hashmap)
+  target_include_directories(${PROJECT_NAME} PRIVATE ${PHMAP_DIR})
+endif()
 
 find_package(Torch REQUIRED)
 target_link_libraries(${PROJECT_NAME} PRIVATE ${TORCH_LIBRARIES})
diff --git a/setup.py b/setup.py
index d1db38a..77bfd23 100644
--- a/setup.py
+++ b/setup.py
@@ -56,7 +56,7 @@ class CMakeBuild(build_ext):
             f'-DCMAKE_PREFIX_PATH={torch.utils.cmake_prefix_path}',
         ]
 
-        if importlib.util.find_spec('ninja') is not None:
+        if importlib.util.find_spec('ninja') is not None or os.environ.get("FORCE_NINJA") is not None:
             cmake_args += ['-GNinja']
         else:
             warnings.warn("Building times of 'pyg-lib' can be heavily improved"

I will just apply it during the package creation so we don't have to wait for a new pyg-lib release and I will remove it once it has been integrated into pyg-lib.

Note that the patch is against the 0.1.0 branch and not the master one.

This patch does not change the default build configuration of pyg-lib, all the new flags are optional.

Current patch file: https://github.com/hadim/staged-recipes/blob/e9c36acadc91e11ed20af0c6b611bd3c7e84c073/recipes/pyg-lib/cmake_compat.patch

rusty1s commented

Looks great, please feel free to send a PR :)

hadim commented

The first builds are now available on conda at https://github.com/conda-forge/pyg-lib-feedstock.

It's only for pytorch 1.12 currently, but I am rebuilding to include 1.11 and 1.13 at conda-forge/pyg-lib-feedstock#3

The above patch is quite straightforward to apply so I prefer to let you doing it.

Hope it helps! Closing here but feel free to reopen for any questions (or open a ticket in the feedstock directly).

rusty1s commented

Thanks, I patched the CMakeLists in #168.

  • 👍1