BVLC/caffe

Optimize build process on debian jessie

cdluminate opened this issue ยท 23 comments

cdluminate commented

I'd suggest that adding this script for debian users.

#!/bin/bash
# manipulate header path, before building caffe on debian jessie
# usage:
#1. cd root of caffe
#2. bash <this_script>
#3. build

# transformations :
#  #include "hdf5/serial/hdf5.h" -> #include "hdf5/serial/hdf5.h"
#  #include "hdf5_hl.h" -> #include "hdf5/serial/hdf5_hl.h"

find . -type f -exec sed -i -e 's^"hdf5.h"^"hdf5/serial/hdf5.h"^g' -e 's^"hdf5_hl.h"^"hdf5/serial/hdf5_hl.h"^g' '{}' \;
shelhamer commented

Thanks for posting your one-liner but this seems like it would be better fixed by the right include dirs instead of transforming the source.

  • ๐Ÿ‘5
  • ๐Ÿ‘Ž3
inexxt commented

Had the same problem on Ubuntu 15.04, solution worked, thanks.

oseiskar commented

To get the build to pass on Debian Jessie, I had to (in addition to the above)

  1. modify INCLUDE_DIRS in Makefile.config

    INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial/

  2. create symlinks as instructed here

    cd /usr/lib/x86_64-linux-gnu
sudo ln -s libhdf5_serial.so.8.0.2 libhdf5.so
sudo ln -s libhdf5_serial_hl.so.8.0.2 libhdf5_hl.so

  3. Work around #337 by installing g++-4.6 from the Wheezy repo and adding to Makefile.config:

    CUSTOM_CXX := g++-4.6
  • ๐Ÿ‘59
  • ๐ŸŽ‰5
  • โค๏ธ14
snj33v commented

3.Work around #337 by installing g++-4.6 from the Wheezy repo and adding to Makefile.config:
another way is seperating host compiler using flags(-ccbin) in nvcc
line:341
from
NVCCFLAGS += -ccbin=$(CXX) -Xcompiler -fPIC $(COMMON_FLAGS)
to
NVCCFLAGS += -ccbin=g++-4.6 -Xcompiler -fPIC $(COMMON_FLAGS)

OverStruck commented

Thanks for this. It worked in Debian Jessie

chenyang-charles commented

Works on Ubuntu 15.04

  • ๐Ÿ‘1
acgtyrant commented

@shelhamer But fixing the right include dirs does not solve another problem:

/usr/bin/ld: cannot find -lhdf5_hl
/usr/bin/ld: cannot find -lhdf5

We have to create two symbolic links manually.

See can't find hdf5.h when build caffe.

  • ๐Ÿ‘2
dgeo commented

Here: instead of using gcc-4.6, using nvidia-cuda-toolkit-6.5.14 from stretch seems to work also to workaround #337

javadba commented

Is there any more complete summary on how to do this build? After attempting to merge various bits and pieces from this as well as StackOverflow questions, I end up with the following all the same .

LD -o .build_debug/lib/libcaffe.so.1.0.0-rc3
/usr/bin/ld: cannot find -lhdf5_hl
/usr/bin/ld: cannot find -lhdf5
/usr/bin/ld: cannot find -lcudnn
/usr/bin/ld: cannot find -lcblas
/usr/bin/ld: cannot find -latlas
collect2: error: ld returned 1 exit status
Makefile:554: recipe for target '.build_debug/lib/libcaffe.so.1.0.0-rc3' failed
make: *** [.build_debug/lib/libcaffe.so.1.0.0-rc3] Error 1

I had used apt-get install on the relevant libraries, updated paths and the Makefile.config . Quite a number of steps. There does not seem to be a firm consensus on the correct aproach so there was a certain degree of mix and match.

I am on ubuntu 15.10 with nVidia cuda .

snj33v commented

@javadba, + where ever you have installed cudnn headers and library file

INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include /usr/include/hdf5/serial/ /usr/local/cuda/include
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu

as for gcc workaroud try to use clang for everthing

Update: Dont follow this anymore.

PaulBGD commented

This worked for me, thanks.

binary1too commented

Don't mess with symlinks! Just edit Makefile:

--- LIBRARIES += glog gflags protobuf boost_system boost_filesystem m hdf5_hl hdf5
+++ LIBRARIES += glog gflags protobuf boost_system boost_filesystem m hdf5_serial_hl hdf5_serial
  • ๐Ÿ‘20
  • ๐Ÿ˜„8
  • ๐ŸŽ‰1
Key-Liu commented

This worked for me on Ubuntu 16.04, thanks.

  • ๐Ÿ‘4
raditha commented

Worked for me on linux mint.Thank you.

TTy32 commented

Works on Debian 8.7. Thanks!

tim-win commented

Can we reopen this? Still failing on Ubuntu 16 without manual workarounds.

And update the title- this is not a request for 'optimization', currently the build process as provided is broken on debian based OSes.

  • ๐Ÿ‘4
iamsaurabhc commented

Update : Works on Ubuntu 16.04 too! Thanks

jeremy-rutman commented

step #1 from @oseiskar works for linux ubuntu 16.04 (actually mint based on that).

skondrashov commented

your comment appears to be wrong:
# #include "hdf5/serial/hdf5.h" -> #include "hdf5/serial/hdf5.h"

but the solution worked! thank you!

StrugglingXu commented

Hello,when i Build all the targets!
cd gmake
make config=release64
cd ../
error:
==== Building terrainrlUtil (release64) ====
Linking terrainrlUtil
/usr/bin/ld: ../external/jsoncpp/build/debug/src/lib_json/libjsoncpp.a(json_value.cpp.o): relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC
../external/jsoncpp/build/debug/src/lib_json/libjsoncpp.a: ๆ— ๆณ•ๆทปๅŠ ็ฌฆๅท: ้”™่ฏฏ็š„ๅ€ผ
collect2: error: ld returned 1 exit status
terrainrlUtil.make:139: recipe for target '../lib/libterrainrlUtil.so' failed
make[1]: *** [../lib/libterrainrlUtil.so] Error 1
Makefile:20: recipe for target 'terrainrlUtil' failed
make: *** [terrainrlUtil] Error 2
please help me to solve this question.Thanks

g4nym3de commented

Works on Ubuntu 16.04, thanks!

Time0o commented

These workarounds don't seem to fix the CMake build. Has anyone gotten that to work (in particular on Debian)? I'm stuck because CMake is unable to locate HDF5 even after creating symlinks for header files and shared objects in the "right" locations. I'm starting to think the whole build process has been broken for years on Debian.

WatsonZhouAnda commented

@StrugglingXu
Hi!
I met the same problem as you when I try to "make config=release64" in /gmake file. Have you solved it yet?