Can you give me some advises on makefile?

Question

Can you give me some advises on makefile?

ZhenhuiTang opened this issue 4 years ago · 18 comments

I encounter some difficulties during make makefile, as follow:

make: Circular tf_nndistance.cu <- tf_nndistance.cu.o dependency dropped.
g++ tf_nndistance.cpp tf_nndistance.cu.o -o tf_nndistance_so.so \
-I /usr/local/cuda-10.0/include/ -I /home/machinelearning/.local/lib/python3.6/site-packages/tensorflow_core/include -L /usr/local/cuda-10.0/lib64/ -lcudart -L /home/machinelearning/.local/lib/python3.6/site-packages/tensorflow_core -ltensorflow_framework \
    -shared -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11 -fPIC -O2
/usr/bin/ld: cannot find -ltensorflow_framework
collect2: error: ld returned 1 exit status
makefile:13: recipe for target 'tf_nndistance_so.so' failed
make: *** [tf_nndistance_so.so] Error 1

The environment is tf.version == 1.15 and cuda == 10.0, and the modified content as follow:

cuda_inc = /usr/local/cuda-10.0/include/
cuda_lib = /usr/local/cuda-10.0/lib64/
nvcc = /usr/local/cuda-10.0/bin/nvcc
tf_inc = /home/machinelearning/.local/lib/python3.6/site-packages/tensorflow_core/include
tf_lib =/home/machinelearning/.local/lib/python3.6/site-packages/tensorflow_core

Thanks a lot.

Answer 1 · 2020-07-02T03:15:47.000Z

Hi. The same problem appears. Have you got the solution?

Answer 2 · 2020-07-02T03:33:50.000Z

Hi. The same problem appears. Have you got the solution?

This can be solved by:
cd /home/machinelearning/.local/lib/python3.6/site-packages/tensorflow_core
ln -s libtensorflow_framework.so.1 libtensorflow_framework.so

Answer 3 · 2020-07-02T05:49:24.000Z

@TerenceCYJ

Thank you.

I've executed same commands, but problem is not solved.
Can you please once look into the same.

Answer 4 · 2020-07-02T09:19:02.000Z

@TerenceCYJ

Thank you.

I've executed same commands, but problem is not solved.
Can you please once look into the same.

Hi, @GopiRajuMatta. I just fix the bug by what I wrote. And my environment is tf.version == 1.15 and cuda == 10.0.

Answer 5 · 2020-07-02T09:47:40.000Z

Hi, @TerenceCYJ

Thank you Terence, It was built without any error. But when I try to run demo.py, it is throwing this error.

"
Traceback (most recent call last):
File "demo.py", line 32, in
model_module = importlib.import_module('.%s' % args.model_type, 'models')
File "/home/gopi/anaconda3/envs/pcn_test2/lib/python3.6/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 994, in _gcd_import
File "", line 971, in _find_and_load
File "", line 955, in _find_and_load_unlocked
File "", line 665, in _load_unlocked
File "", line 678, in exec_module
File "", line 219, in _call_with_frames_removed
File "/home/gopi/git/pcn/models/pcn_cd.py", line 4, in
from tf_util import *
File "/home/gopi/git/pcn/tf_util.py", line 4, in
from pc_distance import tf_nndistance, tf_approxmatch
File "/home/gopi/git/pcn/pc_distance/tf_nndistance.py", line 5, in
nn_distance_module=tf.load_op_library(os.path.join(BASE_DIR, 'tf_nndistance_so.so'))
File "/home/gopi/anaconda3/envs/pcn_test2/lib/python3.6/site-packages/tensorflow_core/python/framework/load_library.py", line 61, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: /home/gopi/git/pcn/pc_distance/tf_nndistance_so.so: undefined symbol: _ZN10tensorflow12OpDefBuilder5InputESs
"

Any idea, where it went wrong?

Thank you
Gopi

Answer 6 · 2020-07-02T10:59:28.000Z

Hi, Gopi. I think you should check your pc_distance.
After successful make, there should be a tf_nndistance_so.so
In my setting, after make, ls pc_distance
Get：__init__.py __pycache__ tf_approxmatch.cu tf_approxmatch.py tf_nndistance.cpp tf_nndistance.cu.o tf_nndistance_so.so makefile tf_approxmatch.cpp tf_approxmatch.cu.o tf_approxmatch_so.so tf_nndistance.cu tf_nndistance.py

Answer 7 · 2020-07-02T12:14:44.000Z

Thank you Terence.

These are the files after make, it has all the files as you reported.
"init_.py tf_approxmatch.cpp tf_approxmatch.py tf_nndistance.cu tf_nndistance_so.so makefile tf_approxmatch.cu tf_approxmatch_so.so tf_nndistance.cu.o pycache tf_approxmatch.cu.o tf_nndistance.cpp tf_nndistance.py"

However, I am getting the reported error.

Thank you
Gopi

Answer 8 · 2020-07-02T12:30:58.000Z

Hi Terence,

It seems, problem is with file tf_nndistance_so.so. It was compiled without errors. But I am not sure, why I am getting that particular error. I have used cuda=9.0, tf.version == 1.15. Is there any problem with this? If you don't mind, can you please share with me your compiled pc_distance folder.

Thank you
Gopi

Answer 9 · 2020-07-04T13:12:01.000Z

I encountered the same issues. After several days of bashing my head against the desk, I got the code up and running using the following steps:

Create a conda environment with cuda=9.2, tf=1.12, open3d=0.8.0.0 and python=3.6:

conda create -n pcn tensorflow=1.12 python=3.6
conda activate pcn
conda install -c conda-forge nvcc_linux-64=9.2
conda install -c nvidia cudatoolkit=9.2
conda install -c open3d-admin open3d=0.8

Modify makefile as follows (replace user with your username):

cuda_inc = /home/user/.conda/envs/pcn/include/
cuda_lib = /home/user/.conda/envs/pcn/lib/
nvcc     = /home/user/.conda/envs/pcn/bin/nvcc
tf_inc   = /home/user/.conda/envs/pcn/lib/python3.6/site-packages/tensorflow/include
tf_lib   = /home/user/.conda/envs/pcn/lib/python3.6/site-packages/tensorflow

all: tf_nndistance_so.so tf_approxmatch_so.so

tf_nndistance.cu.o: tf_nndistance.cu
	$(nvcc) tf_nndistance.cu -o tf_nndistance.cu.o -c -O2 -DGOOGLE_CUDA=1 -x cu -Xcompiler -fPIC

tf_nndistance_so.so: tf_nndistance.cpp tf_nndistance.cu.o
	g++ tf_nndistance.cpp tf_nndistance.cu.o -o tf_nndistance_so.so -shared -std=c++11 -fPIC -O2 \
            -I $(tf_inc) -D_GLIBCXX_USE_CXX11_ABI=1 -L $(tf_lib) -ltensorflow_framework -L $(cuda_lib) -lcudart

tf_approxmatch.cu.o: tf_approxmatch.cu
	$(nvcc) tf_approxmatch.cu -o tf_approxmatch.cu.o -c -O2 -DGOOGLE_CUDA=1 -x cu -Xcompiler -fPIC

tf_approxmatch_so.so: tf_approxmatch.cpp tf_approxmatch.cu.o
	g++ -shared $(CPPFLAGS) tf_approxmatch.cpp tf_approxmatch.cu.o -o tf_approxmatch_so.so -shared -std=c++11 -fPIC -O2 \
            -I $(tf_inc) -D_GLIBCXX_USE_CXX11_ABI=1 -L $(tf_lib) -ltensorflow_framework -L $(cuda_lib) -lcudart

clean:
	rm -rf *.o *.so

Run make clean && make inside the conda environment.

Hope this helps!

Answer 10 · 2020-07-04T18:07:19.000Z

@mihaimorariu, I really appreciate your help.
I am getting new errors. I will work around. I will try to resolve those.
I will update you.

Thank you so much
Gopi

Answer 11 · 2020-07-14T19:10:53.000Z

Hi I am having exactly same error when running demo.py. Have you figured it out @GopiRajuMatta ?

Answer 12 · 2020-07-15T05:37:39.000Z

Hi I am having exactly same error when running demo.py. Have you figured it out @GopiRajuMatta ?

Hello @francisduan

Presently, I am working on other problem.
I will try again and let you know, if it is resolved.

@mihaimorariu would be able to help you.

Thank you
Gopi

Answer 13 · 2020-12-21T11:27:26.000Z

Hello guys,

Has anyone of you resolved the ''make: Circular tf_nndistance.cu <- tf_nndistance.cu.o dependency dropped.'' problem mentioned above? I have tried pretty much anything i could find and no result.
I am working with tf .version==1.12, cuda== 9.0
Thanks in advance,
Yakamoko

Answer 14 · 2021-01-01T06:33:27.000Z

Hi guys, removing -D_GLIBCXX_USE_CXX11_ABI=1 solved my probelm

Answer 15 · 2021-01-03T15:16:31.000Z

Hi gillbam,

Could you plz define the version of tf and cuda-gcc you are using in order to achieve this?

Furthermore, the code you are mentioning for removal is on the makefile right?

Thanks in advance,
Yakamoko

Answer 16 · 2021-01-05T11:55:44.000Z

Hi gillbam,

Could you plz define the version of tf and cuda-gcc you are using in order to achieve this?

Furthermore, the code you are mentioning for removal is on the makefile right?

Thanks in advance,
Yakamoko

Hi Yakamoko,
I have tf 1.12, cuda 10.1 installed. Check my attached code.

cuda_inc = /usr/local/cuda-10.1/include/
cuda_lib = /usr/local/cuda-10.1/lib64/
nvcc = /usr/local/cuda-10.1/bin/nvcc
tf_inc = /root/anaconda3/envs/pcn/lib/python3.6/site-packages/tensorflow/include
tf_lib = /root/anaconda3/envs/pcn/lib/python3.6/site-packages/tensorflow

all: tf_nndistance_so.so tf_approxmatch_so.so

tf_nndistance.cu.o: tf_nndistance.cu
   $(nvcc) tf_nndistance.cu -o tf_nndistance.cu.o -c -O2 -DGOOGLE_CUDA=1 -x cu -Xcompiler -fPIC

tf_nndistance_so.so: tf_nndistance.cpp tf_nndistance.cu.o
   g++ tf_nndistance.cpp tf_nndistance.cu.o -o tf_nndistance_so.so
   -I $(cuda_inc) -I $(tf_inc) -L $(cuda_lib) -lcudart -L $(tf_lib) -ltensorflow_framework
   -shared -std=c++11 -fPIC -O2

tf_approxmatch.cu.o: tf_approxmatch.cu
   $(nvcc) tf_approxmatch.cu -o tf_approxmatch.cu.o -c -O2 -DGOOGLE_CUDA=1 -x cu -Xcompiler -fPIC

tf_approxmatch_so.so: tf_approxmatch.cpp tf_approxmatch.cu.o
    g++ -shared $(CPPFLAGS) tf_approxmatch.cpp tf_approxmatch.cu.o -o tf_approxmatch_so.so
    -I $(cuda_inc) -I $(tf_inc) -L $(cuda_lib) -lcudart -L $(tf_lib) -ltensorflow_framework
    -shared -std=c++11 -fPIC -O2

clean:
    rm -rf *.o *.so

Answer 17 · 2021-11-12T14:20:31.000Z

Hi, I am getting the same error:

make: Circular tf_nndistance.cu <- tf_nndistance.cu.o dependency dropped.
g++ tf_nndistance.cpp tf_nndistance.cu.o -o tf_nndistance_so.so
....
make (e=2): The system cannot find the file specified.
make: *** [makefile:13: tf_nndistance_so.so] Error 2

and I have intalled:
conda install -c nvidia/label/cuda-11.3.0 cuda-nvcc and
conda install -c nvidia cudatoolkit=9.2

using the same makefile above. Is it a version problem still?
Thanks in advance.

Answer 18 · 2022-04-30T17:47:16.000Z

I am trying to run demo.py with the right versions and with the updated versions, and I keep getting an error that I cannot fix. I've been working around this for some days already and can't manage to get anything better.

Traceback (most recent call last):
  File "demo.py", line 121, in <module>
    main()
  File "demo.py", line 77, in main
    model_module = importlib.import_module('.%s' % args.model_type, 'models')
  File "/home/mk/anaconda3/envs/pcnconda/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/mk/Downloads/pcn-master/models/pcn_cd.py", line 26, in <module>
    from tf_util import *
  File "/home/mk/Downloads/pcn-master/tf_util.py", line 26, in <module>
    from pc_distance import tf_nndistance, tf_approxmatch
  File "/home/mk/Downloads/pcn-master/pc_distance/tf_approxmatch.py", line 21, in <module>
    @ops.RegisterShape('ApproxMatch')
AttributeError: module 'tensorflow.python.framework.ops' has no attribute 'RegisterShape'