Tensorflow crashes
SHITIANYU-hue opened this issue · 6 comments
I found that if I installed the macad-agent and run the basic_agent in macad-gym(https://github.com/praveen-palanisamy/macad-gym/blob/master/examples/basic_agent.py), there will be some issues with importing tensorflow modules. The detailed log is:
Traceback (most recent call last):
File "/home/tianyushi/code/macad-gym/examples/basic_agent.py", line 5, in <module>
env = gym.make("HomoNcomIndePOIntrxMASS3CTWN3-v0")
File "/home/tianyushi/miniconda3/envs/macad-gym/lib/python3.6/site-packages/gym/envs/registration.py", line 183, in make
return registry.make(id, **kwargs)
File "/home/tianyushi/miniconda3/envs/macad-gym/lib/python3.6/site-packages/gym/envs/registration.py", line 125, in make
env = spec.make(**kwargs)
File "/home/tianyushi/miniconda3/envs/macad-gym/lib/python3.6/site-packages/gym/envs/registration.py", line 88, in make
cls = load(self._entry_point)
File "/home/tianyushi/miniconda3/envs/macad-gym/lib/python3.6/site-packages/gym/envs/registration.py", line 17, in load
mod = importlib.import_module(mod_name)
File "/home/tianyushi/miniconda3/envs/macad-gym/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/tianyushi/code/macad-gym/src/macad_gym/envs/__init__.py", line 1, in <module>
from macad_gym.carla.multi_env import MultiCarlaEnv
File "/home/tianyushi/code/macad-gym/src/macad_gym/carla/multi_env.py", line 220, in <module>
**from ray.rllib.env import MultiAgentEnv**
File "/home/tianyushi/miniconda3/envs/macad-gym/lib/python3.6/site-packages/ray/rllib/__init__.py", line 11, in <module>
from ray.rllib.evaluation.policy_graph import PolicyGraph
File "/home/tianyushi/miniconda3/envs/macad-gym/lib/python3.6/site-packages/ray/rllib/evaluation/__init__.py", line 2, in <module>
from ray.rllib.evaluation.policy_evaluator import PolicyEvaluator
File "/home/tianyushi/miniconda3/envs/macad-gym/lib/python3.6/site-packages/ray/rllib/evaluation/policy_evaluator.py", line 18, in <module>
from ray.rllib.evaluation.sampler import AsyncSampler, SyncSampler
File "/home/tianyushi/miniconda3/envs/macad-gym/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 14, in <module>
from ray.rllib.evaluation.tf_policy_graph import TFPolicyGraph
File "/home/tianyushi/miniconda3/envs/macad-gym/lib/python3.6/site-packages/ray/rllib/evaluation/tf_policy_graph.py", line 12, in <module>
from ray.rllib.models.lstm import chop_into_sequences
File "/home/tianyushi/miniconda3/envs/macad-gym/lib/python3.6/site-packages/ray/rllib/models/__init__.py", line 1, in <module>
from ray.rllib.models.catalog import ModelCatalog, MODEL_DEFAULTS
File "/home/tianyushi/miniconda3/envs/macad-gym/lib/python3.6/site-packages/ray/rllib/models/catalog.py", line 17, in <module>
from ray.rllib.models.fcnet import FullyConnectedNetwork
File "/home/tianyushi/miniconda3/envs/macad-gym/lib/python3.6/site-packages/ray/rllib/models/fcnet.py", line 6, in <module>
import tensorflow.contrib.slim as slim
File "/home/tianyushi/.local/lib/python3.6/site-packages/tensorflow/contrib/__init__.py", line 28, in <module>
from tensorflow.contrib import cudnn_rnn
File "/home/tianyushi/.local/lib/python3.6/site-packages/tensorflow/contrib/cudnn_rnn/__init__.py", line 33, in <module>
from tensorflow.contrib.cudnn_rnn.python.ops.cudnn_rnn_ops import CudnnCompatibleGRUCell
File "/home/tianyushi/.local/lib/python3.6/site-packages/tensorflow/contrib/cudnn_rnn/python/ops/cudnn_rnn_ops.py", line 21, in <module>
from tensorflow.contrib.rnn.python.ops import lstm_ops
File "/home/tianyushi/.local/lib/python3.6/site-packages/tensorflow/contrib/rnn/__init__.py", line 83, in <module>
from tensorflow.contrib.rnn.python.ops.gru_ops import *
File "/home/tianyushi/.local/lib/python3.6/site-packages/tensorflow/contrib/rnn/python/ops/gru_ops.py", line 33, in <module>
resource_loader.get_path_to_datafile("_gru_ops.so"))
File "/home/tianyushi/.local/lib/python3.6/site-packages/tensorflow/contrib/util/loader.py", line 55, in load_op_library
ret = load_library.load_op_library(path)
File "/home/tianyushi/.local/lib/python3.6/site-packages/tensorflow/python/framework/load_library.py", line 56, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename, status)
File "/home/tianyushi/.local/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Could not parse default value '4000' from Attr("upper_frequency_limit: float = 4000") for Op Mfcc
Could not parse default value '20' from Attr("lower_frequency_limit: float = 20") for Op MfccI found a discussion related to this problem: tensorflow/tensorflow#13963
The main problem is due to these lines in the multi_env.py
Hi @SHITIANYU-hue ,
Could you please paste your full log? There is no crash information in the log you posted.
Also, there is no tensorflow import statement in https://github.com/praveen-palanisamy/macad-gym/blob/master/examples/basic_agent.py. I wonder which line is causing the issue you linked to when you run.
Hello, yes, there is no TensorFlow in basic_agent, but it used the import macad_gym and env = gym.make("HomoNcomIndePOIntrxMASS3CTWN3-v0"). Therefore, the multi_env will be used https://github.com/praveen-palanisamy/macad-gym/blob/master/src/macad_gym/carla/multi_env.py. And the multi_env will use the ray. So it will occur this problem
I am facing a similar issue. Tensorflow gives the same error if I want to merge macad-agents or ray with macad-gym.
Can you post your crash log and your python environment with package version (pip freeze > pyenv.txt)?
As long as you are using the python package versions listed in the conda-env.yml file there should be no crash due to Ray or TensorFlow.
The existing conda-env.yml contains tensorflow-gpu only, and when we install tensorflow (whether of same version or different) the code crashes once env = gym.make is called. Please find the attached files.