ERROR:root:Unable to execute run on Tensor("fi_Mean_1:0", dtype=float32)

Question

ERROR:root:Unable to execute run on Tensor("fi_Mean_1:0", dtype=float32)

abhishektyaagi opened this issue 3 years ago · 1 comments

Hi,
While running the logistic_regression.py test case, I am encountering an error.

System Information:
Python=2.7
TensorFlow=1.1.5 (GPU Support)

Command used:
python Tests/logistic_regression.py

Log
WARNING:tensorflow:From Tests/logistic_regression.py:19: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use alternatives such as official/mnist/dataset.py from tensorflow/models.
WARNING:tensorflow:From /u/atyagi2/anaconda3/envs/tf1.15/lib/python2.7/site-packages/tensorflow_core/contrib/learn/python/learn/datasets/mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Please write your own downloading logic.
WARNING:tensorflow:From /u/atyagi2/anaconda3/envs/tf1.15/lib/python2.7/site-packages/tensorflow_core/contrib/learn/python/learn/datasets/mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use tf.data to implement this functionality.
Extracting /tmp/data/train-images-idx3-ubyte.gz
WARNING:tensorflow:From /u/atyagi2/anaconda3/envs/tf1.15/lib/python2.7/site-packages/tensorflow_core/contrib/learn/python/learn/datasets/mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use tf.data to implement this functionality.
Extracting /tmp/data/train-labels-idx1-ubyte.gz
WARNING:tensorflow:From /u/atyagi2/anaconda3/envs/tf1.15/lib/python2.7/site-packages/tensorflow_core/contrib/learn/python/learn/datasets/mnist.py:110: dense_to_one_hot (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use tf.one_hot on tensors.
Extracting /tmp/data/t10k-images-idx3-ubyte.gz
Extracting /tmp/data/t10k-labels-idx1-ubyte.gz
WARNING:tensorflow:From /u/atyagi2/anaconda3/envs/tf1.15/lib/python2.7/site-packages/tensorflow_core/contrib/learn/python/learn/datasets/mnist.py:290: init (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use alternatives such as official/mnist/dataset.py from tensorflow/models.
WARNING:tensorflow:From Tests/logistic_regression.py:28: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From Tests/logistic_regression.py:39: The name tf.log is deprecated. Please use tf.math.log instead.

WARNING:tensorflow:From Tests/logistic_regression.py:41: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

WARNING:tensorflow:From Tests/logistic_regression.py:44: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.

WARNING:tensorflow:From Tests/logistic_regression.py:47: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2021-06-09 20:47:16.283375: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2021-06-09 20:47:16.295312: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:5e:00.0
2021-06-09 20:47:16.295797: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 1 with properties:
name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:d8:00.0
2021-06-09 20:47:16.297107: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-06-09 20:47:16.299884: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-06-09 20:47:16.302552: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2021-06-09 20:47:16.303956: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2021-06-09 20:47:16.306989: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2021-06-09 20:47:16.309497: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2021-06-09 20:47:16.316184: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-06-09 20:47:16.318198: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0, 1
2021-06-09 20:47:16.321113: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2021-06-09 20:47:16.334672: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2100000000 Hz
2021-06-09 20:47:16.336866: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x563559e9e1d0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-06-09 20:47:16.336902: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2021-06-09 20:47:16.638280: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:5e:00.0
2021-06-09 20:47:16.638773: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 1 with properties:
name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:d8:00.0
2021-06-09 20:47:16.638838: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-06-09 20:47:16.638858: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-06-09 20:47:16.638874: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2021-06-09 20:47:16.638891: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2021-06-09 20:47:16.638908: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2021-06-09 20:47:16.638949: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2021-06-09 20:47:16.638966: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-06-09 20:47:16.640706: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0, 1
2021-06-09 20:47:16.640753: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-06-09 20:47:16.642010: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-06-09 20:47:16.642032: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 1
2021-06-09 20:47:16.642048: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N N
2021-06-09 20:47:16.642058: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 1: N N
2021-06-09 20:47:16.643804: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3050 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:5e:00.0, compute capability: 7.5)
2021-06-09 20:47:16.645540: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 3050 MB memory) -> physical GPU (device: 1, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:d8:00.0, compute capability: 7.5)
2021-06-09 20:47:16.649265: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x56355ad10da0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-06-09 20:47:16.649295: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5
2021-06-09 20:47:16.649308: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (1): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5
2021-06-09 20:47:17.779916: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
Epoch: 0001 cost= 1.183628618
Epoch: 0002 cost= 0.665431309
Epoch: 0003 cost= 0.552812954
Epoch: 0004 cost= 0.498707291
Epoch: 0005 cost= 0.465449734
Epoch: 0006 cost= 0.442609885
Epoch: 0007 cost= 0.425485988
Epoch: 0008 cost= 0.412153071
Epoch: 0009 cost= 0.401357169
Epoch: 0010 cost= 0.392367188
Epoch: 0011 cost= 0.384738742
Epoch: 0012 cost= 0.378193522
Epoch: 0013 cost= 0.372418279
Epoch: 0014 cost= 0.367249739
Epoch: 0015 cost= 0.362749526
Epoch: 0016 cost= 0.358563266
Epoch: 0017 cost= 0.354849673
Epoch: 0018 cost= 0.351454730
Epoch: 0019 cost= 0.348317068
Epoch: 0020 cost= 0.345396632
Epoch: 0021 cost= 0.342762511
Epoch: 0022 cost= 0.340233821
Epoch: 0023 cost= 0.337921471
Epoch: 0024 cost= 0.335748342
Epoch: 0025 cost= 0.333688100
Optimization Finished!
Accuracy: 0.9135
/u/atyagi2/anaconda3/envs/tf1.15/lib/python2.7/site-packages/TensorFI/fiConfig.py:270: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
params = yaml.load(pStream)
WARNING:tensorflow:From /u/atyagi2/anaconda3/envs/tf1.15/lib/python2.7/site-packages/TensorFI/modifyGraph.py:34: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
options available in V2.
- tf.py_function takes a python function which manipulates tf eager
tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
an ndarray (just call tensor.numpy()) but having access to eager tensors
means tf.py_functions can use accelerators such as GPUs as well as
being differentiable using a gradient tape.
- tf.numpy_function maintains the semantics of the deprecated tf.py_func
(it is not differentiable, and manipulates numpy arrays). It drops the
stateful argument making all functions stateful.

WARNING:tensorflow:From /u/atyagi2/anaconda3/envs/tf1.15/lib/python2.7/site-packages/TensorFI/modifyGraph.py:34: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
options available in V2.
- tf.py_function takes a python function which manipulates tf eager
tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
an ndarray (just call tensor.numpy()) but having access to eager tensors
means tf.py_functions can use accelerators such as GPUs as well as
being differentiable using a gradient tape.
- tf.numpy_function maintains the semantics of the deprecated tf.py_func
(it is not differentiable, and manipulates numpy arrays). It drops the
stateful argument making all functions stateful.

Accuracy (with no injections): 0.9135
WARNING:tensorflow:From Tests/logistic_regression.py:85: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

WARNING:tensorflow:From Tests/logistic_regression.py:85: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

2021-06-09 20:49:13.454866: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:5e:00.0
2021-06-09 20:49:13.456646: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 1 with properties:
name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:d8:00.0
2021-06-09 20:49:13.456835: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-06-09 20:49:13.456945: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-06-09 20:49:13.457038: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2021-06-09 20:49:13.457137: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2021-06-09 20:49:13.457219: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2021-06-09 20:49:13.457327: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2021-06-09 20:49:13.457408: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-06-09 20:49:13.462890: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0, 1
2021-06-09 20:49:13.463164: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-06-09 20:49:13.463219: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 1
2021-06-09 20:49:13.463259: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N N
2021-06-09 20:49:13.463295: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 1: N N
2021-06-09 20:49:13.467357: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3050 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:5e:00.0, compute capability: 7.5)
2021-06-09 20:49:13.468799: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 3050 MB memory) -> physical GPU (device: 1, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:d8:00.0, compute capability: 7.5)
2021-06-09 20:49:13.482558: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:5e:00.0
2021-06-09 20:49:13.483020: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 1 with properties:
name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:d8:00.0
2021-06-09 20:49:13.483080: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-06-09 20:49:13.483100: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-06-09 20:49:13.483123: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2021-06-09 20:49:13.483138: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2021-06-09 20:49:13.483155: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2021-06-09 20:49:13.483172: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2021-06-09 20:49:13.483189: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-06-09 20:49:13.484656: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0, 1
2021-06-09 20:49:13.484699: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-06-09 20:49:13.484712: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 1
2021-06-09 20:49:13.484724: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N N
2021-06-09 20:49:13.484734: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 1: N N
2021-06-09 20:49:13.485860: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3050 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:5e:00.0, compute capability: 7.5)
2021-06-09 20:49:13.486318: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 3050 MB memory) -> physical GPU (device: 1, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:d8:00.0, compute capability: 7.5)
ERROR:root:Encountered exception pyfunc_6 returns 2 values, but expects to see 1 values.
[[node fi_add (defined at /u/atyagi2/anaconda3/envs/tf1.15/lib/python2.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]

Original stack trace for u'fi_add':
File "Tests/logistic_regression.py", line 78, in
fi = ti.TensorFI(sess, name = "logistReg", logLevel = 30, disableInjections = True)
File "/u/atyagi2/anaconda3/envs/tf1.15/lib/python2.7/site-packages/TensorFI/tensorFI.py", line 189, in init
self.fiMap = mg.modifyNodes(graph, fiPrefix)
File "/u/atyagi2/anaconda3/envs/tf1.15/lib/python2.7/site-packages/TensorFI/modifyGraph.py", line 98, in modifyNodes
newOp = createFIFunc(op.type, inputs, outputTypeList, name)
File "/u/atyagi2/anaconda3/envs/tf1.15/lib/python2.7/site-packages/TensorFI/modifyGraph.py", line 34, in createFIFunc
res = tf.py_func(fiFunc, inputs, outputTypes, name = name)
File "/u/atyagi2/anaconda3/envs/tf1.15/lib/python2.7/site-packages/tensorflow_core/python/util/deprecation.py", line 324, in new_func
return func(*args, **kwargs)
File "/u/atyagi2/anaconda3/envs/tf1.15/lib/python2.7/site-packages/tensorflow_core/python/ops/script_ops.py", line 513, in py_func
return py_func_common(func, inp, Tout, stateful, name=name)
File "/u/atyagi2/anaconda3/envs/tf1.15/lib/python2.7/site-packages/tensorflow_core/python/ops/script_ops.py", line 495, in py_func_common
func=func, inp=inp, Tout=Tout, stateful=stateful, eager=False, name=name)
File "/u/atyagi2/anaconda3/envs/tf1.15/lib/python2.7/site-packages/tensorflow_core/python/ops/script_ops.py", line 318, in _internal_py_func
input=inp, token=token, Tout=Tout, name=name)
File "/u/atyagi2/anaconda3/envs/tf1.15/lib/python2.7/site-packages/tensorflow_core/python/ops/gen_script_ops.py", line 170, in py_func
"PyFunc", input=input, token=token, Tout=Tout, name=name)
File "/u/atyagi2/anaconda3/envs/tf1.15/lib/python2.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "/u/atyagi2/anaconda3/envs/tf1.15/lib/python2.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/u/atyagi2/anaconda3/envs/tf1.15/lib/python2.7/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "/u/atyagi2/anaconda3/envs/tf1.15/lib/python2.7/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "/u/atyagi2/anaconda3/envs/tf1.15/lib/python2.7/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in init
self._traceback = tf_stack.extract_stack()

ERROR:root:Unable to execute run on Tensor("fi_Mean_1:0", dtype=float32)
Accuracy (with injections): None

Is it possible to elaborate on what the issue might be as this is the only test which is not passing

Answer 1 · 2021-06-13T01:01:31.000Z

Hi @abhishek-t-naive I believe this is a documented bug with injectFaultMean (fi_Mean), see #22 and #42. We will soon have someone look into this, meanwhile feel free to dig deeper and send a fix PR yourself ;)