Error when using TF2.8

Question

Error when using TF2.8

Closed this issue 3 years ago · 16 comments

Hello,
When I run 'mask-rcnn-prediction.py' with TF2.8 and Keras 2.8, I get several errors like:

W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -50 } dim { size: -310 } dim { size: -311 } dim { size: 256 } } } inputs { dtype: DT_FLOAT shape { dim { size: -19 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -19 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } value { dtype: DT_INT32 tensor_shape { dim { size: 2 } } int_val: 7 } } device { type: "GPU" vendor: "NVIDIA" model: "GeForce GTX 1070" frequency: 1746 num_cores: 15 environment { key: "architecture" value: "6.1" } environment { key: "cuda" value: "11020" } environment { key: "cudnn" value: "8100" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 2097152 shared_memory_size_per_multiprocessor: 98304 memory_size: 7497842688 bandwidth: 256256000 } outputs { dtype: DT_FLOAT shape { dim { size: -19 } dim { size: 7 } dim { size: 7 } dim { size: 256 } } }

And the resulting image has a lot of errors, any ideas?.

Thanks,

Carlos.

PS: I've had to change line 27 of 'model.py' to: 'from .parallel_model import ParallelModel'

Answer 1 · 2022-04-28T14:38:45.000Z

Hello again,
I've tested with TF2.7 (in CPU mode) and not errors, but the resulting image has a lot of errors (see attached) and the result is different in every run.

Best regards,
Carlos.

Answer 2 · 2022-04-28T15:09:28.000Z

@cjgarciao Thankyou for posting your issue. I think the issue is popping up because of some library conflicts. There may be two ways to solve this issue,

Please add these few lines at the top your main script and try again...
import tensorflow as tf
config = tf.compat.v1.ConfigProto()
config.gpu_options.allow_growth=True
sess = tf.compat.v1.Session(config=config)
Try with disabling v2 behavior
import tensorflow.compat.v1 as tf
tf.disable_v2_behavior()

Please do let me know if you are able to solve the issue or not.
Best regards,
Kamlesh.

Answer 3 · 2022-04-28T18:47:00.000Z

@Kamlesh364 Thank you for your suggestions.
I have tested separately the two options and the result is the same as without them. I have repeated the tests with GPU and with CPU and it does not work in any of the two cases.
Does it work correctly for you?
What versions of the libraries in the 'requirements.txt' file are you using?

Best regards,

Carlos.

Answer 4 · 2022-04-29T02:54:01.000Z

Hello, When I run 'mask-rcnn-prediction.py' with TF2.8 and Keras 2.8, I get several errors like:

W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -50 } dim { size: -310 } dim { size: -311 } dim { size: 256 } } } inputs { dtype: DT_FLOAT shape { dim { size: -19 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -19 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } value { dtype: DT_INT32 tensor_shape { dim { size: 2 } } int_val: 7 } } device { type: "GPU" vendor: "NVIDIA" model: "GeForce GTX 1070" frequency: 1746 num_cores: 15 environment { key: "architecture" value: "6.1" } environment { key: "cuda" value: "11020" } environment { key: "cudnn" value: "8100" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 2097152 shared_memory_size_per_multiprocessor: 98304 memory_size: 7497842688 bandwidth: 256256000 } outputs { dtype: DT_FLOAT shape { dim { size: -19 } dim { size: 7 } dim { size: 7 } dim { size: 256 } } }

And the resulting image has a lot of errors, any ideas?.

Thanks,

Carlos.

PS: I've had to change line 27 of 'model.py' to: 'from .parallel_model import ParallelModel'

@cjgarciao I am using all the libraries mentioned in the requirements.txt file and it's working perfectly fine.

There might be some warnings and Logs printed by TensorFlow before running the detection. Please share those logs and warnings as well.

Answer 5 · 2022-04-29T03:29:50.000Z

@cjgarciao Thankyou for posting your issue. I think the issue is popping up because of some library conflicts. There may be two ways to solve this issue,

Please add these few lines at the top your main script and try again...
import tensorflow as tf
config = tf.compat.v1.ConfigProto()
config.gpu_options.allow_growth=True
sess = tf.compat.v1.Session(config=config)

Try with disabling v2 behavior
import tensorflow.compat.v1 as tf
tf.disable_v2_behavior()

Please do let me know if you are able to solve the issue or not. Best regards, Kamlesh.

try this one as well -
from tensorflow.compat.v1 import ConfigProto
from tensorflow.compat.v1 import InteractiveSession
config = ConfigProto()
config.gpu_options.allow_growth = True
session = InteractiveSession(config=config)

Answer 6 · 2022-04-29T18:14:43.000Z

@Kamlesh364 I have checked it and it works fine, thank you very much.
Would you know why it works for you without adding those lines and not for me? (just curious).

Carlos.

Answer 7 · 2022-04-29T18:31:37.000Z

@Kamlesh364 I have checked it and it works fine, thank you very much. Would you know why it works for you without adding those lines and not for me? (just curious).

Carlos.

@cjgarciao
I can't tell you the exact reason behind the issue, but it might be because of some conflicts between dependencies.

Well, I am gald that it worked for you.

best regards,
Kamlesh

Answer 8 · 2022-05-20T20:56:12.000Z

hello dear I have same issue when I use tensorflow=2.8
and I installed tesnorflow 2.8 while it makes error in finding compat module ...

Answer 9 · 2022-05-21T00:44:17.000Z

hello dear I have same issue when I use tensorflow=2.8 and I installed tesnorflow 2.8 while it makes error in finding compat module ...

@software93
please downgrade the tensorflow version to 2.7.0 (2.4.0 or 2.4.1, optional) and try again.

Answer 10 · 2022-05-22T23:15:10.000Z

@cjgarciao Thankyou for posting your issue. I think the issue is popping up because of some library conflicts. There may be two ways to solve this issue,

Please add these few lines at the top your main script and try again...
import tensorflow as tf
config = tf.compat.v1.ConfigProto()
config.gpu_options.allow_growth=True
sess = tf.compat.v1.Session(config=config)

Try with disabling v2 behavior
import tensorflow.compat.v1 as tf
tf.disable_v2_behavior()

Please do let me know if you are able to solve the issue or not. Best regards, Kamlesh.

@Kamlesh364 A naive question... Is the main script you referred to "model.py"? Thank you very much!

Answer 11 · 2022-05-23T01:29:35.000Z

@cjgarciao Thankyou for posting your issue. I think the issue is popping up because of some library conflicts. There may be two ways to solve this issue,

Please add these few lines at the top your main script and try again...
import tensorflow as tf
config = tf.compat.v1.ConfigProto()
config.gpu_options.allow_growth=True
sess = tf.compat.v1.Session(config=config)

Try with disabling v2 behavior
import tensorflow.compat.v1 as tf
tf.disable_v2_behavior()

Please do let me know if you are able to solve the issue or not. Best regards, Kamlesh.

@Kamlesh364 A naive question... Is the main script you referred to "model.py"? Thank you very much!

@yongmayer please do let us know your suggestion in both cases-

model.py is main script
it is not the main script.

thankyou very much in advanced.

Answer 12 · 2022-05-23T15:09:51.000Z

modifying model.py worked.

…

On Sun, May 22, 2022 at 8:29 PM Kamlesh Kumar ***@***.***> wrote: @cjgarciao <https://github.com/cjgarciao> Thankyou for posting your issue. I think the issue is popping up because of some library conflicts. There may be two ways to solve this issue, 1. Please add these few lines at the top your main script and try again... import tensorflow as tf config = tf.compat.v1.ConfigProto() config.gpu_options.allow_growth=True sess = tf.compat.v1.Session(config=config) 2. Try with disabling v2 behavior import tensorflow.compat.v1 as tf tf.disable_v2_behavior() Please do let me know if you are able to solve the issue or not. Best regards, Kamlesh. @Kamlesh364 <https://github.com/Kamlesh364> A naive question... Is the main script you referred to "model.py"? Thank you very much! @yongmayer <https://github.com/yongmayer> please do let us know your suggestion in both cases- 1. model.py is main script 2. it is not the main script. thankyou very much in advanced. — Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHKNWV5LZ2IVKVGDLCLGGDVLLNQZANCNFSM5URJPPVA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

Answer 13 · 2022-05-23T15:21:46.000Z

@yongmayer
I request you to post the changes you made, so that everyone can have the benefit.

Answer 14 · 2022-05-23T15:24:16.000Z

I added following lines to model.py, which solved the random prediction error issue...

from tensorflow.compat.v1 import ConfigProto
from tensorflow.compat.v1 import InteractiveSession
config = ConfigProto()
config.gpu_options.allow_growth = True
session = InteractiveSession(config=config)

Answer 15 · 2022-05-23T15:27:52.000Z

I added following lines to model.py, which solved the random prediction error issue...

from tensorflow.compat.v1 import ConfigProto from tensorflow.compat.v1 import InteractiveSession config = ConfigProto() config.gpu_options.allow_growth = True session = InteractiveSession(config=config)

@yongmayer
This is the only solution providing which closed this issue earlier. Thanks for your contribution.

Answer 16 · 2023-01-25T06:40:32.000Z

@cjgarciao Thankyou for posting your issue. I think the issue is popping up because of some library conflicts. There may be two ways to solve this issue,

Please add these few lines at the top your main script and try again...
import tensorflow as tf
config = tf.compat.v1.ConfigProto()
config.gpu_options.allow_growth=True
sess = tf.compat.v1.Session(config=config)

Try with disabling v2 behavior
import tensorflow.compat.v1 as tf
tf.disable_v2_behavior()

Please do let me know if you are able to solve the issue or not. Best regards, Kamlesh.

try this one as well - from tensorflow.compat.v1 import ConfigProto from tensorflow.compat.v1 import InteractiveSession config = ConfigProto() config.gpu_options.allow_growth = True session = InteractiveSession(config=config)

Previously I was also facing the issue of random bounding boxes and mask coming as output of model as faced by @cjgarciao.
After adding the lines of code mentioned by @Kamlesh364 in model.py file as:
from tensorflow.compat.v1 import ConfigProto
from tensorflow.compat.v1 import InteractiveSession
config = ConfigProto() config.gpu_options.allow_growth = True
session = InteractiveSession(config=config)

it is now working and able to detect mask my older model trained using tf1 code as well as tf2 code.

I think the error shown is common for all and should be edited in the main code base.