InconsistencyError: Trying to reintroduce a removed node

Question

InconsistencyError: Trying to reintroduce a removed node

dhirajmadan1 opened this issue 7 years ago · 1 comments

I am facing error 'InconsistencyError: Trying to reintroduce a removed node' while running HRED on ubuntu dataset.
Here is the stack trace:-
ERROR (theano.gof.opt): SeqOptimizer apply <theano.scan_module.scan_opt.PushOutScanOutput object at 0x2b5b16dd7b50>
2017-11-14 01:11:53,554: theano.gof.opt: ERROR: SeqOptimizer apply <theano.scan_module.scan_opt.PushOutScanOutput object at 0x2b5b16dd7b50>
ERROR (theano.gof.opt): Traceback:
2017-11-14 01:11:53,554: theano.gof.opt: ERROR: Traceback:
ERROR (theano.gof.opt): Traceback (most recent call last):
File "/u/dhimadan/anaconda2/envs/tf1.1/lib/python2.7/site-packages/theano/gof/opt.py", line 235, in apply
sub_prof = optimizer.optimize(fgraph)
File "/u/dhimadan/anaconda2/envs/tf1.1/lib/python2.7/site-packages/theano/gof/opt.py", line 87, in optimize
ret = self.apply(fgraph, *args, **kwargs)
File "/u/dhimadan/anaconda2/envs/tf1.1/lib/python2.7/site-packages/theano/scan_module/scan_opt.py", line 685, in apply
node = self.process_node(fgraph, node)
File "/u/dhimadan/anaconda2/envs/tf1.1/lib/python2.7/site-packages/theano/scan_module/scan_opt.py", line 745, in process_node
node, args)
File "/u/dhimadan/anaconda2/envs/tf1.1/lib/python2.7/site-packages/theano/scan_module/scan_opt.py", line 854, in push_out_inner_vars
add_as_nitsots)
File "/u/dhimadan/anaconda2/envs/tf1.1/lib/python2.7/site-packages/theano/scan_module/scan_opt.py", line 906, in add_nitsot_outputs
reason='scanOp_pushout_output')
File "/u/dhimadan/anaconda2/envs/tf1.1/lib/python2.7/site-packages/theano/gof/toolbox.py", line 391, in replace_all_validate_remove
chk = fgraph.replace_all_validate(replacements, reason)
File "/u/dhimadan/anaconda2/envs/tf1.1/lib/python2.7/site-packages/theano/gof/toolbox.py", line 365, in replace_all_validate
fgraph.validate()
File "/u/dhimadan/anaconda2/envs/tf1.1/lib/python2.7/site-packages/theano/gof/toolbox.py", line 256, in validate_
ret = fgraph.execute_callbacks('validate')
File "/u/dhimadan/anaconda2/envs/tf1.1/lib/python2.7/site-packages/theano/gof/fg.py", line 589, in execute_callbacks
fn(self, *args, **kwargs)
File "/u/dhimadan/anaconda2/envs/tf1.1/lib/python2.7/site-packages/theano/gof/toolbox.py", line 422, in validate
raise theano.gof.InconsistencyError("Trying to reintroduce a removed node")
InconsistencyError: Trying to reintroduce a removed node

I am calling it as:-
THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python train.py --prototype prototype_ubuntu_HRED

Thanks and Regards
Dhiraj

Answer 1 · 2019-04-02T06:19:09.000Z

I got the same error, but the training script didn't stop for that.
Command I used:
ssh://cgsdfc@192.168.1.175:22:docker run --runtime nvidia -v /home/cgsdfc/deployment/HRED-VHRED:/root/HRED-VHRED -v /home/cgsdfc/UbuntuDialogueCorpus:/home/cgsdfc/UbuntuDialogueCorpus -w /root/HRED-VHRED -e PYTHONPATH=/root/HRED-VHRED ufoym/deepo:theano-py36-cu90 python bin/train.py prototype_ubuntu_HRED

error log:
2019-04-02 06:08:12,502: serban.dialog_encoder_decoder:1420: DEBUG: Building train function
/root/HRED-VHRED/serban/dialog_encoder_decoder.py:1430: UserWarning: theano.function was asked to create a function computing outputs given certain inputs, but the provided input variable at index 1 is not part of the computational graph needed to compute the outputs: x_data_reversed.
To make this warning into an error, you can pass the parameter on_unused_input='raise' to theano.function. To disable it completely, use on_unused_input='ignore'.
name="train_fn")
/root/HRED-VHRED/serban/dialog_encoder_decoder.py:1430: UserWarning: theano.function was asked to create a function computing outputs given certain inputs, but the provided input variable at index 5 is not part of the computational graph needed to compute the outputs: ran_cost_utterance.
To make this warning into an error, you can pass the parameter on_unused_input='raise' to theano.function. To disable it completely, use on_unused_input='ignore'.
name="train_fn")
/root/HRED-VHRED/serban/dialog_encoder_decoder.py:1430: UserWarning: theano.function was asked to create a function computing outputs given certain inputs, but the provided input variable at index 6 is not part of the computational graph needed to compute the outputs: x_dropmask.
To make this warning into an error, you can pass the parameter on_unused_input='raise' to theano.function. To disable it completely, use on_unused_input='ignore'.
name="train_fn")
ERROR (theano.gof.opt): SeqOptimizer apply <theano.scan_module.scan_opt.PushOutScanOutput object at 0x7fb251ae9240>
2019-04-02 06:08:13,636: theano.gof.opt:198: ERROR: SeqOptimizer apply <theano.scan_module.scan_opt.PushOutScanOutput object at 0x7fb251ae9240>
ERROR (theano.gof.opt): Traceback:
2019-04-02 06:08:13,636: theano.gof.opt:199: ERROR: Traceback:
ERROR (theano.gof.opt): Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/theano/gof/opt.py", line 251, in apply
sub_prof = optimizer.optimize(fgraph)
File "/usr/local/lib/python3.6/dist-packages/theano/gof/opt.py", line 97, in optimize
ret = self.apply(fgraph, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 685, in apply
node = self.process_node(fgraph, node)
File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 745, in process_node
node, args)
File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 854, in push_out_inner_vars
add_as_nitsots)
File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 906, in add_nitsot_outputs
reason='scanOp_pushout_output')
File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 569, in replace_all_validate_remove
chk = fgraph.replace_all_validate(replacements, reason)
File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 543, in replace_all_validate
fgraph.validate()
File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 434, in validate_
ret = fgraph.execute_callbacks('validate')
File "/usr/local/lib/python3.6/dist-packages/theano/gof/fg.py", line 594, in execute_callbacks
fn(self, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 600, in validate
raise theano.gof.InconsistencyError("Trying to reintroduce a removed node")
theano.gof.fg.InconsistencyError: Trying to reintroduce a removed node

2019-04-02 06:08:13,637: theano.gof.opt:200: ERROR: Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/theano/gof/opt.py", line 251, in apply
sub_prof = optimizer.optimize(fgraph)
File "/usr/local/lib/python3.6/dist-packages/theano/gof/opt.py", line 97, in optimize
ret = self.apply(fgraph, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 685, in apply
node = self.process_node(fgraph, node)
File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 745, in process_node
node, args)
File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 854, in push_out_inner_vars
add_as_nitsots)
File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 906, in add_nitsot_outputs
reason='scanOp_pushout_output')
File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 569, in replace_all_validate_remove
chk = fgraph.replace_all_validate(replacements, reason)
File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 543, in replace_all_validate
fgraph.validate()
File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 434, in validate_
ret = fgraph.execute_callbacks('validate')
File "/usr/local/lib/python3.6/dist-packages/theano/gof/fg.py", line 594, in execute_callbacks
fn(self, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 600, in validate
raise theano.gof.InconsistencyError("Trying to reintroduce a removed node")
theano.gof.fg.InconsistencyError: Trying to reintroduce a removed node

ERROR (theano.gof.opt): SeqOptimizer apply <theano.scan_module.scan_opt.PushOutScanOutput object at 0x7fb251ae9240>
2019-04-02 06:08:14,208: theano.gof.opt:198: ERROR: SeqOptimizer apply <theano.scan_module.scan_opt.PushOutScanOutput object at 0x7fb251ae9240>
ERROR (theano.gof.opt): Traceback:
2019-04-02 06:08:14,208: theano.gof.opt:199: ERROR: Traceback:
ERROR (theano.gof.opt): Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/theano/gof/opt.py", line 251, in apply
sub_prof = optimizer.optimize(fgraph)
File "/usr/local/lib/python3.6/dist-packages/theano/gof/opt.py", line 97, in optimize
ret = self.apply(fgraph, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 685, in apply
node = self.process_node(fgraph, node)
File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 745, in process_node
node, args)
File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 854, in push_out_inner_vars
add_as_nitsots)
File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 906, in add_nitsot_outputs
reason='scanOp_pushout_output')
File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 569, in replace_all_validate_remove
chk = fgraph.replace_all_validate(replacements, reason)
File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 543, in replace_all_validate
fgraph.validate()
File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 434, in validate_
ret = fgraph.execute_callbacks('validate')
File "/usr/local/lib/python3.6/dist-packages/theano/gof/fg.py", line 594, in execute_callbacks
fn(self, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 600, in validate
raise theano.gof.InconsistencyError("Trying to reintroduce a removed node")
theano.gof.fg.InconsistencyError: Trying to reintroduce a removed node

2019-04-02 06:08:14,208: theano.gof.opt:200: ERROR: Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/theano/gof/opt.py", line 251, in apply
sub_prof = optimizer.optimize(fgraph)
File "/usr/local/lib/python3.6/dist-packages/theano/gof/opt.py", line 97, in optimize
ret = self.apply(fgraph, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 685, in apply
node = self.process_node(fgraph, node)
File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 745, in process_node
node, args)
File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 854, in push_out_inner_vars
add_as_nitsots)
File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 906, in add_nitsot_outputs
reason='scanOp_pushout_output')
File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 569, in replace_all_validate_remove
chk = fgraph.replace_all_validate(replacements, reason)
File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 543, in replace_all_validate
fgraph.validate()
File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 434, in validate_
ret = fgraph.execute_callbacks('validate')
File "/usr/local/lib/python3.6/dist-packages/theano/gof/fg.py", line 594, in execute_callbacks
fn(self, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 600, in validate
raise theano.gof.InconsistencyError("Trying to reintroduce a removed node")
theano.gof.fg.InconsistencyError: Trying to reintroduce a removed node

ERROR (theano.gof.opt): SeqOptimizer apply <theano.scan_module.scan_opt.PushOutScanOutput object at 0x7fb251ae9240>
2019-04-02 06:08:14,502: theano.gof.opt:198: ERROR: SeqOptimizer apply <theano.scan_module.scan_opt.PushOutScanOutput object at 0x7fb251ae9240>
ERROR (theano.gof.opt): Traceback:
2019-04-02 06:08:14,502: theano.gof.opt:199: ERROR: Traceback:
ERROR (theano.gof.opt): Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/theano/gof/opt.py", line 251, in apply
sub_prof = optimizer.optimize(fgraph)
File "/usr/local/lib/python3.6/dist-packages/theano/gof/opt.py", line 97, in optimize
ret = self.apply(fgraph, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 685, in apply
node = self.process_node(fgraph, node)
File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 745, in process_node
node, args)
File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 854, in push_out_inner_vars
add_as_nitsots)
File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 906, in add_nitsot_outputs
reason='scanOp_pushout_output')
File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 569, in replace_all_validate_remove
chk = fgraph.replace_all_validate(replacements, reason)
File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 543, in replace_all_validate
fgraph.validate()
File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 434, in validate_
ret = fgraph.execute_callbacks('validate')
File "/usr/local/lib/python3.6/dist-packages/theano/gof/fg.py", line 594, in execute_callbacks
fn(self, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 600, in validate
raise theano.gof.InconsistencyError("Trying to reintroduce a removed node")
theano.gof.fg.InconsistencyError: Trying to reintroduce a removed node

2019-04-02 06:08:14,503: theano.gof.opt:200: ERROR: Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/theano/gof/opt.py", line 251, in apply
sub_prof = optimizer.optimize(fgraph)
File "/usr/local/lib/python3.6/dist-packages/theano/gof/opt.py", line 97, in optimize
ret = self.apply(fgraph, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 685, in apply
node = self.process_node(fgraph, node)
File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 745, in process_node
node, args)
File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 854, in push_out_inner_vars
add_as_nitsots)
File "/usr/local/lib/python3.6/dist-packages/theano/scan_module/scan_opt.py", line 906, in add_nitsot_outputs
reason='scanOp_pushout_output')
File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 569, in replace_all_validate_remove
chk = fgraph.replace_all_validate(replacements, reason)
File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 543, in replace_all_validate
fgraph.validate()
File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 434, in validate_
ret = fgraph.execute_callbacks('validate')
File "/usr/local/lib/python3.6/dist-packages/theano/gof/fg.py", line 594, in execute_callbacks
fn(self, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/theano/gof/toolbox.py", line 600, in validate
raise theano.gof.InconsistencyError("Trying to reintroduce a removed node")
theano.gof.fg.InconsistencyError: Trying to reintroduce a removed node

If the training don't stop for this error, can I just ignore it?
Thanks and Regards
Cong