Error when inputting custom data
Closed this issue · 6 comments
Hello,
I've successfully ran ConvNetQuake on the example .mseed data. When I feed my own data though, converting the .mseed data to tf records first, I get this error:
./bin/predict_from_tfrecords.py \
> --dataset data/tfrecordtz \
> --checkpoint_dir models/convnetquake \
> --n_clusters 6 \
> --max_windows 2678400 \
> --output data/output/tznew
/opt/conda/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')
WARNING:tensorflow:VARIABLES collection name is deprecated, please use GLOBAL_VARIABLES instead; VARIABLES will be removed after 2017-03-02.
WARNING:tensorflow:VARIABLES collection name is deprecated, please use GLOBAL_VARIABLES instead; VARIABLES will be removed after 2017-03-02.
WARNING:tensorflow:VARIABLES collection name is deprecated, please use GLOBAL_VARIABLES instead; VARIABLES will be removed after 2017-03-02.
WARNING:tensorflow:VARIABLES collection name is deprecated, please use GLOBAL_VARIABLES instead; VARIABLES will be removed after 2017-03-02.
WARNING:tensorflow:VARIABLES collection name is deprecated, please use GLOBAL_VARIABLES instead; VARIABLES will be removed after 2017-03-02.
WARNING:tensorflow:VARIABLES collection name is deprecated, please use GLOBAL_VARIABLES instead; VARIABLES will be removed after 2017-03-02.
WARNING:tensorflow:VARIABLES collection name is deprecated, please use GLOBAL_VARIABLES instead; VARIABLES will be removed after 2017-03-02.
WARNING:tensorflow:VARIABLES collection name is deprecated, please use GLOBAL_VARIABLES instead; VARIABLES will be removed after 2017-03-02.
WARNING:tensorflow:VARIABLES collection name is deprecated, please use GLOBAL_VARIABLES instead; VARIABLES will be removed after 2017-03-02.
WARNING:tensorflow:VARIABLES collection name is deprecated, please use GLOBAL_VARIABLES instead; VARIABLES will be removed after 2017-03-02.
WARNING:tensorflow:VARIABLES collection name is deprecated, please use GLOBAL_VARIABLES instead; VARIABLES will be removed after 2017-03-02.
WARNING:tensorflow:VARIABLES collection name is deprecated, please use GLOBAL_VARIABLES instead; VARIABLES will be removed after 2017-03-02.
WARNING:tensorflow:VARIABLES collection name is deprecated, please use GLOBAL_VARIABLES instead; VARIABLES will be removed after 2017-03-02.
WARNING:tensorflow:VARIABLES collection name is deprecated, please use GLOBAL_VARIABLES instead; VARIABLES will be removed after 2017-03-02.
WARNING:tensorflow:VARIABLES collection name is deprecated, please use GLOBAL_VARIABLES instead; VARIABLES will be removed after 2017-03-02.
WARNING:tensorflow:VARIABLES collection name is deprecated, please use GLOBAL_VARIABLES instead; VARIABLES will be removed after 2017-03-02.
WARNING:tensorflow:From /data/tflib/model.py:35 in __init__.: all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Please use tf.global_variables instead.
Catalog created to store events data/output/tznew/catalog_detection.csv
WARNING:tensorflow:From ./bin/predict_from_tfrecords.py:89 in main.: initialize_local_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Use `tf.local_variables_initializer` instead.
Loaded model at step 32000 from snapshot models/convnetquake/model-32000.
Predicting using model at step 32000
Evaluation completed (1 epochs).
joining data threads
Prediction took 0.0 min 0.0909900665283 seconds
Traceback (most recent call last):
File "./bin/predict_from_tfrecords.py", line 178, in <module>
main(args)
File "./bin/predict_from_tfrecords.py", line 150, in main
coord.join(threads)
File "/opt/conda/lib/python2.7/site-packages/tensorflow/python/training/coordinator.py", line 386, in join
six.reraise(*self._exc_info_to_raise)
File "/opt/conda/lib/python2.7/site-packages/tensorflow/python/training/queue_runner_impl.py", line 234, in _run
sess.run(enqueue_op)
File "/opt/conda/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 766, in run
run_metadata_ptr)
File "/opt/conda/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 964, in _run
feed_dict_string, options, run_metadata)
File "/opt/conda/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1014, in _do_run
target_list, options, run_metadata)
File "/opt/conda/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1034, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Input to reshape is a tensor with 6003 values, but the requested shape has 3003
[[Node: validation_inputs/Reshape = Reshape[T=DT_FLOAT, Tshape=DT_INT32, _device="/job:localhost/replica:0/task:0/cpu:0"](validation_inputs/DecodeRaw, validation_inputs/Reshape/shape)]]
Caused by op u'validation_inputs/Reshape', defined at:
File "./bin/predict_from_tfrecords.py", line 178, in <module>
main(args)
File "./bin/predict_from_tfrecords.py", line 57, in main
is_training=False)
File "/data/quakenet/data_pipeline.py", line 152, in __init__
samples = self._reader.read()
File "/data/quakenet/data_pipeline.py", line 81, in read
example = self._parse_example(serialized_example)
File "/data/quakenet/data_pipeline.py", line 109, in _parse_example
data = tf.reshape(data, [self.n_traces, self.win_size])
File "/opt/conda/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 2448, in reshape
name=name)
File "/opt/conda/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op
op_def=op_def)
File "/opt/conda/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2240, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/opt/conda/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1128, in __init__
self._traceback = _extract_stack()
InvalidArgumentError (see above for traceback): Input to reshape is a tensor with 6003 values, but the requested shape has 3003
[[Node: validation_inputs/Reshape = Reshape[T=DT_FLOAT, Tshape=DT_INT32, _device="/job:localhost/replica:0/task:0/cpu:0"](validation_inputs/DecodeRaw, validation_inputs/Reshape/shape)]]
Any ideas about how to solve this? I think it is a problem with how I have formatted the .mseed file, but I'm not sure what's wrong. Thanks!!
Also, unrelated, I'll submit a pull request with a Dockerfile (running on CPU) for this project soon! Should help new people get up and running a lot faster
Hi there. sorry for the slow response. It looks like you are trying to reshape a tensor of 6003 values instead of 3003 values (3 channels sampled at 100 Hz for 10 seconds). You can use the plot_windows_from_tfrecords.py to inspect the windows generated and check that they have the correct format.
Hi, I'm installing ConvNetQuake, but I have two errors that are:
-
Error [WinError 2] The system can not find the specified file while executing command git clone -q https://github.com/gem/oq-hazardlib.git C: \ Users \ l2cv \ src \ openquake.hazardlib Can not find command 'git'
-
Could not find a version that satisfies the requirement tensorflow == 0.11.0rc1 (from -r requirements.txt (line 8)) (from versions: 1.2.0rc2, 1.2.0, 1.2.1, 1.3.0rc0, 1.3. 0rc1, 1.3.0rc2, 1.3.0, 1.4.0rc0, 1.4.0rc1, 1.4.0, 1.5.0rc0, 1.5.0rc1, 1.5.0, 1.6.0rc0, 1.6.0rc1, 1.6.0) No matching distribution found for tensorflow == 0.11.0rc1
With respect to tensorflow I will verify if there is another compatible distribution. You could help me with the first error.
regards
Luis
@LuisTheHard, do you have git installed on the system?
Concerning the first error, I would check that git is correctly installed. Otherwise you can go to https://github.com/gem/oq-engine and check how to install the package on your machine.
Closing this. "git" should be installed to fix the first error. Concerning the second, I will try to update to tensor flow > 1.9 soon. That will fix it.