Error when using Exchange_rate data
asollish85 opened this issue · 1 comments
Hi Ehsan. I was hoping to get your insight on an error I'm getting. I'm not so sure it's a bug as much as something I might be doing wrong.
I swapped out the 'electricity' data with the GluonTS provided 'exchange_rate' data. Everything works fine up to the point of testing the endpoint. I get the below error message, which I'm having trouble understanding. I'm very new to MXNet, BTW.
The following Model error seems to be the issue:
Check failed: dshape[axes[i]] == 1 (314 vs. 1) : cannot select an axis to squeeze out which has size=314 not equal to one
Anyway, I was curious if you have any ideas? Any help would be greatly appreciated!
- Andy
Original Error Message:
`ModelError Traceback (most recent call last)
in
11 random_test_data['source'] = []
12
---> 13 random_ret = predictor.predict(random_test_data)
~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/sagemaker/predictor.py in predict(self, data, initial_args, target_model, target_variant)
123
124 request_args = self._create_request_args(data, initial_args, target_model, target_variant)
--> 125 response = self.sagemaker_session.sagemaker_runtime_client.invoke_endpoint(**request_args)
126 return self._handle_response(response)
127
~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
355 "%s() only accepts keyword arguments." % py_operation_name)
356 # The "self" in this scope is referring to the BaseClient.
--> 357 return self._make_api_call(operation_name, kwargs)
358
359 _api_call.name = str(py_operation_name)
~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
674 error_code = parsed_response.get("Error", {}).get("Code")
675 error_class = self.exceptions.from_code(error_code)
--> 676 raise error_class(parsed_response, operation_name)
677 else:
678 return parsed_response
ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (500) from model with message "[15:48:53] src/operator/tensor/./matrix_op-inl.h:2359: Check failed: dshape[axes[i]] == 1 (314 vs. 1) : cannot select an axis to squeeze out which has size=314 not equal to one
Stack trace:
[bt] (0) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x3172a2) [0x7fcf408742a2]
[bt] (1) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x1ad2e0a) [0x7fcf4202fe0a]
[bt] (2) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::imperative::SetShapeType(mxnet::Context const&, nnvm::NodeAttrs const&, std::vector<mxnet::NDArray*, std::allocatormxnet::NDArray* > const&, std::vector<mxnet::NDArray*, std::allocatormxnet::NDArray* > const&, mxnet::DispatchMode*)+0x22d5) [0x7fcf43bb2b05]
[bt] (3) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::Imperative::Invoke(mxnet::Context const&, nnvm::NodeAttrs const&, std::vector<mxnet::NDArray*, std::allocatormxnet::NDArray* > const&, std::vector<mxnet::NDArray*, std::allocatormxnet::NDArray* > const&)+0x35c) [0x7fcf43bc8e0c]
[bt] (4) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x3d84d2b) [0x7fcf442e1d2b]
[bt] (5) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(MXImperativeInvokeEx+0x5b) [0x7fcf442e30bb]
[bt] (6) /usr/local/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(ffi_call_unix64+0x4c) [0x7fcf50ccd4e6]
[bt] (7) /usr/local/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(ffi_call+0x3f1) [0x7fcf50ccc241]
[bt] (8) /usr/local/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(_ctypes_callproc+0x2cf) [0x7fcf50cc38ff]
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sagemaker_inference/transformer.py", line 126, in transform
result = self._transform_fn(self._model, input_data, content_type, accept)
File "/opt/ml/model/code/inference.py", line 41, in transform_fn
forecasts, tss, agg_metrics, _ = evaluate(model, request_list_data, num_samples=1)
File "/opt/ml/model/code/utils.py", line 26, in evaluate
forecasts = list(forecast_it)
File "/usr/local/lib/python3.6/site-packages/gluonts/model/predictor.py", line 330, in predict
num_samples=num_samples,
File "/usr/local/lib/python3.6/site-packages/gluonts/model/forecast_generator.py", line 197, in call
outputs = prediction_net(inputs).asnumpy()
File "/usr/local/lib/python3.6/site-packages/mxnet/gluon/block.py", line 758, in call
out = self.forward(args)
File "/usr/local/lib/python3.6/site-packages/mxnet/gluon/block.py", line 1223, in forward
return self.hybrid_forward(ndarray, x, args, params)
File "/usr/local/lib/python3.6/site-packages/gluonts/model/lstnet/_network.py", line 355, in hybrid_forward
F, past_target, past_observed_values
File "/usr/local/lib/python3.6/site-packages/gluonts/model/lstnet/_network.py", line 234, in hybrid_forward
c = F.squeeze(c, axis=2) # NCT
File "", line 44, in squeeze
File "/usr/local/lib/python3.6/site-packages/mxnet/_ctypes/ndarray.py", line 107, in _imperative_invoke
ctypes.byref(out_stypes)))
File "/usr/local/lib/python3.6/site-packages/mxnet/base.py", line 255, in check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [15:48:53] src/operator/tensor/./matrix_op-inl.h:2359: Check failed: dshape[axes[i]] == 1 (314 vs. 1) : cannot select an axis to squeeze out which has size=314 not equal to one
Stack trace:
[bt] (0) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x3172a2) [0x7fcf408742a2]
[bt] (1) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x1ad2e0a) [0x7fcf4202fe0a]
[bt] (2) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::imperative::SetShapeType(mxnet::Context const&, nnvm::NodeAttrs const&, std::vector<mxnet::NDArray, std::allocatormxnet::NDArray* > const&, std::vector<mxnet::NDArray, std::allocatormxnet::NDArray* > const&, mxnet::DispatchMode)+0x22d5) [0x7fcf43bb2b05]
[bt] (3) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::Imperative::Invoke(mxnet::Context const&, nnvm::NodeAttrs const&, std::vector<mxnet::NDArray, std::allocatormxnet::NDArray* > const&, std::vector<mxnet::NDArray, std::allocatormxnet::NDArray* > const&)+0x35c) [0x7fcf43bc8e0c]
[bt] (4) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x3d84d2b) [0x7fcf442e1d2b]
[bt] (5) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(MXImperativeInvokeEx+0x5b) [0x7fcf442e30bb]
[bt] (6) /usr/local/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(ffi_call_unix64+0x4c) [0x7fcf50ccd4e6]
[bt] (7) /usr/local/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(ffi_call+0x3f1) [0x7fcf50ccc241]
[bt] (8) /usr/local/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(_ctypes_callproc+0x2cf) [0x7fcf50cc38ff]
". See https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#logEventViewer:group=/aws/sagemaker/Endpoints/sagemaker-soln-ddf-demand-endpoint in account 940412083416 for more information.`
Hi Ehsan. I was hoping to get your insight on an error I'm getting. I'm not so sure it's a bug as much as something I might be doing wrong.
I swapped out the 'electricity' data with the GluonTS provided 'exchange_rate' data. Everything works fine up to the point of testing the endpoint. I get the below error message, which I'm having trouble understanding. I'm very new to MXNet, BTW.
The following Model error seems to be the issue:
Check failed: dshape[axes[i]] == 1 (314 vs. 1) : cannot select an axis to squeeze out which has size=314 not equal to oneAnyway, I was curious if you have any ideas? Any help would be greatly appreciated!
* AndyOriginal Error Message:
`ModelError Traceback (most recent call last)
in
11 random_test_data['source'] = []
12
---> 13 random_ret = predictor.predict(random_test_data)~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/sagemaker/predictor.py in predict(self, data, initial_args, target_model, target_variant)
123
124 request_args = self._create_request_args(data, initial_args, target_model, target_variant)
--> 125 response = self.sagemaker_session.sagemaker_runtime_client.invoke_endpoint(**request_args)
126 return self._handle_response(response)
127~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
355 "%s() only accepts keyword arguments." % py_operation_name)
356 # The "self" in this scope is referring to the BaseClient.
--> 357 return self._make_api_call(operation_name, kwargs)
358
359 _api_call.name = str(py_operation_name)~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
674 error_code = parsed_response.get("Error", {}).get("Code")
675 error_class = self.exceptions.from_code(error_code)
--> 676 raise error_class(parsed_response, operation_name)
677 else:
678 return parsed_responseModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (500) from model with message "[15:48:53] src/operator/tensor/./matrix_op-inl.h:2359: Check failed: dshape[axes[i]] == 1 (314 vs. 1) : cannot select an axis to squeeze out which has size=314 not equal to one
Stack trace:
[bt] (0) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x3172a2) [0x7fcf408742a2]
[bt] (1) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x1ad2e0a) [0x7fcf4202fe0a]
[bt] (2) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::imperative::SetShapeType(mxnet::Context const&, nnvm::NodeAttrs const&, std::vector<mxnet::NDArray*, std::allocatormxnet::NDArray* > const&, std::vector<mxnet::NDArray*, std::allocatormxnet::NDArray* > const&, mxnet::DispatchMode*)+0x22d5) [0x7fcf43bb2b05]
[bt] (3) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::Imperative::Invoke(mxnet::Context const&, nnvm::NodeAttrs const&, std::vector<mxnet::NDArray*, std::allocatormxnet::NDArray* > const&, std::vector<mxnet::NDArray*, std::allocatormxnet::NDArray* > const&)+0x35c) [0x7fcf43bc8e0c]
[bt] (4) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x3d84d2b) [0x7fcf442e1d2b]
[bt] (5) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(MXImperativeInvokeEx+0x5b) [0x7fcf442e30bb]
[bt] (6) /usr/local/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(ffi_call_unix64+0x4c) [0x7fcf50ccd4e6]
[bt] (7) /usr/local/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(ffi_call+0x3f1) [0x7fcf50ccc241]
[bt] (8) /usr/local/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(_ctypes_callproc+0x2cf) [0x7fcf50cc38ff]Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sagemaker_inference/transformer.py", line 126, in transform
result = self._transform_fn(self._model, input_data, content_type, accept)
File "/opt/ml/model/code/inference.py", line 41, in transform_fn
forecasts, tss, agg_metrics, _ = evaluate(model, request_list_data, num_samples=1)
File "/opt/ml/model/code/utils.py", line 26, in evaluate
forecasts = list(forecast_it)
File "/usr/local/lib/python3.6/site-packages/gluonts/model/predictor.py", line 330, in predict
num_samples=num_samples,
File "/usr/local/lib/python3.6/site-packages/gluonts/model/forecast_generator.py", line 197, in call
outputs = prediction_net(_inputs).asnumpy() File "/usr/local/lib/python3.6/site-packages/mxnet/gluon/block.py", line 758, in call out = self.forward(_args) File "/usr/local/lib/python3.6/site-packages/mxnet/gluon/block.py", line 1223, in forward return self.hybrid_forward(ndarray, x, _args, __params) File "/usr/local/lib/python3.6/site-packages/gluonts/model/lstnet/network.py", line 355, in hybrid_forward F, past_target, past_observed_values File "/usr/local/lib/python3.6/site-packages/gluonts/model/lstnet/network.py", line 234, in hybrid_forward c = F.squeeze(c, axis=2) # NCT File "", line 44, in squeeze File "/usr/local/lib/python3.6/site-packages/mxnet/ctypes/ndarray.py", line 107, in imperative_invoke ctypes.byref(out_stypes))) File "/usr/local/lib/python3.6/site-packages/mxnet/base.py", line 255, in check_call raise MXNetError(py_str(LIB.MXGetLastError())) mxnet.base.MXNetError: [15:48:53] src/operator/tensor/./matrix_op-inl.h:2359: Check failed: dshape[axes[i]] == 1 (314 vs. 1) : cannot select an axis to squeeze out which has size=314 not equal to one Stack trace: [bt] (0) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x3172a2) [0x7fcf408742a2] [bt] (1) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x1ad2e0a) [0x7fcf4202fe0a] [bt] (2) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::imperative::SetShapeType(mxnet::Context const&, nnvm::NodeAttrs const&, std::vector<mxnet::NDArray, std::allocatormxnet::NDArray* > const&, std::vector<mxnet::NDArray, std::allocatormxnet::NDArray* > const&, mxnet::DispatchMode)+0x22d5) [0x7fcf43bb2b05] [bt] (3) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::Imperative::Invoke(mxnet::Context const&, nnvm::NodeAttrs const&, std::vector<mxnet::NDArray, std::allocatormxnet::NDArray* > const&, std::vector<mxnet::NDArray, std::allocatormxnet::NDArray* > const&)+0x35c) [0x7fcf43bc8e0c]
[bt] (4) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x3d84d2b) [0x7fcf442e1d2b]
[bt] (5) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(MXImperativeInvokeEx+0x5b) [0x7fcf442e30bb]
[bt] (6) /usr/local/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(ffi_call_unix64+0x4c) [0x7fcf50ccd4e6]
[bt] (7) /usr/local/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(ffi_call+0x3f1) [0x7fcf50ccc241]
[bt] (8) /usr/local/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(_ctypes_callproc+0x2cf) [0x7fcf50cc38ff]". See https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#logEventViewer:group=/aws/sagemaker/Endpoints/sagemaker-soln-ddf-demand-endpoint in account 940412083416 for more information.`
I realized the issue was the value for 'NUM_TS'. I didn't have it matched to the value of the processed data for the exchange_rate dataset.