sintel-dev/Draco

Pipeline failure after `mlprimitives` upgrade to `0.3.1`

sarahmish opened this issue · 1 comments

sarahmish commented

After the new mlprimitives version release, our pipelines that use the keras adapters fails due to the new feature. the new feature in mlprimitives tries to infer the shape of the input and the shape of the output.

There is a subtle problem in the pipeline where y gets converted into an array of shape (.., ) and therefore we get the following error in the keras adapter.

  ERROR    mlblocks.mlpipeline:mlpipeline.py:662 Exception caught fitting MLBlock keras.Sequential.LSTMTimeSeriesClassifier#1
  Traceback (most recent call last):
    File "/home/runner/work/GreenGuard/GreenGuard/.tox/unit/lib/python3.6/site-packages/mlblocks/mlpipeline.py", line 644, in _fit_block
      block.fit(**fit_args)
    File "/home/runner/work/GreenGuard/GreenGuard/.tox/unit/lib/python3.6/site-packages/mlblocks/mlblock.py", line 311, in fit
      getattr(self.instance, self.fit_method)(**fit_kwargs)
    File "/home/runner/work/GreenGuard/GreenGuard/.tox/unit/lib/python3.6/site-packages/mlprimitives/adapters/keras.py", line 99, in fit
      self._augment_hyperparameters(y, 'target', kwargs)
    File "/home/runner/work/GreenGuard/GreenGuard/.tox/unit/lib/python3.6/site-packages/mlprimitives/adapters/keras.py", line 90, in _augment_hyperparameters
      length = shape[0]
  IndexError: tuple index out of range
  ERROR    greenguard.benchmark:benchmark.py:444 Could not score template probability.unstack_lstm_timeseries_classifier 
  Traceback (most recent call last):
    File "/home/runner/work/GreenGuard/GreenGuard/greenguard/benchmark.py", line 436, in evaluate_templates
      cache_path=cache_path
    File "/home/runner/work/GreenGuard/GreenGuard/greenguard/benchmark.py", line 179, in evaluate_template
      pipeline.fit(train, readings)
    File "/home/runner/work/GreenGuard/GreenGuard/greenguard/pipeline.py", line 561, in fit
      start_=start_, output_=output_, **kwargs)
    File "/home/runner/work/GreenGuard/GreenGuard/.tox/unit/lib/python3.6/site-packages/mlblocks/mlpipeline.py", line 802, in fit
      self._fit_block(block, block_name, context, debug_info)
    File "/home/runner/work/GreenGuard/GreenGuard/.tox/unit/lib/python3.6/site-packages/mlblocks/mlpipeline.py", line 644, in _fit_block
      block.fit(**fit_args)
    File "/home/runner/work/GreenGuard/GreenGuard/.tox/unit/lib/python3.6/site-packages/mlblocks/mlblock.py", line 311, in fit
      getattr(self.instance, self.fit_method)(**fit_kwargs)
    File "/home/runner/work/GreenGuard/GreenGuard/.tox/unit/lib/python3.6/site-packages/mlprimitives/adapters/keras.py", line 99, in fit
      self._augment_hyperparameters(y, 'target', kwargs)
    File "/home/runner/work/GreenGuard/GreenGuard/.tox/unit/lib/python3.6/site-packages/mlprimitives/adapters/keras.py", line 90, in _augment_hyperparameters
      length = shape[0]
  IndexError: tuple index out of range