padding='same' and time dimension not specified

[eayvali]

When I call TCN using padding='same', I get the following error:

output_slice_index = int(self.build_output_shape.as_list()[1] / 2) if self.padding == 'same' else -1
AttributeError: 'tuple' object has no attribute 'as_list'

Any insight?

Also the ReadMe says one can use padding='valid' or 'same', but using 'valid' throws ValueError

Thank you!

[AlbertoSabater]

Hello! I also have an error on that line of code when working with padding='same' and undefined timesteps.

This is the pice of code to replicate this error:

i = Input(batch_shape=(None, None, 300))
o = TCN(return_sequences=False, padding='same')(i)  # The TCN layers are here.
o = Dense(1)(o)
m = Model(inputs=[i], outputs=[o])
m.compile(optimizer='adam', loss='mse')
pred = m(np.random.rand(1,5,300))

And the error:

Traceback (most recent call last):

  File "<ipython-input-18-bce9d99d462e>", line 2, in <module>
    o = TCN(return_sequences=False, padding='same')(i)  # The TCN layers are here.

  File "/home/asabater/anaconda3/envs/core/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 897, in __call__
    self._maybe_build(inputs)

  File "/home/asabater/anaconda3/envs/core/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 2416, in _maybe_build
    self.build(input_shapes)  # pylint:disable=not-callable

  File "/home/asabater/anaconda3/envs/core/lib/python3.6/site-packages/tcn/tcn.py", line 272, in build
    output_slice_index = int(self.build_output_shape.as_list()[1] / 2) if self.padding == 'same' else -1

TypeError: unsupported operand type(s) for /: 'NoneType' and 'int'

Same code but with causal convolutions works fine:

i = Input(batch_shape=(None, None, 300))
o = TCN(return_sequences=False, padding='causal')(i)  # The TCN layers are here.
o = Dense(1)(o)
m = Model(inputs=[i], outputs=[o])
m.compile(optimizer='adam', loss='mse')
pred = m(np.random.rand(1,5,300))

I wonder what these lines of code do in the TCN model:

output_slice_index = int(self.build_output_shape.as_list()[1] / 2) if self.padding == 'same' else -1
self.lambda_layer = Lambda(lambda tt: tt[:, output_slice_index, :])
self.lambda_ouput_shape = self.lambda_layer.compute_output_shape(self.build_output_shape)

I am not sure if specifying the timesteps before building the model is a requirement for 'same' convolutions. Any insight or fix of this issue will be welcomed.

[philipperemy]

@AlbertoSabater those steps were introduced for the non causal TCN:

https://github.com/philipperemy/keras-tcn#non-causal-tcn.

The idea is that you take the middle instead of the last value on the right when you're not working in a causal setting.

It's possible that a bug got introduced recently since this non causal mode ("same" mode) is not very much used.

[philipperemy]

I'll look into it. Are you guys running the latest tensorflow? >2.1 and the latest Keras? >2.2.4
I know they changed a lot in those versions and this could explain why we have those problems happening here.

[AlbertoSabater]

Thank you for your answer! I understand the current implementation but it does not handle multiple length inputs.

In order to handle multiple length inputs, one solution could be to output the value that is calculated with the last part of the sequence, no matter its length. For instance, if we have a receptive field of 5, the predicted value on time t, depends on the inputs values corresponding to t-2 ... t ... t+2 so, the output value for any sequence with size T would be the one at time T-3.
I hope I have made myself clear.

Btw, I am using tensorflow-gpu 2.2.0 and keras-tcn 3.1.0.

[philipperemy]

#140
So first, I added a check but I'm going to see what we can do in the meantime.
The length of the sequence is unknown at build time and that's the tricky part.
So I guess like what you said we could have T-(receptive_field/2+1).

[AlbertoSabater]

Perfect! Thank you

[philipperemy]

So at build time we don't have any information so we can't really do it here.

But during the call time we can hack it by fetching the value of the tensor and perform the x/2 operation.

def call(self, inputs, training=None):
...
        if not self.return_sequences:
            output_slice_index = int(K.shape(self.layers_outputs[-1])[1] / 2)
            self.lambda_layer = Lambda(lambda tt: tt[:, output_slice_index, :])
            self.lambda_ouput_shape = self.lambda_layer.compute_output_shape(self.build_output_shape)
            x = self.lambda_layer(x)
            self.layers_outputs.append(x)

The call self.layers_outputs[-1].shape.as_list()[1]works too

[philipperemy]

I can recover the value 2 with your example when the sequence is of length 5.

[philipperemy]

I'm just concerned by this function that will not be correct in this case:

(But I've never had any cases that used this function in practice)...

    def compute_output_shape(self, input_shape):
        """
        Overridden in case keras uses it somewhere... no idea. Just trying to avoid future errors.
        """
        if not self.built:
            self.build(input_shape)
        if not self.return_sequences:
            return self.lambda_ouput_shape
        else:
            return self.build_output_shape

[philipperemy]

cc @eayvali @AlbertoSabater
TL;DR: We are close to a solution for the case where the time dimension is not specified (None).

[philipperemy]

Ok so there's a pretty huge pull request here: #143
With a lot of testing. I'll merge it once it passes the CI.

[philipperemy]

Version 3.1.1 with the changes of the PR mentioned above (merged).
It should be working well now!
Let me know if it's not clear. I'll close this PR in the meantime.