Add Concatenate layer in nengo_dl

[Louuiissaa]

Hi all,
I want to implement the U-Net architecture in nengo_dl. Therefore, I need to combine two layers in order to give both as an input to the following layer. When I build a Concatenate layer as following, I get a validation error. But I have already checked the type of the TensorNodes and they are both: <class 'nengo_dl.tensor_node.TensorNode'>. Can anyone help me and explain to me what I am doing wrong?

layer_b = nengo_dl.Layer(tf.keras.layers.Conv2DTranspose(
                filters=12, kernel_size=2, strides=2))(layer_a, shape_in=(27, 27, 24))
layer_cropped = nengo_dl.Layer(tf.keras.layers.Cropping2D(
                cropping=((3, 3), (3, 3)), data_format=None))(layer3, shape_in=(60, 60, 12), shape_out=(54, 54, 12))
lconc = nengo_dl.Layer(tf.keras.layers.Concatenate(
                axis=-1))([layer_cropped, layer_b], shape_in=(54, 54, 12), shape_out=(54, 54, 24))

ValidationError: Connection.pre: '[<TensorNode (unlabeled) at 0x7ff9008e2278>, <TensorNode (unlabeled) at 0x7ff90094bf60>]' is not a Nengo object

Thanks!

[drasmuss]

Hi Louuissaa, unfortunately nengo_dl.Layer only works with single-input single-output layers (it's on our TODO list to support multi-input/multi-output).

However, you could write a little function that takes in a single input, splits it into your two inputs, and then concatenates them. Something like:

    def concatenate(x):
        x0 = x[:, :layer_b.size_out]
        x1 = x[:, layer_b.size_out:]
        x0 = tf.keras.layers.Reshape((54, 54, 12))(x0)
        x1 = tf.keras.layers.Reshape(((54, 54, 12))(x1)
        y = tf.keras.layers.Concatenate(axis=-1)([x0, x1])
        return y


    lconc = nengo_dl.TensorNode(
        concatenate,
        shape_in=(layer_b.size_out + layer_cropped.size_out,),
        shape_out=(54, 54, 24),
        pass_time=False,
    )
    nengo.Connection(layer_b, lconc[: layer_b.size_out:], synapse=None)
    nengo.Connection(layer_cropped, lconc[layer_b.size_out :], synapse=None)

Alternatively, you could use the NengoDL Converter, which can automatically convert Concatenate layers to native Nengo objects (see the code here https://github.com/nengo/nengo-dl/blob/master/nengo_dl/converter.py#L1252, which you could also copy if you want a more general concatenate implementation than above).