Using keras functional API in model

[kapsl]

Morning,
i tried to create a preprocessor model where a part of the input gets sliced and is put into a dense layer another part is going into a convolutional network.

The problem is, that softlearning uses the sequential api here. Currently I don't see a way how I could slice the input and put it into different network branches. I guess this would require using the functional API. Do you see some workaround, how this could be done? I cannot come up with a solution...

[hartikainen]

I think you're right, and it might be difficult to implement with Sequential API. You should be able to use the functional API with softlearning though. For example, when we create the value functions, we have access to all the input information, which is all you need to build a functional model vs. sequential model:

softlearning/softlearning/value_functions/vanilla.py

Line 32 in 0596f68

inputs = create_inputs(input_shapes)

. Same information should be available for policy as well.

I would actually create the preprocessor in the trainable's setup/build method:

softlearning/examples/development/main.py

Line 54 in 0596f68

variant['Q_params']['config'].update({

. There again, you have the environments available, meaning that you know all the input shapes, so you can create a functional preprocessor and then pass it down to the Q or policy.

Hope that helps! Let me know if any of that is unclear.

[charlesjsun]

I have some experience with this (assuming you are using gym). If you just want your observation to be split between pixels output to a standard feedforward convnet and some other state information to a normal feedforward dense net, say velocity, then softlearning has a built-in way of handling this.

In variants.py, these lines specify a convnet preprocessor:

softlearning/examples/development/variants.py

Lines 531 to 541 in 0596f68

	if is_image_env(universe, domain, task, variant_spec):
	preprocessor_params = {
	'class_name': 'convnet_preprocessor',
	'config': {
	'conv_filters': (64, ) * 3,
	'conv_kernel_sizes': (3, ) * 3,
	'conv_strides': (2, ) * 3,
	'normalization_type': 'layer',
	'downsampling_type': 'conv',
	},
	}

Then in environment params in variants, you have to specify pixel_wrapper_kwargs with pixels_only be set to False, and also specify the observation keys, like this (my own experiment setup):
https://github.com/externalhardrive/mobilemanipulation-tf2/blob/eb16f10844efaebd8c645c85508a46ca61436261/examples/development/variants.py#L390-L393

Specifying a pixel_wrapper_kwargs will cause your environment to be wrapped by a PixelObservationWrapper (https://github.com/openai/gym/blob/master/gym/wrappers/pixel_observation.py). You may have to manually import this inside https://github.com/rail-berkeley/softlearning/blob/master/softlearning/environments/adapters/gym_adapter.py and update to a newer gym version (I haven't made an issue to fix this yet @hartikainen ) because the gym library doesn't actually include this in its __init__.py file for some reason (https://github.com/openai/gym/blob/master/gym/wrappers/__init__.py)

The pixel wrapper expects a render() method inside your environment to give it a image. Also, your environment's observation_space needs to be a spaces.Dict that contains the state observation you need like "velocity", but don't include "pixels", because the wrapper adds it automatically. Something like this: https://github.com/externalhardrive/mobilemanipulation-tf2/blob/eb16f10844efaebd8c645c85508a46ca61436261/softlearning/environments/gym/locobot/nav_grasp_envs.py#L119-L121

Let me know if you want more help with this. There might be more changes you need to make to the code to make this work, I don't remember exactly.

[kapsl]

Ok @hartikainen this works. When doing something like this in the preprocessor:

def mypreprocessor(args):
 def my_model(x):
        x = tfkl.Lambda(cast_and_concat)(x)
        x = tfkl.Reshape([stack_size, obs_size])(x)

        part = x[:, 0:5, :]
        x = tfkl.Concatenate()([x, part])

        return x

return my_model