Multiple modalities (image channels)

[jaysnanavati]

Hello, from the APIs it is not clear if/how multiple image channels are supported. Given that the loader takes all the images and stacks them in a skyscraper, how do we work with multiple image channels for training and inference?

[jaysnanavati]

Ping!

[roman-kh]

@akoryagin, do you have any suggestions?

[jaysnanavati]

Ping! It would be cool to get more insights and pointers into this, I am opening to contributing :)

[akoryagin]

Hey, @jaysnanavati!

Thank you very much for the issue and your interest!

True, RadIO isn't originally built for supporting several image-channels. However, there is a workaround. The idea is to load channels one by one in images-component, perform radio-actions on it, and store the preprocessed channel in pipeline-variable. In the end, the channels can be easily stacked together in another pipeline-variable and used for model-training. This will look like

# assuming you store several channels of each input image in `dicom`
# the files are stored as follows: 
# data_dir/file_1/channel_1.dcm`, `data_dir/file_1/channel_2.dcm; 
# data_dir/file_2/channel_1.dcm`, `data_dir/file_2/channel_2.dcm, ...

fix = FilesIndex('data_dir/*', dirs=True)
ds = Dataset(fix, CTImagesMaskedBatch)
pipeline = (Pipeline()
            .init_variables(['channel_1', 'channel_2', 'full_image'])
            .load(fmt='dicom', src='channel_1.dcm') # an op that loads first channel
            .some_action(...)        # an op that might change images-component
            .update_variable('channel_1', B('images'))        # store `images`-comp in pipeline-variable
            .load(fmt='dicom', src='channel_2.dcm')  # an op that loads second channel
            .some_action(...)        # change second-channel in the same way
            .update_variable('channel_2', B('images'))
            .update_variable('full_image',
                             L(lambda im1, im2: np.stack([im_1, im_2], axis=-1),
                             V('images_1'), V('images_2')))    # stack 2 channels in pipeline-variable
            ... # ops with model-init
            .train_model('model', feed_dict={'images': V('full_image'), # use the stacked image for training
                                             'masks':...})) 

You might benefit from reading about V, B, L and other pipeline-names expressions here.

Best,
Alex!

[jaysnanavati]

Thanks @akoryagin this looks promising, I will give it a shot! :)