Support for MNIST-360

[Kishaan]

Hi,

I'm trying to incorporate MNIST-360 dataset proposed in https://arxiv.org/pdf/2004.07211.pdf. In short, I need to exclude the class 9 and pair up two consecutive classes ((0,1), (1,2)) in every batch. The digits in the incoming stream would be rotated with increasing values of angles. Can someone help me how can I start implementing this into continuum?

Specifically, I would like to know how to create the scenarios for MNIST-360 from the MNIST dataset in continuum:
from continuum.datasets import MNIST

[arthurdouillard]

I'm not sure to understand this dataset.

• an angle is associated to a pair? Or every pair can be seen in any angle?
• for the pair (0,1), what is the class to predict? Is it multilabel?

[TLESORT]

Hi Kishaan, thanks for your issue and for trying to add a new scenario to continuum! :)

This setting is a bit tricky I need to think a bit how to implement it in the best way in continuum.
But I think you will need to implement a new class in continuum for that.
If I understand correctly, in MNIST-360 the rotation depends not only on the task but also on the class, which is not possible yet in continuum.

One trick to make it possible, could be to create two incremental scenarios and concatenate after batches.
Something like this (but it is very dirty :`) )

continual_dataset = MNIST(.....)
list_rotation_first = [ trsf.rotation(angle * i) for i in range(10) ]
first_digit_scenario = ClassIncremental(continual_dataset, increment=1, transformations=list_rotation_first)

list_rotation_second = [ trsf.rotation(45 +angle * i) for i in range(10) ]
second_digit_scenario = ClassIncremental(continual_dataset, increment=1, transformations=list_rotation_second)


for task_id in range(3*9):
     taskset_1 = first_digit_scenario[task_id%9]
     taskset_2 = second_digit_scenario[1+task_id%9]
     # batch_1 ~ taskset_1
     # batch_2 ~ taskset_2
     # batch = concatenate(batch_1, batch_2 )

Tell me if it helps you :)
I will think of a way to make it cleaner.
Any better idea @arthurdouillard ?

[TLESORT]

@arthurdouillard I think the goal is to predict the label of a given class whatever the rotation. The 1's data stay labelized as one whatever the rotation.
if I understand correctly, this is a normal classification scenario but with some variation to enable better transfer.

[arthurdouillard]

@TLESORT hum, but there are two digits on the image, which one is supposed to be predicted?

[TLESORT]

@arthurdouillard you are speaking about this representation?

I think that each "subplot" is just 9 images together sampled form the same task. So each image has only one label. Or do I miss something?

[arthurdouillard]

Oh ok, so that just RotMNIST with a kind of task sampler...

Well, I think your solution @TLESORT is nice. No need to create a new classes for that I guess. Although, we can add in the doc an explanation how to do it.

[TLESORT]

@Kishaan did you manage to make the proposed implementation run?
Is it doing what is expected?

[Kishaan]

HI,
Sorry for the delay, I was not expecting such a prompt reply. I was going through the implementation from the author's who proposed this dataset. Like you said, it seems a bit tricky to implement it directly. They make batches with pairs of classes at a time, {0,1}, {1,2} ... etc which ends up in 9 pairs. Every class's samples are rotated with increasing rotation spanning an entire 360* angle. They fix a round parameter R (which is set to 3) where these batches of 9 pairs repeat (from what I understood, with increasing rotation angles). In summary, the model sees batches of digit pairs, in which the digits are rotated with gradually increasing angles (the distribution changes gradually).

In the shared image, as @TLESORT pointed out, every subplot contains 9 images, one label for each image, and the rotations in each of this subplot increase gradually. I haven't had time to try out your solution yet. From what I see in your example, the rotation angle you specified was one per class, am I right? In MNIST-360, they do it in per class manner, as in, every sample in a class would be rotated with incrementally increasing angle.

I will post here if I find a way to do this.

[TLESORT]

hmm, then maybe with a supplementary loop in could work.
(something like this)

continual_dataset = MNIST(.....)
for i in range(3):
  start_angle = 120 * i
  angle = 120 / 9
  list_rotation_first = [ trsf.rotation(start_angle + angle * i) for i in range(10) ]
  first_digit_scenario = ClassIncremental(continual_dataset, increment=1, transformations=list_rotation_first)
  
  list_rotation_second = [ trsf.rotation(start_angle  + 45 +angle * i) for i in range(10) ]
  second_digit_scenario = ClassIncremental(continual_dataset, increment=1, transformations=list_rotation_second)
  
  
  for task_id in range(9):
       taskset_1 = first_digit_scenario[task_id]
       taskset_2 = second_digit_scenario[1+task_id%9]
       # batch_1 ~ taskset_1
       # batch_2 ~ taskset_2
       # batch = concatenate(batch_1, batch_2 )

[TLESORT]

Hi, I made an implementation for MNIST-360

    import os
    import torch
    from continuum.scenarios import ClassIncremental
    from torchvision import transforms

    folder = "tests/samples/mnist360/"
    if not os.path.exists(folder):
        os.makedirs(folder)
    continual_dataset = MNIST(data_path=DATA_PATH, download=False, train=True)

    for i in range(3):
        start_angle = 120 * i
        angle = int(120 / 9)

        list_rotation_first = [[transforms.RandomAffine(degrees=[start_angle + angle * j, start_angle + angle * j + 5])]
                               for j in range(10)]
        list_rotation_second = [
            [transforms.RandomAffine(degrees=[start_angle + 45 + angle * j, start_angle + 45 + angle * j + 5])] for j in
            range(10)]
        first_digit_scenario = ClassIncremental(continual_dataset, increment=1, transformations=list_rotation_first)
        second_digit_scenario = ClassIncremental(continual_dataset, increment=1, transformations=list_rotation_second)

        for task_id in range(9):
            taskset_1 = first_digit_scenario[task_id]
            taskset_2 = second_digit_scenario[1 + task_id % 9]

            # / ! \ we can not concatenate taskset here, since transformations would not work correctly

            loader_1 = DataLoader(taskset_1, batch_size=64)
            loader_2 = DataLoader(taskset_2, batch_size=64)

            nb_minibatches = min(len(loader_1), len(loader_2))
            for minibatch in range(nb_minibatches):
                x_1, y_1, t_1 = next(iter(loader_1))
                x_2, y_2, t_2 = next(iter(loader_2))

                x, y, t = torch.cat([x_1, x_2]), torch.cat([y_1, y_2]), torch.cat([t_1, t_2])

                # train here on x, y, t

                #### to visualize result ####
                # from continuum.viz import visualize_batch
                # visualize_batch(batch=x[:100], number=100, shape=[28, 28, 1], path=folder + f"MNIST360_{task_id + 9 * i}.jpg")

I close the issue since it has been inactive since few weeks.
Feel free to reopen it if needed :)

[TLESORT]

This implementation will be added in the documentation.