ImageDataGenerator with masks as labels
Closed this issue ยท 60 comments
@fchollet
We know that ImageDataGenerator provides a way for image data augmentation: ImageDataGenerator.flow(X, Y). Now consider the image segmentation task where Y is not a categorical label but a image mask which is the same size as input X, e.g. 256x256 pixels. If we would like to use data augmentation, the same transformation should also be adopted to Y. Is there any simple way to handle this?
@pengpaiSH I don't know if this would work, but maybe its enough to do it like this:
datagen = ImageDataGenerator( rotation_range=4) and then you could use
for batch in datagen.flow(x, batch_size=1,seed=1337 ): with random seed and use datagen.flow once on X and then on the mask y and save the batches. This should do the same rotations on both X and y, but maybe dont work with ZCA and other normalizations.
@MayorPain Thank you for your response. In your proposed solution, you set batch_size=1. I don't understand why X and y will be transformed simultaneously?
@pengpaiSH have a look at this https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html
My thoughts were you fix the transformations with seed and then apply the same generation on the masks. So not simiultanesously but first the actual image and then the mask
@MayorPain Thanks for your reference. Is batch_size = 1 necessary? And,
for batch in datagen.flow(x, batch_size=1, seed=1337):
# batch...
In this way, you could get batch one by one which is the augmented x. Then how to apply on the y?
@pengpaiSH i dont know if you can use batchsize > 1. If you look at the example on the page, they treat x as 1 image. Ok they did this just for plotting reasons but i dont know if this would also work for batchsizes > 1. I would Loop over my imageset and set x to the current image, then apply the imagegenerator like in the example and then use the same generator again on the mask like
mask=[] img=[]
for batch in datagen.flow(x, batch_size=1, seed=1337): img.append(batch)
for batch in datagen.flow(ymask, batch_size=1, seed=1337): mask.append(batch)
@MayorPain Really thank you for your detailed comments. I think your idea is right, looks in the right direction. If we are appending each image or mask, then we should set batch_size=1.
@MayorPain I have tried this idea and it works! Thank you again! By the way, it seems that @oeway is trying to extend ImageGenerator to support more flexibilities.
Yes, you could try my fork(branch: extendImageDataGenerator), for now you can do:
train_generator = dataGen1+dataGen2
model.fit_generator(train_generator)Suggestions would be appreciated.
@oeway Thank you for your contribution for extending current ImageDataGenerator!
For reference, here's another extension by He Xie (HEXIE). It should be useful when this enhancement is finalised and added to the unit tests.
I see, so he added y directly to random_transformation, it will work but will less likely to be generalized into a customizable preprocessing pipeline. For my extension, I used a separate ImageDataGenerator for X and y, so the pipeline can be easily extended with your own functions.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.
If anyone else gets here from search, the new answer is that you can do this with the imagedatagenerator
From Docs
Example of transforming images and masks together.
we create two instances with the same arguments
data_gen_args = dict(featurewise_center=True,
featurewise_std_normalization=True,
rotation_range=90.,
width_shift_range=0.1,
height_shift_range=0.1,
zoom_range=0.2)
image_datagen = ImageDataGenerator(**data_gen_args)
mask_datagen = ImageDataGenerator(**data_gen_args)Provide the same seed and keyword arguments to the fit and flow methods
seed = 1
image_datagen.fit(images, augment=True, seed=seed)
mask_datagen.fit(masks, augment=True, seed=seed)image_generator = image_datagen.flow_from_directory(
'data/images',
class_mode=None,
seed=seed)mask_generator = mask_datagen.flow_from_directory(
'data/masks',
class_mode=None,
seed=seed)combine generators into one which yields image and masks
train_generator = zip(image_generator, mask_generator)
model.fit_generator(
train_generator,
steps_per_epoch=2000,
epochs=50)
Hi @peachthiefmedia ! I have two questions:
- What would be the "images" and "masks" arguments in the lines
image_datagen.fit(images, augment=True, seed=seed)
mask_datagen.fit(masks, augment=True, seed=seed)
from the code of your last comment?
And, 2)I've tried flow_from_directory() with images of type '.jpg', giving the folder containing them as parameter of the function, i. e., image_datagen.flow_from_directory(folder_of_jpg_images,target_size,class_mode='categorical')``
But I get the Output: "Found 0 images belonging to 0 classes"
I've double checked the folder and the image type is correct. What could be the problem?
Thanks in advance!
-
Ah for the first question that is if you are using the model.fit, not the generator I think, if your flowing from a directory you don't need it.
-
The images and masks should be the directories for the images and masks respectively. One thing to keep in mind is that the flow from directory expects to be one directory above the files i.e.
data/images/where_they_are/
data/masks/where_they_are/
It's expecting that you will give it the data/images and data/masks directories, not the /images/where_they_are . Flow from directory is really built around mulitclass problems, but still works if you only have one folder in there (single class).
@ peachthiefmedia
Did you try image segmentation with above code snippet?
I want to try just want confirmation
@chaitanya1chaitanya Mine was slightly different but based on that, I used
data_gen_args = dict(width_shift_range=0.2, height_shift_range=0.2)
image_datagen = ImageDataGenerator(**data_gen_args)
mask_datagen = ImageDataGenerator(**data_gen_args)
seed = 1
image_generator = image_datagen.flow_from_directory('raw/train_image', target_size=(img_rows,img_cols),
class_mode=None, seed=seed, batch_size=batchsize, color_mode='grayscale')
mask_generator = mask_datagen.flow_from_directory('raw/train_mask', target_size=(img_rows,img_cols),
class_mode=None, seed=seed, batch_size=batchsize, color_mode='grayscale')
train_generator = zip(image_generator, mask_generator)
model.fit_generator(train_generator, steps_per_epoch=5635/batchsize, epochs=100, verbose=1)
Some of that was specific to my segmentation, but it should give you the idea. I use my own generators now for the most part.
@peachthiefmedia
i)For path in flow from directory, i gave the directory as path which contains two subdirectories one named images contains images and other named masks contain all masks.
In this case while executing, I'm getting ,found 1440 images belonging to 2 classes for image generator
and found 1440 images belonging to 2 classes for mask generator.
while data/images contains 720 images.
and data/masks contain 720 ground truth images.
but the generators in both cases stating 1440 images belonging to 2 classes.
How it is working(flow_from_directory)? The problem is image segmentation,720 training images and its masks.
ii)if i give path for image generator as data/images instead of data/ and for mask generator data/masks instead of data/,its showing found 0 images in 0 classes for both cases.
For classification problem its fine,for segmentation problem how to give path correctly? if path to be given as in case(i),then why its showing double the no. of images for both generators.
finally,can we use Imagedatagenerator for segmentation problem?
im the same point as you @chaitanya1chaitanya , the exact same mistake and the same thoughts
@chaitanya1chaitanya @ixoneioseba You need to use the following structure, the example is with 2 image but works with however many
/data/images/0/image1.jpg
/data/images/0/image2.jpg
/data/masks/0/image1.jpg
/data/masks/0/image1.jpg
Then you would use /data/images as your image directory and /data/masks for your mask directory in the generator, i.e.
image_datagen.flow_from_directory('data/images' ...
mask_datagen.flow_from_directory('data/masks' ...
You need to have a directory in the directory because it is built for classification really, so it expects that you have more than one label set.
The generator works well, however when fed to a neural network that does segmentation, it gives the error : "Error when checking target: expected activation_layer(final softmax) to have 3 dimensions, but got array with shape (32, 360, 480, 3)".
I tried both with SegNet and FCNN from: https://github.com/divamgupta/image-segmentation-keras/tree/master/Models and I get the same error.
How is it possible to solve such an error? I've been struggling for two weeks on this issue and no one has been able to help me(I thought it was an architectural problem, but it gives the same error for both neural network architectures.
The code for image_segmentation is the following:
def applyImageAugmentationAndRetrieveGenerator():
from keras.preprocessing.image import ImageDataGenerator
# We create two instances with the same arguments
data_gen_args = dict(rotation_range=90.,
width_shift_range=0.1,
height_shift_range=0.1,
zoom_range=0.2
)
image_datagen = ImageDataGenerator(**data_gen_args)
mask_datagen = ImageDataGenerator(**data_gen_args)
# Provide the same seed and keyword arguments to the fit and flow methods
seed = 1
image_generator = image_datagen.flow_from_directory('dataset/train_images',
target_size=(360,480),
class_mode=None,
seed=seed,
batch_size = 32)
mask_generator = mask_datagen.flow_from_directory('dataset/train_masks',
target_size=(360,480),
class_mode=None,
seed=seed,
batch_size = 32)
#Combine generators into one which yields image and masks
#print(image_generator[0])
#print(mask_generator[0])
train_generator = zip(image_generator, mask_generator)
return train_generator
#TRYING TO FIT_GENERATOR THROWS THIS ERROR(BOTH FOR FCNN AND SEGNET)
** "Error when checking target: expected activation_layer(final softmax) to have 3 dimensions, but got array with shape (32, Height, Width, 3)"**
segmentation_model.fit_generator(training_generator,
steps_per_epoch=186,
epochs=50,
verbose=1
)
it's because the loss function is expecting the masks to be 2d arrays, and the image generator is reading 3d rgb arrays.
@peachthiefmedia so many thanks for your solution. I'm trying to make it work and the process takes a lot of RAM memory (from CPU not GPU) in the zip step (more than 16Gb for 1000 512x512 images) and takes a lot of time to work, do you think this is normal? Somebody have an idea to reduce this RAM memory consumption?
Will this work if batch_size is greater than 1 ???
Hi the data generator changes the truth/pixel/array values of the mask/image if it is being rotated or sheared or shifted (horizontally or vertically). Was this taken into account while creating this method?
@amlarraz Yes I noticed it was taking a large amount of ram to do, I think your zipping float arrays by this point so that's why it is so large. If that's the case you can probably do them as ints by changing things around. Unfortunately I don't use this method for segmentation and have written my own generator instead, but another option which I was using was to output a large set first to numpy arrays/hdf5 and then just use that instead when actually training. Depends how fast your GPU is, with mine training was slower than the images so it didn't matter too much.
@RishalAggarwal Yes as long as they both have the same seed then they should do the same thing.
@jayshah19949596 It was working for me for any batch sizes, I didn't check if there was multiple workers for the generators but I assume it was fine.
@pengpaiSH
model.fit_generator(
train_generator,
steps_per_epoch=2000,
epochs=2) not working...plz hlp me
@peachthiefmedia thanks for this solution
/data/images/0/image1.jpg
/data/images/0/image2.jpg
/data/masks/0/image1.jpg
/data/masks/0/image1.jpg
it worked :) ๐
I'm running the code for segmentation on HPC GPU cluster. I have problems with the "zip" step. Practically it takes eternity to finish the zip. Is there any way around it? @peachthiefmedia @amlarraz I'd appreciate if you help me .
Hey @aliechoes ! I solved this problem by the creation of my own datagenerator for images and masks simultaneously. Here you have the code I used:
def train_generator(img_dir, label_dir, batch_size, input_size):
list_images = os.listdir(img_dir)
shuffle(list_images) #Randomize the choice of batches
ids_train_split = range(len(list_images))
while True:
for start in range(0, len(ids_train_split), batch_size):
x_batch = []
y_batch = []
end = min(start + batch_size, len(ids_train_split))
ids_train_batch = ids_train_split[start:end]
for id in ids_train_batch:
img = cv2.imread(os.path.join(img_dir, list_images[id]))
img = cv2.resize(img, (input_size[0], input_size[1]))
mask = cv2.imread(os.path.join(label_dir, list_images[id].replace('jpg', 'png')), 0)
mask = cv2.resize(mask, (input_size[0], input_size[1]))
mask = np.expand_dims(mask, axis=2)
x_batch.append(img)
y_batch.append(mask)
x_batch = np.array(x_batch, np.float32) / 255.
y_batch = np.array(y_batch, np.float32)
yield x_batch, y_batch
Note I have not used image augmentation but the code resize the images to feed the network.
@amlarraz : thanks. It works. However, I start having a problem with the fit generator. it goes file by file
Epoch 1/10
5/1788 [..............................] - ETA: 6:51:51 - loss: 0.9206 - dice_coeff: 0.7141
Epoch 1/10
6/1788 [..............................] - ETA: 6:41:35 - loss: 0.8679 - dice_coeff: 0.7291
However I chose the batch_size to be 64 for example. Did you have the same issue? Am I making a mistake? Thanks
Hey @aliechoes, how many images do you have for train? I think you're seeing the number of iterations, not the images. Each iteration is one batch, I mean, if you have 6 images with a batch size of 3, you only have 2 steps. Please check how many images do you have. Happy to help!
@amlarraz : oh yeah. I totally missed the difference between fit_generator and fit. Thanks a lot :)
I have 40k images.
Without the hassle of organizing the folders for the case that one map matches on one mask as the answer above, we can use .flow()
my working code:
`def augmentationForTrainImageAndMask(imgs, masks):
data_gen_args = dict(rotation_range=40.,
width_shift_range=0.1,
height_shift_range=0.1,
zoom_range=0.2,
horizontal_flip=True,
fill_mode='nearest'
)
image_datagen = ImageDataGenerator(**data_gen_args)
mask_datagen = ImageDataGenerator(**data_gen_args)
seed = 1
image_datagen.fit(imgs, augment=True, seed=seed)
mask_datagen.fit(masks, augment=True, seed=seed)
image_generator = image_datagen.flow(imgs,
seed=seed,
batch_size=batch_size,
shuffle=False)
mask_generator = mask_datagen.flow(masks,
seed=seed,
batch_size=batch_size,
shuffle=False)
return zip(image_generator, mask_generator)`
note: make sure you don't have other np random seed generators in the code before this function.
for those who don't get a GPU and get stuck with zip(), please read this:
- using loop and yield to combine two generators:
https://github.com/keras-team/keras/issues/5720
- python2's user might need this:
https://stackoverflow.com/questions/20910213/loop-over-two-generator-together/20910294
other good examples: - Split train data into training and validation when using ImageDataGenerator: https://github.com/keras-team/keras/issues/5862
- using fit() or do it from scratch: https://spark-in.me/post/unet-adventures-part-one-getting-acquainted-with-unet
- Example with Unet(worked!): https://www.kaggle.com/takuok/keras-generator-starter-lb-0-326
Hi, @peachthiefmedia , I have a question that In a multi-class segmentation, masks have multiple colors, and their each pixel need to be converted to a one-hot vector or a integer label. How to use the API of ImageDataGenerator to augment mask correctly?
As far as i am concerned, there is two ways, one is that we do the convert after augmentation, but the number of masks will ba huge; the other is that we do the convert before augmentation, but I am worried that some methods will bring errors๏ผsuch as zoom_range.
Is there a better solution for this problem? Thanks for any suggestions.
@JianyingLi I have always been directly outputting the masks themselves for segmentation, but I've only done up to 3 classes at a time for it so its fine outputting R,G,B as the effective integer 0,1,2 class number, I find it gets progressively more difficult to segment more classes so I normally split the training into single class only at the moment and then run all the models on the image and use some image based work afterwards to get the final segmentation.
If you have a high number of classes I guess I would augment the rgb base masks first and then loop through them to get the labels, but it would be slow I'd say.
Mask_RCNN does high number class segmentation, so it might be worth looking through how they have done it.
@JianyingLi I have always been directly outputting the masks themselves for segmentation, but I've only done up to 3 classes at a time for it so its fine outputting R,G,B as the effective integer 0,1,2 class number, I find it gets progressively more difficult to segment more classes so I normally split the training into single class only at the moment and then run all the models on the image and use some image based work afterwards to get the final segmentation.
If you have a high number of classes I guess I would augment the rgb base masks first and then loop through them to get the labels, but it would be slow I'd say.
Mask_RCNN does high number class segmentation, so it might be worth looking through how they have done it.
I will try these methods. Thanks for your reply. :)
@sammilei Hi, I did something pretty similar (identical) to your code but my images and masks (that I save through "save_to_dir" option) do not seem to match.
Moreover, masks are ok but images are saved as totally black (I have rgb images).
def train_and_predict():
imgs_train, imgs_mask_train, _ = load_train_data()
imgs_train = imgs_train.astype('float32')
mean = np.mean(imgs_train)
std = np.std(imgs_train)
imgs_train -= mean
imgs_train /= std
imgs_mask_train = imgs_mask_train.astype('float32')
imgs_mask_train /= 255. # scale masks to [0, 1]
image_datagen = ImageDataGenerator(
#featurewise_center=True,
#featurewise_std_normalization=True,
#rotation_range=90.,
#width_shift_range=0.1,
#height_shift_range=0.1,
#zoom_range=0.2,
#rescale=1./255,
validation_split=0.2,
horizontal_flip=1,
vertical_flip=1)
mask_datagen = ImageDataGenerator(
#featurewise_center=True,
#featurewise_std_normalization=True,
#rotation_range=90.,
#width_shift_range=0.1,
#height_shift_range=0.1,
#zoom_range=0.2,
#rescale=1./255,
validation_split=0.2,
horizontal_flip=1,
vertical_flip=1)
seed = 1
image_datagen.fit(imgs_train, augment=True, seed=seed)
mask_datagen.fit(imgs_mask_train, augment=True, seed=seed)
img_train_generator = image_datagen.flow(imgs_train,shuffle=False, subset='training', batch_size=8,save_to_dir='../mypath',save_prefix='img_train', seed=seed)
mask_train_generator = mask_datagen.flow(imgs_mask_train, shuffle=False,subset='training',batch_size=8,save_to_dir='./mypath',save_prefix='mask_train', seed=seed)
img_val_generator = image_datagen.flow(imgs_train, subset='validation',batch_size=8,save_to_dir='./mypath',save_prefix='img_val', seed=seed)
mask_val_generator = mask_datagen.flow(imgs_mask_train, subset='validation',batch_size=8,save_to_dir='./mypath',save_prefix='mask_val', seed=seed)
train_generator = zip(img_train_generator, mask_train_generator)
val_generator = zip(img_val_generator, mask_val_generator)
model = get_resnet(f=16, bn_axis=3, classes=1)
model.fit_generator(
train_generator,
steps_per_epoch=10,
epochs=3,
validation_data=(val_generator),
validation_steps=10,
verbose=1)
Does someone have some suggestions?
I followed the tutorial from the official keras documentation for image and mask generators till this line: train_generator = zip(image_generator, mask_generator), but when I actually call
model.fit_generator(
train_generator,
steps_per_epoch=2000,
epochs=50)
I got this error message:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-159-d71f198d7e01> in <module>
2 train_generator,
3 steps_per_epoch=100,
----> 4 epochs=20)
c:\users\38909\appdata\local\conda\conda\envs\ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
1424 use_multiprocessing=use_multiprocessing,
1425 shuffle=shuffle,
-> 1426 initial_epoch=initial_epoch)
1427
1428 def evaluate_generator(self,
c:\users\38909\appdata\local\conda\conda\envs\ml\lib\site-packages\tensorflow\python\keras\engine\training_generator.py in model_iteration(model, data, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch, mode, batch_size, **kwargs)
113 batch_size=batch_size,
114 epochs=epochs - initial_epoch,
--> 115 shuffle=shuffle)
116
117 do_validation = validation_data is not None
c:\users\38909\appdata\local\conda\conda\envs\ml\lib\site-packages\tensorflow\python\keras\engine\training_generator.py in convert_to_generator_like(data, batch_size, steps_per_epoch, epochs, shuffle)
375
376 # Create generator from NumPy or EagerTensor Input.
--> 377 num_samples = int(nest.flatten(data)[0].shape[0])
378 if batch_size is None:
379 raise ValueError('You must specify `batch_size`')
AttributeError: 'zip' object has no attribute 'shape'
Does anyone know why? Thanks!
@yanfengliu would you able to resolve the issue, I also have the same one.
@shivg7706 I actually ended up writing my own generator. To make sure that the images and masks have the same augmentation, I recommend https://github.com/albu/albumentations
@shivg7706 This is the data generator I wrote. I haven't optimized it for speed/multi-worker, but at least it works:
import os
import albumentations as albu
import cv2
import keras
import numpy as np
def read_image_from_list(image_list, idx):
img_path = image_list[idx]
img = cv2.imread(img_path)
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
return img
def read_mask_from_list(mask_list, idx):
mask_path = mask_list[idx]
mask = cv2.imread(mask_path)
mask = cv2.cvtColor(mask, cv2.COLOR_BGR2GRAY)
mask = np.expand_dims(mask, axis=-1)
return mask
def augment_image_and_mask(image, mask, aug):
augmented = aug(image=image, mask=mask)
image_augmented = augmented['image']
mask_augmented = augmented['mask']
return image_augmented, mask_augmented
def normalize_img(img):
img = img/255.0
img[img > 1.0] = 1.0
img[img < 0.0] = 0.0
return img
class SegmentationDataGenerator:
def __init__(self, image_list, mask_list, img_size,
batch_size, num_classes, augmentation, shuffle):
self.image_list = image_list
self.mask_list = mask_list
self.img_size = img_size
self.batch_size = batch_size
self.num_classes = num_classes
self.aug = augmentation
self.shuffle = shuffle
self.indices = np.arange(start = 0, stop = len(image_list), step = 1).tolist()
self.idx = 0
self.batch_num = len(image_list) // batch_size
if shuffle:
np.random.shuffle(self.indices)
def read_batch(self):
image_batch = np.zeros((self.batch_size, self.img_size, self.img_size, 3))
mask_batch = np.zeros((self.batch_size, self.img_size, self.img_size, 1))
indices = self.indices[(self.idx*self.batch_size):((self.idx+1)*self.batch_size)]
for i in range(self.batch_size):
idx = indices[i]
img = read_image_from_list(self.image_list, idx)
mask = read_mask_from_list(self.mask_list, idx)
img, mask = augment_image_and_mask(img, mask, self.aug)
img = normalize_img(img)
image_batch[i] = img
mask_batch[i] = mask
self.idx += 1
return image_batch, mask_batch
def get_batch(self):
if (self.idx < self.batch_num):
image_batch, mask_batch = self.read_batch()
else:
if self.shuffle:
np.random.shuffle(self.indices)
self.idx = 0
image_batch, mask_batch = self.read_batch()
return image_batch, mask_batch
def prep_for_model(self, img, mask):
img = img * 2 - 1
mask = keras.utils.to_categorical(mask, self.num_classes)
return img, mask
# constants
IMG_SIZE = 256
BATCH_SIZE = 4
NUM_CLASSES = 7
# read list of filenames from dir
image_list = os.listdir("image")
mask_list = os.listdir("mask")
# shuffle files with a fixed seed for reproducibility
idx = np.arange(len(image_list))
np.random.seed(1)
np.random.shuffle(idx)
image_list = [image_list[i] for i in idx]
mask_list = [mask_list[i] for i in idx]
# split train and test data
train_test_split_idx = int(0.9 * len(image_list))
train_image_list = image_list[:train_test_split_idx]
test_image_list = image_list[train_test_split_idx:]
train_mask_list = mask_list[ :train_test_split_idx]
test_mask_list = mask_list[ train_test_split_idx:]
# define image augmentation operations for train and test set
aug_train = albu.Compose([
albu.Blur(blur_limit = 3),
albu.RandomGamma(),
albu.augmentations.transforms.Resize(height = IMG_SIZE, width = IMG_SIZE),
albu.RandomSizedCrop((IMG_SIZE - 50, IMG_SIZE - 1), IMG_SIZE, IMG_SIZE)
])
aug_test = albu.Compose([
albu.augmentations.transforms.Resize(height = IMG_SIZE, width = IMG_SIZE)
])
# construct train and test data generators
train_generator = SegmentationDataGenerator(
image_list = train_image_list,
mask_list = train_mask_list,
img_size = IMG_SIZE,
batch_size = BATCH_SIZE,
num_classes = NUM_CLASSES,
augmentation = aug_train,
shuffle = True)
test_generator = SegmentationDataGenerator(
image_list = test_image_list,
mask_list = test_mask_list,
img_size = IMG_SIZE,
batch_size = BATCH_SIZE,
num_classes = NUM_CLASSES,
augmentation = aug_test,
shuffle = False)To use the generator in training, do the following:
# training
step = 0
EPOCHS = 100
loss_history = []
for epoch in range(EPOCHS):
print(f'Training, epoch {epoch}')
for i in range(train_generator.batch_num):
step += 1
img_batch, mask_batch = train_generator.get_batch()
img_batch, mask_batch = train_generator.prep_for_model(img_batch, mask_batch)
history = model.fit(img_batch, mask_batch, batch_size = BATCH_SIZE, verbose = False)
loss_history.append(history.history['loss'][-1])I hope this helps. If you spot anything to correct or change, feel free to leave a comment :)
When I use the ImageDataGenerator with masks as labels,the value of the masks will change,but I do not set rescale.The original value of mask is between 0 and 5.After imagedatagenerator ,it become 255.
`# -- coding: utf-8 --
"""
@author: nzl
"""
###########################################################
####change picture and mask
###########################################################
we create two instances with the same arguments
from keras.preprocessing.image import ImageDataGenerator
train_datagen = ImageDataGenerator(
rotation_range=40,
width_shift_range=0.1,
height_shift_range=0.1,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True,
vertical_flip=True,
fill_mode='constant',
cval=0,)
seed = 1
train_images_generator = train_datagen.flow_from_directory(
'./data_raw/argfile',
target_size=(540, 960),
batch_size=1,
color_mode="rgb",
class_mode=None,
save_to_dir='data_arg/5',
save_format='png',
seed=seed,
shuffle=True)
train_masks_generator = train_datagen.flow_from_directory(
'./data_label/argfile',
target_size=(540, 960),
batch_size=1,
color_mode="grayscale",
class_mode=None,
save_to_dir='mask_arg/5',
save_format='png',
seed=seed,
shuffle=True)
i = 0
for batch in train_images_generator:
i += 1
if i > 69:
break # otherwise the generator would loop indefinitely
if i == 1:
print(batch.shape) # this is a Numpy array with shape (1, 224, 224, 3)
i = 0
for batch in train_masks_generator:
i += 1
if i > 69:
break # otherwise the generator would loop indefinitely
if i == 1:
print(batch.shape) # this is a Numpy array with shape (1, 224, 224, 1)
`
While @peachthiefmedia 's solution is very neat, the use of zip prevents the use of use_multiprocessing=True when fitting the data. Indeed we are now using a generator which is not thread safe (we get the error message UserWarning: Using a generator with use_multiprocessing=True and multiple workers may duplicate your data. Please consider using thekeras.utils.Sequence class.`).
One solution is to implement a Sequence object to merge the 2 generators:
from keras.utils import Sequence
class MergedGenerators(Sequence):
def __init__(self, *generators):
self.generators = generators
# TODO add a check to verify that all generators have the same length
def __len__(self):
return len(self.generators[0])
def __getitem__(self, index):
return [generator[index] for generator in self.generators]
train_generator = MergedGenerators(image_generator, mask_generator)Given the folder structure,
/data/images/image1.jpg
/data/images/image2.jpg
/data/masks/image1.jpg
/data/masks/image1.jpg
and the corresponding error output: "Found 0 images belonging to 0 classes"
Workaround discussed above is to tweak the folder structure as below.
/data/images/0/image1.jpg
/data/images/0/image2.jpg
/data/masks/0/image1.jpg
/data/masks/0/image1.jpg
However, this can be avoided with changes to the path and using additional "classes" parameter.
image_generator = image_datagen.flow_from_directory(
'data/',
classes=['images'],
class_mode=None,
seed=seed)
mask_generator = mask_datagen.flow_from_directory(
'data/',
classes=['masks'],
class_mode=None,
seed=seed)
https://kylewbanks.com/blog/loading-unlabeled-images-with-imagedatagenerator-flowfromdirectory-keras
should help to understand better.
What will be the scenario if I have to send multiple masks for a same image as input? What changes are needed to be done to the ImageData Generator
@sammilei Hi, I did something pretty similar (identical) to your code but my images and masks (that I save through "save_to_dir" option) do not seem to match.
Moreover, masks are ok but images are saved as totally black (I have rgb images).
def train_and_predict(): imgs_train, imgs_mask_train, _ = load_train_data() imgs_train = imgs_train.astype('float32') mean = np.mean(imgs_train) std = np.std(imgs_train) imgs_train -= mean imgs_train /= std imgs_mask_train = imgs_mask_train.astype('float32') imgs_mask_train /= 255. # scale masks to [0, 1] image_datagen = ImageDataGenerator( #featurewise_center=True, #featurewise_std_normalization=True, #rotation_range=90., #width_shift_range=0.1, #height_shift_range=0.1, #zoom_range=0.2, #rescale=1./255, validation_split=0.2, horizontal_flip=1, vertical_flip=1) mask_datagen = ImageDataGenerator( #featurewise_center=True, #featurewise_std_normalization=True, #rotation_range=90., #width_shift_range=0.1, #height_shift_range=0.1, #zoom_range=0.2, #rescale=1./255, validation_split=0.2, horizontal_flip=1, vertical_flip=1) seed = 1 image_datagen.fit(imgs_train, augment=True, seed=seed) mask_datagen.fit(imgs_mask_train, augment=True, seed=seed) img_train_generator = image_datagen.flow(imgs_train,shuffle=False, subset='training', batch_size=8,save_to_dir='../mypath',save_prefix='img_train', seed=seed) mask_train_generator = mask_datagen.flow(imgs_mask_train, shuffle=False,subset='training',batch_size=8,save_to_dir='./mypath',save_prefix='mask_train', seed=seed) img_val_generator = image_datagen.flow(imgs_train, subset='validation',batch_size=8,save_to_dir='./mypath',save_prefix='img_val', seed=seed) mask_val_generator = mask_datagen.flow(imgs_mask_train, subset='validation',batch_size=8,save_to_dir='./mypath',save_prefix='mask_val', seed=seed) train_generator = zip(img_train_generator, mask_train_generator) val_generator = zip(img_val_generator, mask_val_generator) model = get_resnet(f=16, bn_axis=3, classes=1) model.fit_generator( train_generator, steps_per_epoch=10, epochs=3, validation_data=(val_generator), validation_steps=10, verbose=1)Does someone have some suggestions?
@michirup I also have the problem, that the masks and images do not match anymore! Could you fix it?
I do form the dataset and the input pipeline differently now:
file_list_train = [f for f in os.listdir(img_dir_train) if os.path.isfile(os.path.join(img_dir_train, f))]
#Separate frame and mask files lists, exclude unnecessary files
frames_list_train = [file for file in file_list_train if ('_L' not in file) and ('txt' not in file)]
#print(file_list_train)
masks_list_train = [file for file in file_list_train if ('_L' in file) and ('txt' not in file)]
frames_paths_train = [os.path.join(img_dir_train, fname) for fname in frames_list_train]
masks_paths_train = [os.path.join(img_dir_train, fname) for fname in masks_list_train]
frame_data_train = tf.data.Dataset.from_tensor_slices(frames_paths_train)
masks_data_train = tf.data.Dataset.from_tensor_slices(masks_paths_train)
frame_tensors_train = frame_data_train.map(_read_to_tensor)
masks_tensors_train = masks_data_train.map(_read_to_tensor)
frame_batches_train = tf.compat.v1.data.make_one_shot_iterator(frame_tensors_train)
#outside of TF Eager, we would use make_one_shot_iterator
mask_batches_train = tf.compat.v1.data.make_one_shot_iterator(masks_tensors_train)
#n_images_to_iterate = len(frames_paths_train)
n_images_to_iterate_train = len(frames_list_train)
n_images_to_show_train = 20
list_all_train_frames=[]
list_all_train_masks=[]
for i in range(n_images_to_iterate_train):
# Get the next image from iterator
frame = frame_batches_train.next().numpy().astype(np.uint8)
list_all_train_frames.append(frame)
mask = mask_batches_train.next().numpy().astype(np.uint8)
mask = rgb_to_onehot(mask)
train_dataset = tf.data.Dataset.from_tensor_slices((list_all_train_frames, list_all_train_masks))
print("sliced")
train_dataset = train_dataset.repeat(14)
train_dataset = train_dataset.shuffle(buffer_size=500, seed=seed)
print("shuffled")
train_dataset = train_dataset.batch(batch_size)
print("batched")
print(len(list_all_train_frames), "train_dataset is ready")
The same I do for the validation dataset.
Later on I call:
result = model.fit_generator(train_dataset, steps_per_epoch=steps_per_epoch , validation_data = val_dataset, validation_steps = validation_steps, epochs=num_epochs, callbacks=callbacks)
If I execute the same code on my laptop instead of the computer I usually execute the code on, I do again face the problem that the mask and image files do not match anymore. I figured out that it is caused by the line:
file_list_val = [f for f in os.listdir(img_dir_val) if os.path.isfile(os.path.join(img_dir_val, f))]
This line of code seems to work differently on the two computers. Maybe it's caused by the version of the package or some other package? I am not sure but on one of my computers it behaves the way it is supposed to.
@YasarL Thanks for the swift response. Is it due to version changes? Never faced such issue before.
Hey @aliechoes ! I solved this problem by the creation of my own datagenerator for images and masks simultaneously. Here you have the code I used:
@amlarraz , do you have RGB images in your 'img_dir' and black & white images in your 'label_dir'?
@EtagiBI yes, I had RGB images in img_dir and grayscale images in label_dir, however if you want to use grayscale images in img_dir you can (but the typical architectures pretrained in imagenet need 3-channel images as input)
@EtagiBI yes, I had RGB images in img_dir and grayscale images in label_dir
Hmm. It should work then. I get the following error at the very beginning of the learning process:
ValueError: Error when checking input: expected img to have shape (1536, 1536, 1) but got array with shape (1536, 1536, 3)
I'm using the same generator but without resizing (my source images already have desired size).
def data_gen(img_folder, mask_folder, batch_size):
c = 0
n = os.listdir(img_folder) #List of training images
random.shuffle(n)
while (True):
img = np.zeros((batch_size, 512, 512, 3)).astype('float')
mask = np.zeros((batch_size, 512, 512, 1)).astype('float')
for i in range(c, c+batch_size): #initially from 0 to 16, c = 0.
train_img = cv2.imread(img_folder+'/'+n[i])/255.
img[i-c] = train_img #add to array - img[0], img[1], and so on.
train_mask = cv2.imread(mask_folder+'/'+n[i], cv2.IMREAD_GRAYSCALE)/255.
train_mask = train_mask.reshape(512, 512, 1) # Add extra dimension for parity with train_img size [512 * 512 * 3]
mask[i-c] = train_mask
c+=batch_size
if(c+batch_size>=len(os.listdir(img_folder))):
c=0
random.shuffle(n)
# print "randomizing again"
yield img, mask
could you please share all the error to see the line where the problem is?
could you please share all the error to see the line where the problem is?
@amlarraz here's a complete traceback:
Traceback (most recent call last):
File "E:/Explorium/python/unet_trainer.py", line 83, in <module>
results = model.fit_generator(train_generator, epochs=EPOCHS, steps_per_epoch=STEPS_PER_EPOCH, validation_data=val_generator, validation_steps=VALIDATION_STEPS, callbacks=callbacks)
File "C:\Users\E-soft\Anaconda3\envs\Explorium\lib\site-packages\tensorflow_core\python\keras\engine\training.py", line 1297, in fit_generator
steps_name='steps_per_epoch')
File "C:\Users\E-soft\Anaconda3\envs\Explorium\lib\site-packages\tensorflow_core\python\keras\engine\training_generator.py", line 265, in model_iteration
batch_outs = batch_function(*batch_data)
File "C:\Users\E-soft\Anaconda3\envs\Explorium\lib\site-packages\tensorflow_core\python\keras\engine\training.py", line 973, in train_on_batch
class_weight=class_weight, reset_metrics=reset_metrics)
File "C:\Users\E-soft\Anaconda3\envs\Explorium\lib\site-packages\tensorflow_core\python\keras\engine\training_v2_utils.py", line 253, in train_on_batch
extract_tensors_from_dataset=True)
File "C:\Users\E-soft\Anaconda3\envs\Explorium\lib\site-packages\tensorflow_core\python\keras\engine\training.py", line 2472, in _standardize_user_data
exception_prefix='input')
File "C:\Users\E-soft\Anaconda3\envs\Explorium\lib\site-packages\tensorflow_core\python\keras\engine\training_utils.py", line 574, in standardize_input_data
str(data_shape))
ValueError: Error when checking input: expected img to have shape (1536, 1536, 1) but got array with shape (1536, 1536, 3)
I think everything is ok in your generator, the error says that you have images with 3 channels and your images have 3 channels.
The error shows that your problem is in the function "standardize_input_data", are you using an automatic standarize keras method? Maybe you can standarize your images in the data_gen directly:
train_mask = ((cv2.imread(mask_folder+'/'+n[i], cv2.IMREAD_GRAYSCALE)/255.) - (mean/255.))/(std/255.)
Where "mean" and "std" are your train set channel mean and std.