Varying sample size in Siamese network

[debmalya97]

I have an image dataset where the folder structure is as follows- there are 900 folders (each folder will be a class) and in each folder, we have a varying number of images(each image has been resized to the same pixel intensity).

Now when we construct the 4d matrix to be fed to the deep learning model it should be of the dimension-

(900,A,105,105)

900 because that's the total number of classes
105 x 105 because I have resized each image to that
now A should be the number of images in each folder.
In my case, the number of images in each folder is varying so my question is-

If I keep the original varying number of images in each folder that means in my 4d array A would be the dimension of the largest folder size which means for folders of lesser number of images it will have zero 2d matrix or matrices. So is that technically right or should I make all the folder sizes the same?