Training loss reduces sooo slow

[vodp]

Using ResnetBuilder.build((1, 32, 32), 1, 'basic_block', [1, 1, 1, 1]) to build my own network for image data of size 32x32 with depth 1, I cannot make the training loss decrease as normally it does. In fact the loss reduces very very slow (two number after decimal point) no matter how big the learning rate is set. I tried both SGD and Adam but do not seem to work. In the meantime, my data used with a simple feed forward convnet run very good. What problem it may happen with resnet arch. if you have any idea?

Edit: my bad. I use the resnet for regression purpose so I need to change activation in the last layer as 'linear'.