r/deeplearning - Better result with small batch size

Iam working on a deep neural network (U-net architecture) where the input are segmentations and the output is a heatmap. The problem is that there is a lot of variation in my dataset. I noticed that when training with a batch size of 32, the output of my network is more a mean of all inputs (which makes sense). Now I did one training with a batch size of 2. Here I could observe that I, at some epochs, reach a validation loss which is much much better compared to the larger batch size.