r/MachineLearning - [D] PyTorch Global GPU Flag

You are saying that dataloaders give CPU tensors by default but that is usually preferred. For instance, for images, loader backends (like PIL) are implemented on CPU so the data is first loaded in RAM and then passed to GPU. And when you want to do some pre-processing operations, it's actually preferable to make them on CPU since it won't slow down your model because it's made in parallel. Moreover most of these operations, like resizing, are well optimized for CPUs (using Pillow SIMD for example). I think, like others said, that having control over when and where data is moved is a nice way to make sure that you are doing exactly what you want.