r/MachineLearning - [D] Best practices for running deep learning experiments on the cloud

I'm working on a machine translation research project, and appealing to the wisdom of the masses here to hopefully improve my workflow. I'm looking for recommendations on workflow optimization and cloud provider tools (I want to avoid buying a GPU specifically for this task) for my project. I know which deep learning framework to use (and what other python libraries, so not looking for recommendations in that area). The data is large-ish -- too large for Git but not large enough for Hadoop. I'm anticipating many experiment variations, where hyper parameters as well as pre-processing of the data may change between experiments.