Neural network augmented inverse problems for PDEs