Operator learning without the adjoint