tensorflow/tensorflow

K-FAC in TensorFlow is an implementation of K-FAC, an approximate second-order optimization method, in TensorFlow. When applied to feedforward and convolutional neural networks, K-FAC can converge 3.5x faster in 14x fewer iterations than SGD with Momentum. K-FAC, short for "Kronecker-factored Approximate Curvature", is an approximation to the Natural Gradient algorithm designed specifically for neural networks. It maintains a block-diagonal approximation to the Fisher Information matrix, whose inverse preconditions the gradient. K-FAC can be used in place of SGD, Adam, and other Optimizer implementations.