Efficient Expansion and Gradient Based Task Inference for Replay Free Incremental Learning