MemoryFormer: Minimize Transformer Computation by Removing Fully-Connected Layers