HERA: High-efficiency Matrix Compression via Element Replacement