CLOVer: Cross-Layer Orthonormal Vectors Adaption