LEF: Late-to-Early Temporal Fusion for LiDAR 3D Object Detection