Lower Bounds for Time-Varying Kernelized Bandits