RMem: Restricted Memory Banks Improve Video Object Segmentation