6 Supplementary Material 6.1 Network Architecture

The section explains detailed CipherNav network architecture in Table 4, 5 and 6. The view encoder E is shown in Table 4 and map encoder E is shown in Table 5. The encoders are trained end-to-end during plaintext training and freezed during ciphertext training. Each party has a copy of the encoder models and locally computes all forward passes in ciphertext training. The action classification network Gis shown in Table 6.