Retrieval or Global Context Understanding? On Many-Shot In-Context Learning for Long-Context Evaluation