CoNVOI: Context-aware Navigation using Vision Language Models in Outdoor and Indoor Environments