Language Generation with Infinite Contamination