FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language

Open in new window