Goto

Collaborating Authors

 train delay analysis


A Data Pipeline for Go Trains Delay Analysis -- ML in Action with Rust

#artificialintelligence

This function is responsible for connecting to the database to extract the data, as some target format depends on the storage we decide on. We have multiple options, such as tradition, Hadoop data lake, cloud storage bucket (s3, ADLS gen2, GCP storage). For the simplicity of this project, and it seems the data volume is minimal, let's keep it in the csv format. Another exciting decision point will be how to get the data consistently and efficiently, which meets one of our NFR, such as incremental or batch load. However, instead of the batch load (SQL select *) and then write to file in a for loop, we will consider using the COPY out for performance needed, why?