Does data for artificial intelligence and machine learning need their own workflows and orchestration system? It does, according to Union.ai, which offers an open source solution called Flyte that provides workflow and orchestration to fit the unique demands of data, not software. "The number one feedback we get from people who use orchestrators for machine learning is that they're not made for AI workflows, machine learning workflows, because you're forced to write YAML code, you're forced to do understand Docker files," Martin Stein, chief marketing officer and head of developer relations at Union.ai, told The New Stack. "You're forced to really do things that machine learning engineers, data scientists and researchers don't do." Basically, with Flyte, developers write their code and then run it locally or remotely, he added.

Finally, one more interesting aspect of our training infrastructure is that we use a multi-cloud setup in practice. As it was told earlier, GCP is our main vendor for training instances for cost and powerful machines availability-related reasons, while our default production infrastructure is AWS. It means that sometimes we need to combine the two: e.g. We use Flyte to orchestrate this process. Flyte is a workflow management system that allows us to define a pipeline as a DAG of tasks. It is useful for us because it allows us to define a pipeline once and run its steps on different machines with different computational resources allocation, and it also provides a nice UI for monitoring the progress of the pipeline.

Apache Airflow is an open source platform that can be used to author, monitor, and schedule data pipelines. It is used by companies like Airbnb, Lyft, and Twitter and has been the go-to tool in the data engineering ecosystem. With an increased necessity for orchestration of data pipelines, Airflow witnessed tremendous growth. It has broadened its scope from data to machine learning and is now being used for a variety of use cases. But since machine learning in itself demands a distinctive orchestration, Airflow needs to be extended to accommodate all the MLOps requirements.

Flyte is helping organizations like Spotify and Lyft build a new generation of products that make elegant use of complex data and machine learning. Now Union AI, the team behind Flyte, is creating a managed version of the workflow orchestrator. Sign up now for access to the pre-release version of Union Cloud, and help us shape the future of ML and data-pipeline development. We use machine learning every day: it powers our internet searches, shopping carts, song lists, social media feeds, shared rides, even how we order our food.

Lyft describes'Flyte as a "structured and distributed platform for concurrent, scalable, and maintainable machine learning workflows." 'Flyte' is built to en-power and speedup machine learning models and data orchestration to be compatible with the latest products and applications. Flyte comes with Flytekit -- a Python SDK to develop applications on Flyte to allow contributors to provide rapid integrations with new services or systems. Apart from Flytekit, 'Flyte' also provides backend plugins which can be used to create and manage Kubernetes resources, including CRDs like Spark-on-k8s, or any remote system like Amazon Sagemaker, Qubole, BigQuery, and more.

The reveal of the ridiculous Cybertruck design last week made me curious about the history of cars. If you look at pictures of cars from the early days (as I, a Normal Person, did last Friday night), you'll see some insane ideas. Before we got to the Ford Model-T that standardized car production, people iterated on a ton of crazy stuff. It took some time for people to experiment and agree on what a car even was, what features it had, and how it needed to work. For example, for a long time in the beginning, quite a few cars ran on steam, until gasoline began to overtake them (thanks in part to Henry Ford's standardization of the assembly line, which made non-gasoline cars harder to produce.) Eventually, all the cars standardized to the form we know today: a closed car, powered by gasoline, with four wheels, four windows, seating 4-8 people. Even the godawful Cyberthing follows this model.