On-Demand Spark clusters with GPU acceleration

Apache Spark has become the de-facto standard for processing large amounts of stationary and streaming data in a distributed fashion. The addition of the MLlib library, consisting of common learning algorithms and utilities, opened up Spark for a wide range of machine learning tasks and paved the way for running complex machine learning workflows on top of Apache Spark clusters. To address the challenges associated with complexity and costs Domino offers the ability to dynamically provision and orchestrate a Spark cluster directly on the infrastructure backing the Domino instance. This allows Domino users to get quick access to Spark without having to rely on their IT team to create and manage one for them. The Spark workloads are fully containerized on the Domino Kubernetes cluster and users can access Spark interactively through a Domino workspace (e.g.