Today, we are excited to announce the beta release of TensorFlow in the Mesosphere DC/OS Service Catalog. Using a single command, you can now deploy distributed TensorFlow on any bare-metal, virtual, or public cloud infrastructure. As with other packages available for DC/OS, the new TensorFlow package also includes the ability to use GPUs to accelerate your machine learning and deep learning applications. In the race to leverage deep learning capabilities, data scientists specializing in deep learning are highly sought after. An efficient data science infrastructure allows you to attract the best data scientists and get the best work out of them, which gives your business a strategic advantage over competitors.
Yahoo has announced TensorFlowOnSpark, its latest open source framework for distributed deep learning on big data clusters. Deep learning (DL) has evolved significantly in recent years. At Yahoo, we've found that in order to gain insight from massive amounts of data, we need to deploy distributed deep learning. Existing DL frameworks often require us to set up separate clusters for deep learning, forcing us to create multiple programs for a machine learning pipeline (see Figure 1 below). Having separate clusters requires us to transfer large datasets between them, introducing unwanted system complexity and end-to-end learning latency.
In the context of machine learning, tensor refers to the multidimensional array used in the mathematical models that describe neural networks. In other words, a tensor is usually a higher-dimension generalization of a matrix or a vector. Through a simple notation that uses a rank to show the number of dimensions, tensors allow the representation of complex n-dimensional vectors and hyper-shapes as n-dimensional arrays. Tensors have two properties: a datatype and a shape. TensorFlow is an open source deep learning framework that was released in late 2015 under the Apache 2.0 license.
The art and science of training neural networks from large data sets in order to make predictions or classifications has experienced a major transition over the past several years. Through popular and growing interest from scientists and engineers, this field of data analysis has come to be called deep learning. Put succinctly, deep learning is the ability of machine learning algorithms to acquire feature hierarchies from data and then persist those features within multiple non-linear layers which comprise the machine's learning center, or neural network. Two years ago, questions were mainly about what deep learning is, and how it might be applied to problems in science, engineering, and finance. Over the past year, however, the climate of interest has changed from a curiosity about what deep learning is, and into a focus on acquiring hardware and software in order to apply deep learning frameworks to specific problems across a wide range of disciplines.