Intel Software Guard Extensions (Intel SGX) is a securing computing tool that generates a trusted execution environment (TEE) for users that need secure and confidential environments for such use cases as private key management, multi-party computing with private data, and securing public cloud deployment for critical applications. While the Intel SGX SDK for Linux* OS successfully tackles these important use cases, its implementation is not simple. It can require significant system redesign and code changes by engineers because under the SGX SDK's threat model, the OS is not trusted, and only trusted applications and code can be worked on in the secure environment portioned out by SGX, i.e., an "enclave." Therefore, the trusted and untrusted components of the applications involved need to be separated. Moreover, engineers will then need to re-engineer some of their code base to ensure it will be trusted in this enclave.

The following blog post is part of Machine Learning in Production (17634) coursework at Carnegie Mellon University. In this discussion, we will consider the scenario of Movie Recommendation with respect to the following two frameworks. Now that we have mentioned the resources and formed a basic blog premise let's talk about some standard definitions. Movie recommendations are challenging because we can approach it in multiple ways, especially since we have numerous data points that can be modeled for the recommendation. Let's try to analyze some of the problems.

Sponsored Everyone seems alive to the potential of Artificial Intelligence in business and the public sector. According to research from PwC, worldwide we can expect to see a boost to economies of $15.7tn by 2030 as more organisations unlock new opportunities in big data and advanced analytics in such fields as financial services, retail, transport and government. The sweet spot of AI is the potential scale of data processing at a volume beyond human capabilities – combined with the capacity for systems to learn and develop unprompted responses. But while PwC's numbers are impressive – and explain why so many are keen to start big-data and AI initiatives – it's important to remember we're still in a very early stage of deployment. "We have to think that AI is still a teenager," says Walter Riviera, EMEA AI Technical Engineer at Intel .

In more ways than one, software has become the last mile between the developers and the underlying hardware infrastructure, enabling them to utilise the optimization capabilities of processors. Analytics India Magazine spoke to Akanksha Bilani, Country Lead -- India, Singapore, ANZ at Intel Software to understand why, in today's world, transformation of software is key to driving effective business, usage models and market opportunity. "Gone are the days where adding more racks to existing platforms helped drive productivity. Moore's law and AI advocates that the way to take advantage of hardware is by driving innovation on software that runs on top of it. Studies show that modernization, parallelisation and optimization of software on the hardware helps in doubling the performance of our hardware," she emphasizes.

"TensorFlow is by far the most popular tool among our respondents, with Keras in second place, and PyTorch in third. Other frameworks like MXNet, CNTK, and BigDL have growing audiences as well" As if businesses today didn't already have enough to worry about, then along comes a new wave of game-changing technologies that they must master quickly if they are not to fall behind their competitors, with pressure mounting to start using AI. . Artificial Intelligence is the most visible of these technologies – and arguably the most important. Open a newspaper, and it might seem as if every business is making great strides towards developing and using AI applications that will transform their operations and enable them to deliver new products and services to their customers. It's easy for businesses yet to achieve success by using AI – or even to get started on their journey – to get despondent about the lead they perceive their competitors to have.

In this paper, we present BigDL, a distributed deep learning framework for Big Data platforms and workflows. It is implemented on top of Apache Spark, and allows users to write their deep learning applications as standard Spark programs (running directly on large-scale big data clusters in a distributed fashion). It provides an expressive, "data-analytics integrated" deep learning programming model, so that users can easily build the end-to-end analytics + AI pipelines under a unified programming paradigm; by implementing an AllReduce like operation using existing primitives in Spark (e.g., shuffle, broadcast, and in-memory data persistence), it also provides a highly efficient "parameter server" style architecture, so as to achieve highly scalable, data-parallel distributed training. Since its initial open source release, BigDL users have built many analytics and deep learning applications (e.g., object detection, sequence-to-sequence generation, visual similarity, neural recommendations, fraud detection, etc.) on Spark.

Intel recently released its BigDL project for distributed deep learning on Apache Spark. BigDL has native Spark integration, allowing it to leverage Spark during model training, prediction, and tuning. This blog post gives highlights of BigDL and a tutorial showing how to get started with BigDL on Databricks. BigDL is an open source deep learning library from Intel. Modeled after Torch, BigDL provides functionality both for low-level numeric computing and high-level neural networks.

BigDL runs natively on Apache Spark, and because Qubole offers a greatly enhanced and optimized Spark as a service, it makes for a perfect deployment platform. In this Part 1 of a two-part series, you will learn how to get started with distributed Deep Learning library BigDL on Qubole. By the end, you will have BigDL installed on a Spark cluster with a distributed Deep Learning library readily available for you to use in your Deep Learning applications running on Qubole. In Part 2, you will learn how to write a Deep Learning application on Qubole that uses BigDL to identify handwritten digits (0 to 9) using a LeNet-5 (Convolutional Neural Networks) model that you will train and validate using MNIST database. Before we get started, here's some introduction and background on the technologies involved.

This work was done in collaboration with Ding Ding and Sergey Ermolin from Intel. In recent years, the scale of datasets and models used in deep learning has increased dramatically. Although larger datasets and models can improve the accuracy in many AI applications, they often take much longer to train on a single machine. However, it is not very common to distribute the training to large clusters using current popular deep learning frameworks, compared to what's been long around in the Big Data area, as it's often harder to gain access to a large GPU cluster and lack of convenient facilities in popular DL frameworks for distributed training. By leveraging the cluster distribution capabilities in Apache Spark, BigDL successfully performs very large-scale distributed training and inference.

The field of machine learning – and deep learning in particular – has made significant progress recently and use cases for deep learning are becoming more common in the enterprise. We've seen more of our customers adopt machine learning and deep learning frameworks for use cases like natural language processing with free-text data analysis, image recognition systems, threat detection, fraud detection, and more. And as with other use cases in Big Data analytics and data science, they want to run their preferred deep learning frameworks and tools in Docker containers on the BlueData EPIC software platform. So What is Deep Learning? "Deep learning (also known as deep structured learning or hierarchical learning) is part of a broader family of machine learning methods based on learning data representations, as opposed to task-specific algorithms."