Knowledge distillation (KD) is a very popular method for model size reduction. Recently, the technique is exploited for quantized deep neural networks (QDNNs) training as a way to restore the performance sacrificed by word-length reduction. KD, however, employs additional hyper-parameters, such as temperature, coefficient, and the size of teacher network for QDNN training. We analyze the effect of these hyper-parameters for QDNN optimization with KD. We find that these hyper-parameters are inter-related, and also introduce a simple and effective technique that reduces \textit{coefficient} during training. With KD employing the proposed hyper-parameters, we achieve the test accuracy of 92.7% and 67.0% on Resnet20 with 2-bit ternary weights for CIFAR-10 and CIFAR-100 data sets, respectively.

Prior work poses a diversity paradox for science. Diversity breeds scientific innovation, and yet, diverse individuals have less successful scientific careers. But if diversity is good for innovation, why is science not rewarding diversity? We answer this question by utilizing a near-population of ~1.03 million US doctoral recipients from 1980-2015 and their careers into publishing and faculty roles. The article uses text analysis and machine learning techniques to answer a series of questions: How can we detect scientific innovation? Does diversity breed innovation? And are the innovations of diverse individuals adopted and rewarded? Our analyses show that underrepresented groups produce higher rates of scientific novelty. However, their novel contributions are discounted: e.g., innovations by gender minorities are taken up by other scholars at lower rates than innovations by gender majorities, and innovations by gender and racial minorities result in fewer academic positions. This suggests an unfair system in which diverse individuals innovate, but their innovations are disproportionately ignored and fail to convert into career success at the same rate as majority groups. In sum, there may be an unwarranted reproduction of stratification in academic careers that discounts diversity's role in innovation and partly explains the underrepresentation of some groups in academia.

Deep learning algorithms often require solving a highly non-linear and nonconvex unconstrained optimization problem. Methods for solving optimization problems in large-scale machine learning, such as deep learning and deep reinforcement learning (RL), are generally restricted to the class of first-order algorithms, like stochastic gradient descent (SGD). While SGD iterates are inexpensive to compute, they have slow theoretical convergence rates. Furthermore, they require exhaustive trial-and-error to fine-tune many learning parameters. Using second-order curvature information to find search directions can help with more robust convergence for non-convex optimization problems. However, computing Hessian matrices for large-scale problems is not computationally practical. Alternatively, quasi-Newton methods construct an approximate of the Hessian matrix to build a quadratic model of the objective function. Quasi-Newton methods, like SGD, require only first-order gradient information, but they can result in superlinear convergence, which makes them attractive alternatives to SGD. The limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) approach is one of the most popular quasi-Newton methods that construct positive definite Hessian approximations. In this chapter, we propose efficient optimization methods based on L-BFGS quasi-Newton methods using line search and trust-region strategies. Our methods bridge the disparity between first- and second-order methods by using gradient information to calculate low-rank updates to Hessian approximations. We provide formal convergence analysis of these methods as well as empirical results on deep learning applications, such as image classification tasks and deep reinforcement learning on a set of ATARI 2600 video games. Our results show a robust convergence with preferred generalization characteristics as well as fast training time.

Existing machine reading comprehension (MRC) models do not scale effectively to real-world applications like web-level information retrieval and question answering (QA). We argue that this stems from the nature of MRC datasets: most of these are static environments wherein the supporting documents and all necessary information are fully observed. In this paper, we propose a simple method that reframes existing MRC datasets as interactive, partially observable environments. Specifically, we "occlude" the majority of a document's text and add context-sensitive commands that reveal "glimpses" of the hidden text to a model. We repurpose SQuAD and NewsQA as an initial case study, and then show how the interactive corpora can be used to train a model that seeks relevant information through sequential decision making. We believe that this setting can contribute in scaling models to web-level QA scenarios.

In this article you will learn how to automatically generate a regression model to predict taxi fare prices by using automated machine learning capabilities within Azure Machine Learning service. Moreover, you will learn how to launch an automated machine learning process to allow algorithm selection and hyperparameter tuning. Automated machine learning iterates over many combinations of algorithms and hyperparameters until it finds the best model based on your criterion. In this article, I assume you have already downloaded the data from Azure Open Datasets and ran through the data preparation steps in this tutorial for the NYC Taxi data so it could be used to build our machine learning model. Let's start by creating our workspace object from the existing workspace.

Automated machine learning pre-processing steps (feature normalization, handling missing data, converting text to numeric, etc.) become part of the underlying model. When using the model for predictions, the same pre-processing steps applied during training are applied to your input data automatically.

Deep learning is a subset of machine learning that's based on artificial neural networks. The learning process is deep because the structure of artificial neural networks consists of multiple input, output, and hidden layers. Each layer contains units that transform the input data into information that the next layer can use for a certain predictive task. Thanks to this structure, a machine can learn through its own data processing.

I recently gave an informal talk to a class of botany students at Gavilan College. The original topic was nature photography, but I also talked about the data science techniques that I used to create my recently completed photo book, Portraits of Birds: Shoreline Park. The concept for the book was to try to personally take photos of all of the bird species in a particular area, in this case Shoreline at Mountain View Park in Mountain View, California, which I later expanded to include the Palo Alto Baylands. To enumerate the species that have been seen in this area, I turned to two citizen science sites, iNaturalist and eBird, both of which have application programmatic interfaces (APIs). Note that while eBird is specific to birds, iNaturalist contains data on plants and other animals as well.

Technology is everywhere in our lives today-- the workplace, in our homes, on our person and, now, in our schools. It figures that given the digital transformation of workplaces across industries-- and the increasing automation of our businesses-- the next generation of workers are early immersed in a culture of experimentation and innovation with emerging technologies and software. But that's not the sole aim of technology for education-- a growing market more commonly referred to as EdTech. It also offers an alternative to'traditional' classroom learning methods, engaging students which new, innovative formats, and the ability to tailor approaches to fit the individual student. According to a forecast laid out by Knowledgemotion CEO David Bainbridge in Forbes, the EdTech industry will reach a global value of US$252 billion by 2020.

If your company could benefit from predicting the future (or what factors impact a particular outcome), then Predictive Analytics and Machine Learning technology may represent a wise investment -- one that gives your business a competitive advantage. At SevenTablets, we partner with companies worldwide to overcome obstacles using cutting-edge technology, including Predictive Analytics (PA), Machine Learning and Augmented Reality (AR). Our experience speaks volumes, as we've worked with clients in a number of fields, including medical and healthcare, insurance, manufacturing and beyond. We can integrate these technologies into a custom software platform, including Enterprise Resource Planning (ERP) platforms, Customer Relationship Management (CRM) software, SaaS solutions and mobile apps.