A computational problem in biological reward-based learning is how credit assignment is performed in the nucleus accumbens (NAc). Much research suggests that NAc dopamine encodes temporal-difference (TD) errors for learning value predictions. However, dopamine is synchronously distributed in regionally homogeneous concentrations, which does not support explicit credit assignment (like used by backpropagation). It is unclear whether distributed errors alone are sufficient for synapses to make coordinated updates to learn complex, nonlinear reward-based learning tasks. We design a new deep Q-learning algorithm, Artificial Dopamine, to computationally demonstrate that synchronously distributed, per-layer TD errors may be sufficient to learn surprisingly complex RL tasks. We empirically evaluate our algorithm on MinAtar, the DeepMind Control Suite, and classic control tasks, and show it often achieves comparable performance to deep RL algorithms that use backpropagation.

At a glance, the tech giants don't seem to have a lot in common. Meta connects you to friends and family. Apple makes phones and computers. Microsoft is all about business software. But under the hood, they are united by advertising, referred to as the "dark beating heart of the internet" by the author Tim Hwang in his book Subprime Attention Crisis.

The Biden administration just released a document that they believe should define the standards for responsible use of one of the more critical technologies that is set to define the future – Artificial Intelligence (AI). The document, "The Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People," was released by the White House Office of Science and Technology Policy (WHOSTP). It lays out the five guiding principles that the WHOSTP feels should guide the "design, use, and deployment" of automated systems in order to protect Americans in the age of AI. The Blueprint emphasizes creating safe and effective AI systems, providing algorithmic discrimination protections, data privacy, clarified notice and explanations of how AI may be used, and providing alternative options for consumers that choose to opt out. This idea of governmental guidance in AI may seem innovative, but the truth is, at least 60 countries already have national AI protocols and the United States is merely playing catch-up at this point.

Online advertising is a major source of income for many online companies. One common approach is to sell online advertisements via waterfall auctions, where a publisher makes sequential price offers to ad networks. The publisher controls the order and prices of the waterfall and by that aims to maximize his revenue. In this work, we propose a methodology to learn a waterfall strategy from historical data by wisely searching in the space of possible waterfalls and selecting the one leading to the highest revenue. The contribution of this work is twofold; First, we propose a novel method to estimate the valuation distribution of each user with respect to each ad network. Second, we utilize the valuation matrix to score our candidate waterfalls as part of a procedure that iteratively searches in local neighborhoods. Our framework guarantees that the waterfall revenue improves between iterations until converging to a local optimum. Real-world demonstrations are provided to show that the proposed method improves the total revenue of real-world waterfalls compared to manual expert optimization. Finally, the code and the data are available here.

Mixed-signal analog/digital circuits emulate spiking neurons and synapses with extremely high energy efficiency, an approach known as "neuromorphic engineering". However, analog circuits are sensitive to process-induced variation among transistors in a chip ("device mismatch"). For neuromorphic implementation of Spiking Neural Networks (SNNs), mismatch causes parameter variation between identically-configured neurons and synapses. Each chip exhibits a different distribution of neural parameters, causing deployed networks to respond differently between chips. Current solutions to mitigate mismatch based on per-chip calibration or on-chip learning entail increased design complexity, area and cost, making deployment of neuromorphic devices expensive and difficult. Here we present a supervised learning approach that produces SNNs with high robustness to mismatch and other common sources of noise. Our method trains SNNs to perform temporal classification tasks by mimicking a pre-trained dynamical system, using a local learning rule from non-linear control theory. We demonstrate our method on two tasks requiring memory, and measure the robustness of our approach to several forms of noise and mismatch. We show that our approach is more robust than common alternatives for training SNNs. Our method provides robust deployment of pre-trained networks on mixed-signal neuromorphic hardware, without requiring per-device training or calibration.

A popular approach to selling online advertising is by a waterfall, where a publisher makes sequential price offers to ad networks for an inventory, and chooses the winner in that order. The publisher picks the order and prices to maximize her revenue. A traditional solution is to learn the demand model and then subsequently solve the optimization problem for the given demand model. This will incur a linear regret. We design an online learning algorithm for solving this problem, which interleaves learning and optimization, and prove that this algorithm has sublinear regret. We evaluate the algorithm on both synthetic and real-world data, and show that it quickly learns high quality pricing strategies. This is the first principled study of learning a waterfall design online by sequential experimentation.

The large number of mobile devices, the volume of apps on each phone, and the basic mobility of the devices all mean there is a lot of information being creating in the mobile world. Managing that large volume of information is impossible in a reasonable timeframe using older technologies. Machine learning (ML) is critical to mobile advertising in a number of ways. Advertising is complex even in the older channels of print and broadcast. Cable increased the need for better data to more finely segment the audiences.

The large number of mobile devices, the volume of apps on each phone, and the basic mobility of the devices all mean there is a lot of information being creating in the mobile world. Managing that large volume of information is impossible in a reasonable timeframe using older technologies. Machine learning (ML) is critical to mobile advertising in a number of ways. Advertising is complex even in the older channels of print and broadcast. Cable increased the need for better data to more finely segment the audiences.

In the cost per click (CPC) pricing model, an advertiser pays an ad network only when a user clicks on an ad; in turn, the ad network gives a share of that revenue to the publisher where the ad was impressed. Still, advertisers may be unsatisfied with ad networks charging them for "valueless" clicks, or so-called accidental clicks. [...] Charging advertisers for such clicks is detrimental in the long term as the advertiser may decide to run their campaigns on other ad networks. In addition, machine-learned click models trained to predict which ad will bring the highest revenue may overestimate an ad click-through rate, and as a consequence negatively impacting revenue for both the ad network and the publisher. In this work, we propose a data-driven method to detect accidental clicks from the perspective of the ad network. We collect observations of time spent by users on a large set of ad landing pages - i.e., dwell time. We notice that the majority of per-ad distributions of dwell time fit to a mixture of distributions, where each component may correspond to a particular type of clicks, the first one being accidental. We then estimate dwell time thresholds of accidental clicks from that component. Using our method to identify accidental clicks, we then propose a technique that smoothly discounts the advertiser's cost of accidental clicks at billing time. Experiments conducted on a large dataset of ads served on Yahoo mobile apps confirm that our thresholds are stable over time, and revenue loss in the short term is marginal. We also compare the performance of an existing machine-learned click model trained on all ad clicks with that of the same model trained only on non-accidental clicks. There, we observe an increase in both ad click-through rate (+3.9%) and revenue (+0.2%) on ads served by the Yahoo Gemini network when using the latter. [...]

The following is a guest contributed post from Jeremy Fain, CEO and founder of Cognitiv. A lot of people have asked me about my thoughts for 2018, and what I think the overarching trends in advertising technology and deep learning will be in the coming year. Far be it from me to disappoint my fans – so here's my take on next year's biggest topics: AI has been a buzzword for the past several years, as clearly evidenced by the vast numbers of products claiming to be AI currently on the market. While most of these applications don't really live up to the picture of AI that most people have in their heads of a Jetsons-like robot, there have lately been a series of discoveries, most notably in the field of deep learning, that are sure to have a serious impact on the way that most businesses operate. Deep learning and neural networks are at the heart of some of the most astonishing machine learning developments, from image recognition to the natural language processing that enables gadgets like Amazon's Alexa and Google Home to operate.