In this work, we propose an information-directed objective for infinite-horizon reinforcement learning (RL), called the occupancy information ratio (OIR), inspired by the information ratio objectives used in previous information-directed sampling schemes for multi-armed bandits and Markov decision processes as well as recent advances in general utility RL. The OIR, comprised of a ratio between the average cost of a policy and the entropy of its induced state occupancy measure, enjoys rich underlying structure and presents an objective to which scalable, model-free policy search methods naturally apply. Specifically, we show by leveraging connections between quasiconcave optimization and the linear programming theory for Markov decision processes that the OIR problem can be transformed and solved via concave programming methods when the underlying model is known. Since model knowledge is typically lacking in practice, we lay the foundations for model-free OIR policy search methods by establishing a corresponding policy gradient theorem. Building on this result, we subsequently derive REINFORCE- and actor-critic-style algorithms for solving the OIR problem in policy parameter space. Crucially, exploiting the powerful hidden quasiconcavity property implied by the concave programming transformation of the OIR problem, we establish finite-time convergence of the REINFORCE-style scheme to global optimality and asymptotic convergence of the actor-critic-style scheme to (near) global optimality under suitable conditions. Finally, we experimentally illustrate the utility of OIR-based methods over vanilla methods in sparse-reward settings, supporting the OIR as an alternative to existing RL objectives.

Reinforcement learning (RL) is widely used in applications where one needs to perform sequential decision-making while interacting with the environment. The standard RL problem with safety constraints is generally mathematically modeled by constrained Markov Decision Processes (CMDP), which is linear in objective and rules in occupancy measure space, where the problem becomes challenging in the case where the model is unknown apriori. The problem further becomes challenging when the decision requirement includes optimizing a concave utility while satisfying some nonlinear safety constraints. To solve such a nonlinear problem, we propose a conservative stochastic primal-dual algorithm (CSPDA) via a randomized primal-dual approach. By leveraging a generative model, we prove that CSPDA not only exhibits Õ(1/ε2)sample complexity, but also achieves zero constraint violations for the concave utility CMDP. Compared with the previous works, the best available sample complexity for CMDP with zero constraint violation is Õ(1/ε5). Hence, the proposed algorithm provides a significant improvement as compared to the state-of-the-art.

Avodah, a transformative SaaS company powering artificial intelligence (AI) capabilities that operates healthcare division AvodahMed, added four new members to the AvodahMed Medical Advisory Council. "We are delighted to welcome Dr. Koppel, Dr. Lytle, Will, and Tony to our multidisciplinary team of the most established and distinguished professionals" Mark Koppel, M.D., Bruce Lytle, M.D., Will Rideout, and Anthony Black will join existing Council members whose collective role is to provide strategic guidance, clinical leadership, and scientific and ethical direction to advance AvodahMed. The Council members will focus on the company's Nsight conversational AI solution's development roadmap designed to detect and boost a medical practice's care management, cost savings, and revenue-boosting opportunities. The solution is also aimed at reducing physician burnout. Mark Koppel, M.D., is a seasoned executive with nearly 20 years of experience in the business of medicine and healthcare.

In recent years, reinforcement learning (RL) systems with general goals beyond a cumulative sum of rewards have gained traction, such as in constrained problems, exploration, and acting upon prior experiences. In this paper, we consider policy optimization in Markov Decision Problems, where the objective is a general concave utility function of the state-action occupancy measure, which subsumes several of the aforementioned examples as special cases. Such generality invalidates the Bellman equation. As this means that dynamic programming no longer works, we focus on direct policy search. Analogously to the Policy Gradient Theorem \cite{sutton2000policy} available for RL with cumulative rewards, we derive a new Variational Policy Gradient Theorem for RL with general utilities, which establishes that the parametrized policy gradient may be obtained as the solution of a stochastic saddle point problem involving the Fenchel dual of the utility function. We develop a variational Monte Carlo gradient estimation algorithm to compute the policy gradient based on sample paths. We prove that the variational policy gradient scheme converges globally to the optimal policy for the general objective, though the optimization problem is nonconvex. We also establish its rate of convergence of the order $O(1/t)$ by exploiting the hidden convexity of the problem, and proves that it converges exponentially when the problem admits hidden strong convexity. Our analysis applies to the standard RL problem with cumulative rewards as a special case, in which case our result improves the available convergence rate.

We study the estimation of risk-sensitive policies in reinforcement learning problems defined by a Markov Decision Process (MDPs) whose state and action spaces are countably finite. Prior efforts are predominately afflicted by computational challenges associated with the fact that risk-sensitive MDPs are time-inconsistent. To ameliorate this issue, we propose a new definition of risk, which we call caution, as a penalty function added to the dual objective of the linear programming (LP) formulation of reinforcement learning. The caution measures the distributional risk of a policy, which is a function of the policy's long-term state occupancy distribution. To solve this problem in an online model-free manner, we propose a stochastic variant of primal-dual method that uses Kullback-Lieber (KL) divergence as its proximal term. We establish that the number of iterations/samples required to attain approximately optimal solutions of this scheme matches tight dependencies on the cardinality of the state and action spaces, but differs in its dependence on the infinity norm of the gradient of the risk measure. Experiments demonstrate the merits of this approach for improving the reliability of reward accumulation without additional computational burdens.

Wine growers have a neat, if unusual, trick for making more flavorful wine--don't water the vines. Let the vines go dry right before harvest, and they will yield smaller grapes with more skin and less juice. Smaller grapes produce wine with a deeper color and more complex flavor. Trinchero Family Estates in Napa Valley, California wanted to make sure it was watering its grapes just the right amount, so they worked with Ceres Imaging to map their fields. Ceres used fixed-wing aircrafts to capture color, thermal, and infrared images of the vineyard, and they used artificial intelligence to analyze those images to see if the wine producer was overwatering its grapes.

Artificial intelligence has allowed scientists to make significant progress in cracking a mysterious ancient text, the meaning of which has eluded scholars for centuries. Dated to the 15th century, the Voynich manuscript is a hand-written text in an unknown script, accompanied by pictures of plants...

Since its discovery over a hundred years ago, the 240-page Voynich manuscript, filled with seemingly coded language and inscrutable illustrations, has confounded linguists and cryptographers. Using artificial intelligence, Canadian researchers have taken a huge step forward in unraveling the document's hidden meaning. Named after Wilfrid Voynich, the Polish book dealer who procured the manuscript in 1912, the document is written in an unknown script that encodes an unknown language--a double-whammy of unknowns that has, until this point, been impossible to interpret. The Voynich manuscript contains hundreds of fragile pages, some missing, with hand-written text going from left to right. Most pages are adorned with illustrations of diagrams, including plants, nude figures, and astronomical symbols.

Artificial intelligence has allowed scientists to make significant progress in cracking a mysterious ancient text, the meaning of which has eluded scholars for centuries. Dated to the 15th century, the Voynich manuscript is a hand-written text in an unknown script, accompanied by pictures of plants, astronomical observations and nude figures. Since its discovery in the 19th century, many historians and cryptographers have attempted to unravel its meaning - including code breakers during the Second World War - but none have been successful. While some have written the Voynich manuscript off as a hoax, use of modern techniques has previously suggested the presence of "a genuine message" inside the book. Now, computer scientists at the University of Alberta have applied artificial intelligence to the text, with their first goal to establish its language of origin. They used text from the Universal Declaration of Human Rights in 380 languages to "train" their system and then ran their algorithms, which determined the most likely language for the document was Hebrew.

Tel Aviv-based startup Prospera has raised a $15 million Series B to expand the scope of its technology, which uses computer vision and artificial intelligence to help farmers analyze data gathered from their fields. The round was led by Qualcomm Ventures, with participation from Cisco Investments, ICV and returning investor Bessemer Venture Partners, and brings Prospera's total funding to $22 million (its Series A was covered by TechCrunch in July 2016). The startup's new capital will be used enter more global markets, add people to its delivery and customer-facing teams and broaden its services to cover more crops, "including making a key shift from indoor farms to outdoor farms, which has huge implications given that 40 percent of U.S. land is farmland," says co-founder and CEO Daniel Koppel. Since its Series A, Prospera has added new customers in Europe, Mexico, and the U.S. and now claims thousands of users, including produce growers for Walmart, Tesco, Sainbury's, and Aldi. Its technology has also evolved from its focus on automatically detecting pests and diseases to "every aspect of farm production," says Koppel.