In reinforcement learning (RL), the goal is to obtain an optimal policy, for which the optimality criterion is fundamentally important. Two major optimality criteria are average and discounted rewards, where the later is typically considered as an approximation to the former. While the discounted reward is more popular, it is problematic to apply in environments that have no natural notion of discounting. This motivates us to revisit a) the progression of optimality criteria in dynamic programming, b) justification for and complication of an artificial discount factor, and c) benefits of directly maximizing the average reward. Our contributions include a thorough examination of the relationship between average and discounted rewards, as well as a discussion of their pros and cons in RL. We emphasize that average-reward RL methods possess the ingredient and mechanism for developing the general discounting-free optimality criterion (Veinott, 1969) in RL.

In this work we theoretically and experimentally analyze Multi-Agent Advantage Actor-Critic (MA2C) and Independent Advantage Actor-Critic (IA2C), two recently proposed multi-agent reinforcement learning methods that can be applied to control traffic signals in urban areas. The two methods differ in their use of a reward calculated locally or globally and in the management of agents' communication. We analyze the methods theoretically with the framework provided by non-Markov decision processes, which provides useful insights in the analysis of the algorithms. Moreover, we analyze the efficacy and the robustness of the methods experimentally by testing them in two traffic areas in the Bologna (Italy) area, simulated by SUMO, a software tool. The experimental results indicate that MA2C achieves the best performance in the majority of cases, outperforms the alternative method considered, and displays sufficient stability during the learning process.

Quantum physicist Mario Krenn remembers sitting in a café in Vienna in early 2016, poring over computer printouts, trying to make sense of what MELVIN had found. MELVIN was a machine-learning algorithm Krenn had built, a kind of artificial intelligence. Its job was to mix and match the building blocks of standard quantum experiments and find solutions to new problems. And it did find many interesting ones. But there was one that made no sense. "The first thing I thought was, 'My program has a bug, because the solution cannot exist,'" Krenn says.

Many games often share common ideas or aspects between them, such as their rules, controls, or playing area. However, in the context of General Game Playing (GGP) for board games, this area remains under-explored. We propose to formalise the notion of "game concept", inspired by terms generally used by game players and designers. Through the Ludii General Game System, we describe concepts for several levels of abstraction, such as the game itself, the moves played, or the states reached. This new GGP feature associated with the ludeme representation of games opens many new lines of research. The creation of a hyper-agent selector, the transfer of AI learning between games, or explaining AI techniques using game terms, can all be facilitated by the use of game concepts. Other applications which can benefit from game concepts are also discussed, such as the generation of plausible reconstructed rules for incomplete ancient games, or the implementation of a board game recommender system.

This paper considers predicting future statuses of multiple agents in an online fashion by exploiting dynamic interactions in the system. We propose a novel collaborative prediction unit (CoPU), which aggregates the predictions from multiple collaborative predictors according to a collaborative graph. Each collaborative predictor is trained to predict the status of an agent by considering the impact of another agent. The edge weights of the collaborative graph reflect the importance of each predictor. The collaborative graph is adjusted online by multiplicative update, which can be motivated by minimizing an explicit objective. With this objective, we also conduct regret analysis to indicate that, along with training, our CoPU achieves similar performance with the best individual collaborative predictor in hindsight. This theoretical interpretability distinguishes our method from many other graph networks. To progressively refine predictions, multiple CoPUs are stacked to form a collaborative graph neural network. Extensive experiments are conducted on three tasks: online simulated trajectory prediction, online human motion prediction and online traffic speed prediction, and our methods outperform state-of-the-art works on the three tasks by 28.6%, 17.4% and 21.0% on average, respectively.

This paper proposes a novel primal heuristic for Mixed Integer Programs, by employing machine learning techniques. Mixed Integer Programming is a general technique for formulating combinatorial optimization problems. Inside a solver, primal heuristics play a critical role in finding good feasible solutions that enable one to tighten the duality gap from the outset of the Branch-and-Bound algorithm (B&B), greatly improving its performance by pruning the B&B tree aggressively. In this paper, we investigate whether effective primal heuristics can be automatically learned via machine learning. We propose a new method to represent an optimization problem as a graph, and train a Graph Convolutional Network on solved problem instances with known optimal solutions. This in turn can predict the values of decision variables in the optimal solution for an unseen problem instance of a similar type. The prediction of variable solutions is then leveraged by a novel configuration of the B&B method, Probabilistic Branching with guided Depth-first Search (PB-DFS) approach, aiming to find (near-)optimal solutions quickly. The experimental results show that this new heuristic can find better primal solutions at a much earlier stage of the solving process, compared to other state-of-the-art primal heuristics.

I was surprised myself recently when we had a child in the consultation room and I thought'This looks like a bacterial infection,'

Editor's Note: This article is part of a series on external trends. In Part 3, we study technology trends that will impact business in 2021 and beyond. One learning from the pandemic is the need for a new construct and mental model for technology. In almost every industry, some form of technology will become the product or an enabler to it. Those who cannot participate in digital transformation will be left behind.

SCI COMMUN### COVID-19 At least a dozen countries across Africa are facing a steep rise in coronavirus infections, pushing hospitals in several countries to their limits. The highly infectious Delta variant of SARS-CoV-2 is largely driving the increase; it has been documented in at least 14 African countries. Liberia, Rwanda, Namibia, and the Democratic Republic of the Congo all reported record numbers of new COVID-19 cases this week, as countries across the continent were reporting more than 30,000 new cases each day, racing toward the previous peak from early January. At the same time, the continent is facing an acute shortage of vaccines, with less than 1% of the population vaccinated. Officials at the World Health Organization said 18 African countries had used more than 80% of their vaccine stocks, and eight countries had exhausted their supplies. > “It is impossible to look at these findings and not see a reflection of the systemic racism in the U.S.” > > Duke University health services researcher Lesley Curtis , to NPR, on life expectancy declines from 2018 to 2020: 3.9 years for Hispanic Americans, 3.3 for Black Americans, and 1.4 for white Americans. ### Archaeology Merchants of the Bronze Age faced a problem still familiar today: how to know you're getting what you pay for. Historians have long assumed that standard weights—used to measure and trade goods of equivalent value—were handed down from on high, first created by a king or religious authority to collect taxes or tribute. But a new study suggests that more than 3000 years ago, informal networks of merchants established a standardized weight system that started in Mesopotamia and spread across Europe. Researchers analyzed weights from previously excavated sites spanning nearly 5000 kilometers. More than 2000 of the weights—crafted over 2000 years—weighed nearly the same amount: between 8 and 10.5 grams, they report this week in the Proceedings of the National Academy of Sciences . They propose that as traders compared weights at each meeting, a standard emerged, forming the first known common Eurasian market. ### Astronomy Representatives from seven member nations this week gave the go-ahead to start construction later this year on the world's biggest scientific instrument: the twin telescope networks of the Square Kilometre Array (SKA). Once complete in 2028, the nearly €2 billion SKA will comprise hundreds of radio dishes scattered across South Africa and thousands of wire antennas in Western Australia. Combining the signals from this vast array of detectors will give astronomers unprecedented sensitivity and resolution as they search for the universe's first stars and galaxies and seek to understand gravity and cosmic magnetism. The green light means the U.K.-based SKA Observatory can begin to award industrial contracts. ### Astrophysics Gravitational wave astronomers have twice spotted a black hole consuming a neutron star, researchers with the Laser Interferometer Gravitational-Wave Observatory (LIGO) in the United States and the Virgo Observatory in Italy announced this week. LIGO and Virgo had previously spotted dozens of pairs of black holes spiraling together and two pairs of merging neutron stars—including one that set off a spectacular explosion seen by telescopes of all kinds in 2017. Astronomers saw no similar explosions from the newly detected black hole-neutron star mergers, either because they were too far away or because the black holes swallowed the neutron stars whole, a possibility that could put a damper on hopes that such a collision might someday lay bare the innards of a neutron star. ### Research integrity Although researchers have valid reasons to reuse their text across papers—in literature reviews or methods descriptions, for example—peers often frown on this practice as “self-plagiarism.” Some oversight bodies, including the Committee on Publication Ethics, have considered the practice acceptable in some circumstances. This week, the Text Recycling Research Project, based at Duke University and funded by the U.S. National Science Foundation, released new guidance on the finer points, drawing on advice from journal publishers and other specialists. The document describes when the practice is both ethical and legal and how to present reused text transparently. One aim is to ease the workload of authors who are currently forced to reword passages unnecessarily, purely to avoid the appearance of self-plagiarism, says project leader Cary Moskovitz. ### Scientific community The U.S. National Academy of Sciences (NAS) last week expelled evolutionary biologist Francisco Ayala from its ranks 3 years after he was found to have sexually harassed women colleagues. Ayala resigned from the University of California, Irvine, in 2018 after a university investigation found him guilty of sexual harassment. Ayala declined to comment on NAS's action, but has denied the allegations against him, which included making sexually suggestive comments and inviting a junior professor to sit on his lap. Women who filed complaints with the university over Ayala's behavior applauded the move, but charged that NAS's process was too slow. Ayala is the second NAS member to be ousted over sexual harassment allegations since the academy revised its bylaws 2 years ago to allow members to be removed if they violate its code of conduct. ### Public health The World Health Organization (WHO) on 30 June certified China as free of malaria, making it the 40th country—and the most populous one by far—to gain that status. In the 1940s, China had an estimated 30 million malaria cases and 300,000 deaths annually, but antimalarial drugs, insecticides, and other countermeasures brought cases to zero in 2017. Along the way, pharmaceutical chemist Tu Youyou bagged a Nobel Prize for isolating a powerful malaria drug, artemisinin, in sweet wormwood ( Artemisia annua ), a plant used in traditional Chinese medicine. “China's ability to think outside the box served the country well in its own response to malaria,” Pedro Alonso, director of WHO's Global Malaria Programme, said in a statement this week. The last three countries awarded WHO's malaria-free status were El Salvador, in February, and Algeria and Argentina, both in 2019. ### Publishing The controversial journal impact factor will be supplemented by a new metric that allows accurate comparisons of journal citation rates in different disciplines, its creator, Clarivate Analytics, said last week. The Journal Citation Indicator (JCI), released on 30 June as part of Clarivate's 2021 update to its Journal Citation Reports database, covers a wider range of journals, measured over a longer time period, than the company's existing impact factor. The impact metric captures how many citations a journal accumulated per article published over a 2-year period; the new metric is an average that attempts to take into account the substantially different rates of publication and citation in different fields, according to Clarivate. The JCI is a step forward but has important limitations, says Henk Moed, a bibliometrician at the Sapienza University of Rome. He says that, like impact factors, the new metric will be problematic if applied to individual researchers. ### Public health In its inaugural award ceremony last week, a coalition of public, private, and philanthropic organizations known as the Trinity Challenge gave out a total of $8 million to eight projects focused on preventing the next pandemic. Launched in September 2020 with support from 42 organizations including the Bill & Melinda Gates Foundation, Google, GlaxoSmithKline, and Imperial College London, the challenge recognizes projects that use data and analytics to respond to health emergencies. The $1.8 million grand prize went to a project called Participatory One Health Disease Detection, which aims to help farmers in Asia and Africa identify and report sick livestock via a mobile app to prevent the spread of disease among animals and to humans. Second place prizes of $1.4 million each will support an effort to help health authorities in West Africa forecast emerging diseases and a project that uses artificial intelligence to spot infectious disease outbreaks using routine blood tests. ### Space science An expert panel at the U.S. National Academies of Sciences, Engineering, and Medicine (NASEM) is encouraging NASA to push forward with a proposed change to limits on radiation exposure that would place women astronauts on equal footing with their male counterparts. Current standards limit astronauts to a radiation level that increases their risk of exposure-induced death by 3%, a metric that varies based on age and sex. A change under consideration at NASA, endorsed in a NASEM report released last week, would limit all astronauts to 600 millisieverts of radiation over their careers. The report also proposes a color-coded system to communicate the risks of longer missions and proposes that astronauts sign a waiver if a mission is expected to exceed their radiation limit. ### Science policy The U.S. House of Representatives this week overwhelmingly approved two bills that would authorize massive spending increases at the National Science Foundation (NSF) and the Department of Energy's Office of Science. H.R. 2225 calls for more than doubling NSF's annual budget of $8.5 billion to $17.9 billion by 2026, and H.R. 3593 would give the Office of Science a 63% boost, to $11.1 billion, over the same period. The bills represent a slimmer alternative to the sprawling and more costly one passed last month by the Senate to address the growing scientific, economic, and military threat of China. Science lobbyists generally prefer the House bills' approach to tightening research security and correcting the uneven geographic distribution of funding. Reconciling these competing visions could take months, and separate legislation will be needed to determine the 2022 budgets for each agency. 267 million —People worldwide who live on land less than 2 meters above sea level—areas at greatest flood risk from sea level rise. Researchers predict an increase to 410 million people by 2100. ( Nature Communications ) 250–350 million years —Estimated age of the universe at “cosmic dawn,” when the first stars switched on, based on new telescope observations of the most distant known galaxies. ( Monthly Notices of the Royal Astronomical Society )

The winners of three IJCAI awards have been announced. These three distinctions are: the Award for Research Excellence, the John McCarthy Award and the Computers and Thought Award. The Research Excellence award is given to a scientist who has carried out a program of research of consistently high quality throughout an entire career yielding several substantial results. The winner of the 2021 Award for Research Excellence is Richard Sutton (University of Alberta). Richard is recognized for his fundamental contributions to the foundation of reinforcement learning.