Because LMs now surpass the performance of most humans except domain experts, evaluating them becomes increasingly challenging.

Because LMs now surpass the performance of most humans except domain experts, evaluating them becomes increasingly challenging.

Since language models (LMs) now outperform average humans on many challenging tasks, it has become increasingly difficult to develop challenging, high-quality, and realistic evaluations. We address this issue by examining LMs' capabilities to generate code for solving real scientific research problems. Incorporating input from scientists and AI researchers in 16 diverse natural science sub-fields, including mathematics, physics, chemistry, biology, and materials science, we created a scientist-curated coding benchmark, SciCode. The problems in SciCode naturally factorize into multiple subproblems, each involving knowledge recall, reasoning, and code synthesis. In total, SciCode contains 338 subproblems decomposed from 80 challenging main problems. It offers optional descriptions specifying useful scientific background information and scientist-annotated gold-standard solutions and test cases for evaluation. Claude3.5-Sonnet, the best-performing model among those tested, can solve only 4.6% of the problems in the most realistic setting. We believe that SciCode demonstrates both contemporary LMs' progress towards becoming helpful scientific assistants and sheds light on the development and evaluation of scientific AI in the future.

A big part of our product at Lumen5 is our video rendering engine (we're a video creation tool, after all). A couple of years ago, we started experimenting with WebGL as a potential new way to render videos. It's totally transformed the way we think about JavaScript development, and opened our eyes to a completely different web paradigm. When we first started, it wasn't clear that WebGL was the way to go for us. It took a lot of research into the different alternatives before we broke ground on our existing render engine.

What is new is how quickly malicious actors can spread disinformation when the world is tightly connected across social networks and internet news sites. We can give up on the problem and rely on the platforms themselves to fact-check stories or posts and screen out disinformation--or we can build new tools to help people identify disinformation as soon as it crosses their screens. Preslav Nakov is a computer scientist at the Qatar Computing Research Institute in Doha specializing in speech and language processing. He leads a project using machine learning to assess the reliability of media sources. That allows his team to gather news articles alongside signals about their trustworthiness and political biases, all in a Google News-like format. "You cannot possibly fact-check every single claim in the world," Nakov explains. Instead, focus on the source. "I like to say that you can fact-check the fake news before it was even written." His team's tool, called the Tanbih News Aggregator, is available in Arabic and English and gathers articles in areas such as business, politics, sports, science and technology, and covid-19. Business Lab is hosted by Laurel Ruma, editorial director of Insights, the custom publishing division of MIT Technology Review. The show is a production of MIT Technology Review, with production help from Collective Next. This podcast was produced in partnership with the Qatar Foundation. "Even the best AI for spotting fake news is still terrible," MIT Technology Review, October 3, 2018 Laurel Ruma: From MIT Technology Review, I'm Laurel Ruma, and this is Business Lab, the show that helps business leaders make sense of new technologies coming out of the lab and into the marketplace.

In the beginning, there were files. Later there were navigational databases based on structured files. Then there were IMS and CODASYL, and around 40 years ago we had some of the first relational databases. Throughout much of the 1980s and 1990s "database" strictly meant "relational database." Then with the growing popularity of object-oriented programming languages, some thought the solution to the "impedance mismatch" of object-oriented languages and relational databases was to map objects in the database. Thus we ended up with "object-oriented databases."

Is the majority of us starting to assume that the planet is the primary source of knowledge by actually splitting the system or machines and humans as we get close to substantial technological progress? While this has become a subject in science fiction, there's increasingly worry about their ability for certain people such as Stephen Hawking and Elon Musk. Yet this case of philosophy is still overly neurotic. Both these arguments are merely an assertion that has little justification and that at least remains so for now. Nonetheless, if you question what the AI recruitment issue is all you have to do is evaluate the current growth process, which can achieve this solution. This report should be of interest to you in this case.

Due to the exponential growth and development of technology in the last couple decades, the computing world can be broken up into three core technologies: cognitive (AI), connected (IoT), and distributed (Blockchain/Digital-Currency) computing. The'trinity' works together because each type of computing tackles the others' weaknesses and drawbacks. For instance, the main problems of cognitive and connected computing regarding ethics, security, and privacy can be addressed by distributed computing. The main problems of limited hardware and detailed data for distributed and cognitive computing, respectively, can be addressed by connected computing. The main problems of customer protection and automation limitations for distributed and connected computing, respectively, can be addressed by cognitive computing.

A Stanford undergrad's AI-based chatbot has already helped us with our parking tickets and various legal issues, but now his DoNotPay bot is taking on an even bigger, trickier issue. After receiving acclaim for the bot, which challenged over 160,000 tickets, Joshua Browder taught the program how to help homeless people in the UK claim their right to public housing. The user simply asks for help, and the bot will ask them a series of questions to determine how best to help them. Usually this will involve crafting a claim letter, which the bot fills in with the information that's been provided. In the end, people can save hundreds of dollars in legal advising fees when they need that money the most.

In the efficient social choice problem, the goal is to assign values, subject to side constraints, to a set of variables to maximize the total utility across a population of agents, where each agent has private information about its utility function. In this paper we model the social choice problem as a distributed constraint optimization problem (DCOP), in which each agent can communicate with other agents that share an interest in one or more variables. Whereas existing DCOP algorithms can be easily manipulated by an agent, either by misreporting private information or deviating from the algorithm, we introduce M-DPOP, the first DCOP algorithm that provides a faithful distributed implementation for efficient social choice. This provides a concrete example of how the methods of mechanism design can be unified with those of distributed optimization. Faithfulness ensures that no agent can benefit by unilaterally deviating from any aspect of the protocol, neither information-revelation, computation, nor communication, and whatever the private information of other agents. We allow for payments by agents to a central bank, which is the only central authoritythat we require. To achieve faithfulness, we carefully integrate the Vickrey-Clarke-Groves (VCG) mechanism with the DPOP algorithm, such that each agent is only asked to perform computation, report information, and send messages that is in its own best interest. Determining agent i's payment requires solving the social choice problem without agent i. Here, we present a method to reuse computation performed in solving the main problem in a way that is robust against manipulation by the excluded agent. Experimental results on structured problems show that as much as 87% of the computation required for solving the marginal problems can be avoided by re-use, providing very good scalability in the number of agents. On unstructured problems, we observe a sensitivity of M-DPOP to the density of the problem, and we show that reusability decreases from almost 100% for very sparse problems to around 20% for highly connected problems. We close with a discussion of the features of DCOP that enable faithful implementations in this problem, the challenge of reusing computation from the main problem to marginal problems in other algorithms such as ADOPT and OptAPO, and the prospect of methods to avoid the welfare loss that can occur because of the transfer of payments to the bank.