Large Language Models have shown prominent capabilities in generating functional code from natural language descriptions. However, a standardized way to evaluate these capabilities in an objective and unbiased manner is still to be found. In this paper we review the current evaluation methods available to this end, and run a new evaluation of the performance of one state-of-the-art model (GPT4-o-mini) in solving curated coding challenges in 8 programming languages, obtained from Codewars, a software development community. Our analysis shows that the chance of success of the model has a positive correlation with the task difficulty, the popularity of the programming language being used and the time elapsed since the publication of the challenge. A further approximate explanatory analysis in terms of high-level features hints that while 46.6% of the model performance could be attributed to task difficulty, a 37.4% seems to be related to leakage of the challenge solutions into the model training set, while the remaining 16% depends on the programming language. These results suggest that current evaluation methodologies might be overestimating the actual skill of Large Language Models for generating functional code.

In the burgeoning field of artificial intelligence (AI), understanding the capabilities and limitations of programming-oriented models is crucial. This paper presents a novel evaluation of the programming proficiency of Generative Pretrained Transformer (GPT) models, specifically GPT-3.5 and GPT-4, against coding problems of varying difficulty levels drawn from Codewars. The experiments reveal a distinct boundary at the 3kyu level, beyond which these GPT models struggle to provide solutions. These findings led to the proposal of a measure for coding problem complexity that incorporates both problem difficulty and the time required for solution. The research emphasizes the need for validation and creative thinking capabilities in AI models to better emulate human problem-solving techniques. Future work aims to refine this proposed complexity measure, enhance AI models with these suggested capabilities, and develop an objective measure for programming problem difficulty. The results of this research offer invaluable insights for improving AI programming capabilities and advancing the frontier of AI problem-solving abilities.

In previous blogs, we have covered Eclipse Collections and Deep Learning. Recently, a couple of the highly recommended katas for Eclipse Collections have been revamped to include "pet" and "fruit" emojis for a little bit of extra fun. What could be better than Learning Eclipse Collections? First, we create a PetType enum with the emoji toString, and then Pet and Person records. We'll populate a people list as is done in the kata.

U.S. troops in eastern Syria came under rocket attack Monday, with no reported casualties, one day after U.S. Air Force planes carried out airstrikes near the Iraq-Syria border against what the Pentagon said were facilities used by Iran-backed militia groups to support drone strikes inside Iraq. Iraq's military condemned the U.S. airstrikes, and the militia groups called for revenge against the United States. Pentagon Press Secretary John Kirby said the militias were using the facilities to launch unmanned aerial vehicle attacks against U.S. troops in Iraq. It was the second time the administration has taken military action in the region since Biden took over earlier this year. There was no indication that Sunday's attacks were meant as the start of a wider, sustained U.S. air campaign in the border region.

Indonesian artificial intelligence based chatbot platform Kata.ai has reportedly secured $3.5 Mn in a Series A funding round led by the Taiwanese Trans-Pacific Technology Fund (TPTF). The investors that participated in the round are Korea-based Access Ventures, MDI Ventures, VPG Asia, and Convergence Ventures, the VC arm of Indonesian state-run telecommunications provider Telkom. The Series A funding round also saw participation from Red Sails Investment and angel investor Eddy Chan. As per reports, TPTF principal Barry Lee will be joined Kata.ai as a board member. With the newly-raised capital, the company is looking to expand operations to emerging markets in Taiwan and other Southeast Asian countries.