Welcome back to, TIME's new twice-weekly newsletter about AI. If you're reading this in your browser, why not subscribe to have the next one delivered straight to your inbox? The three most valuable private companies in the U.S. have big reputations: OpenAI, SpaceX, and Anthropic. But the fourth, Databricks, flies a little more under the radar. This company, which is currently raising funds at a valuation of $134 billion according to reports this week, is a quiet workhorse of the AI revolution.

Databricks, a company that helps big businesses build custom artificial intelligence models, has developed a machine learning trick that can boost the performance of an AI model without the need for clean labelled data. Jonathan Frankle, chief AI scientist at Databricks, spent the past year talking to customers about the key challenges they face in getting AI to work reliably. The problem, Frankle says, is dirty data. "Everybody has some data, and has an idea of what they want to do," Frankle says. But the lack of clean data makes it challenging to fine-tune a model to perform a specific task.. "Nobody shows up with nice, clean fine-tuning data that you can stick into a prompt or an [application programming interface]," for a model.

Jerome Pesenti has a few reasons to celebrate Meta's decision last week to release Llama 3, a powerful open source large language model that anyone can download, run, and build on. Pesenti used to be vice president of artificial intelligence at Meta and says he often pushed the company to consider releasing its technology for others to use and build on. But his main reason to rejoice is that his new startup will get access to an AI model that he says is very close in power to OpenAI's industry-leading text generator GPT-4, but considerably cheaper to run and more open to outside scrutiny and modification. "The release last Friday really feels like a game-changer," Pesenti says. His new company, Sizzle, an AI tutor, currently uses GPT-4 and other AI models, both closed and open, to craft problem sets and curricula for students.

This past Monday, about a dozen engineers and executives at data science and AI company Databricks gathered in conference rooms connected via Zoom to learn if they had succeeded in building a top artificial intelligence language model. The team had spent months, and about 10 million, training DBRX, a large language model similar in design to the one behind OpenAI's ChatGPT. But they wouldn't know how powerful their creation was until results came back from the final tests of its abilities. "We've surpassed everything," Jonathan Frankle, chief neural network architect at Databricks and leader of the team that built DBRX, eventually told the team, which responded with whoops, cheers, and applause emojis. Frankle usually steers clear of caffeine but was taking sips of iced latte after pulling an all-nighter to write up the results.

Former House Speaker Nancy Pelosi has invested up to 5 million in a San Francisco-based company, adding to her successful portfolio of Big Tech. Documents revealed Pelosi's transaction with privately held Databricks, which is a software company based on AI technology, took place on March 3 and was disclosed on March 21. Databricks is just the latest newcomer to Pelosi's long list of companies, but there are eight major names that she has invested 16 million in since 2022. While she has not broken any laws by buying and selling stocks, many Americans and other government officials see the investments as conflicts of interest since she has access to confidential intelligence and the power to impact businesses. Documents revealed Pelosi's transaction with privately held Databricks, which is a software company based on AI technology, took place on March 3 and disclosed on March 21 Databricks is just the latest newcomer to Pelosi's long list of companies, but there are eight major names that she has invested up to 16.1 million in since 2022 Databricks, founded in 2013, raised 500 million last year based on a 43 billion valuation.

In this study, we propose a big data pipeline for cotton bloom detection using a Lambda architecture, which enables real-time and batch processing of data. Our proposed approach leverages Azure resources such as Data Factory, Event Grids, Rest APIs, and Databricks. This work is the first to develop and demonstrate the implementation of such a pipeline for plant phenotyping through Azure's cloud computing service. The proposed pipeline consists of data preprocessing, object detection using a YOLOv5 neural network model trained through Azure AutoML, and visualization of object detection bounding boxes on output images. The trained model achieves a mean Average Precision (mAP) score of 0.96, demonstrating its high performance for cotton bloom classification. We evaluate our Lambda architecture pipeline using 9000 images yielding an optimized runtime of 34 minutes. The results illustrate the scalability of the proposed pipeline as a solution for deep learning object detection, with the potential for further expansion through additional Azure processing cores. This work advances the scientific research field by providing a new method for cotton bloom detection on a large dataset and demonstrates the potential of utilizing cloud computing resources, specifically Azure, for efficient and accurate big data processing in precision agriculture.

Update Apr 12, 2023: We have released Dolly 2.0, licensed for both research and commercial use. See the new blog post here. We show that anyone can take a dated off-the-shelf open source large language model (LLM) and give it magical ChatGPT-like instruction following ability by training it in 30 minutes on one machine, using high-quality training data. Surprisingly, instruction-following does not seem to require the latest or largest models: our model is only 6 billion parameters, compared to 175 billion for GPT-3. We open source the code for our model (Dolly) and show how it can be re-created on Databricks.

Recently, significant public efforts have been directed towards developing low-cost models with capabilities akin to ChatGPT, thereby fostering the growth of open-source conversational models. However, there remains a scarcity of comprehensive and in-depth evaluations of these models' performance. In this study, we examine the influence of training data factors, including quantity, quality, and linguistic distribution, on model performance. Our analysis is grounded in several publicly accessible, high-quality instruction datasets, as well as our own Chinese multi-turn conversations. We assess various models using a evaluation set of 1,000 samples, encompassing nine real-world scenarios. Our goal is to supplement manual evaluations with quantitative analyses, offering valuable insights for the continued advancement of open-source chat models. Furthermore, to enhance the performance and training and inference efficiency of models in the Chinese domain, we extend the vocabulary of LLaMA - the model with the closest open-source performance to proprietary language models like GPT-3 - and conduct secondary pre-training on 3.4B Chinese words. We make our model, data, as well as code publicly available.

Join top executives in San Francisco on July 11-12, to hear how leaders are integrating and optimizing AI investments for success. Databricks, a company specializing in data lakehouse technology, announced on Tuesday a new platform designed for the manufacturing industry. Called lakehouse for manufacturing, the platform aims to unify data and artificial intelligence (AI) for various analytics use cases such as predictive maintenance, quality control and supply chain optimization. The platform builds on Databricks' core data lakehouse platform, which leverages Delta Lake, Apache Spark and MLFlow, open-source projects that enable scalable data processing and machine learning (ML) workflows. The platform also integrates with model serving, a service that Databricks introduced last month to simplify the deployment and management of ML models in production.

In an attempt to open up its technology to a wider audience, enterprise software company Databricks has released Dolly, a large language model and its associated training code under an open-source licence. Despite being based on a much smaller underlying model, the company says it has ChatGPT-like functionality and can be run "in-house". The move was inspired by the success of OpenAI's natural language platform ChatGPT, which became one of the fastest-growing consumer apps within a couple of months of its release in November last year. It has since caused some of the world's largest companies including Microsoft and Google to pivot and release generative and natural language AI tools. "We show that anyone can take a dated off-the-shelf open source LLM and give it magical ChatGPT-like instruction-following ability by training it in 30 minutes on one machine, using high-quality training data," Databricks wrote in a blog post explaining the decision.