For each benchmark, we generate three different training datasets. If the dataset is a sample, then what is the larger set? Is the samplerepresentativeofthe larger set(e.g., geographic coverage)? The provided datasets are self-contained and will remain constant. However, more datasets could be generated using the proposed benchmarking platform.

In this paper, we propose an architecture to harness the collective knowledge of multiple trained LLMs to create a new state-of-the-art. At the core of this framework is a LLM-based orchestrator that is adept at picking the right underlying LLM experts for optimal task execution. Inspired by self-play in reinforcement learning, we created a loop of query generation, orchestration, and evaluation to generate training data for the orchestrator. Our evaluation focused on the MMLU benchmark, employing models with 7B, 13B, and 34B parameters available on Hugging Face. The results demonstrate new state-of-the-art open-source models: Our Leeroo orchestrator achieves performance on par with the Mixtral model while incurring only two-thirds of its cost. Moreover, increasing the allowed cost surpasses Mixtral's accuracy by over 5% at the same cost level, reaching an accuracy of 75.9%. Further enhancements were observed when integrating GPT4 into the underlying model pool. The Leeroo orchestrator nearly matches GPT4's performance at half the cost and even exceeds GPT4's results with a 25% cost reduction. These findings illustrate the potential of our architecture in creating state-of-the-art and cost-effective LLMs by optimizing the synergy between multiple LLMs to achieve superior performance outcomes.

PyTorch Lightning has its humble beginnings as a project that I developed during the first few years of my Ph.D. at NYU CILVR and later at Facebook AI Research. At NYU it gained the powers of rapid iteration and standardization that makes Lightning a pleasure to work with today -- it standardizes AI research code so everyone's code can be formatted the same way, and thus it becomes more readable and reproducible. At FAIR it learned how to train massive neural networks across hundreds of GPUs. But had I remained the only developer of the project it would be nowhere near where it is today as a quickly rising favorite for deep learning research. Our first non-facebook contributor Jirka, forced much-needed formatting and structuring to the internals.

Finally! TensorFlow 2.0 is out! We've been an early user since the alpha release and so far we really love it. All the new features such as tf.data, eager execution, tf.function etc. definitely provide a much better user experience compared to TensorFlow 1.x. So if you haven't used it yet, you should pip install it now!

If you're attending the All Things Open conference in Raleigh, NC this year be sure to check out our Lightning Talk series on Tuesday, October 23. This is an amazing line-up of quick talks you won't want to miss. Speakers have five minutes to enlighten the audience about an open source topic they are passionate about. We've got everything from containers to AI and Itseo to Blockchain, Raspberry Pi and more. Grab your lunch, find a seat, warm up your Twitter fingers, and get ready for the fastest hour at All Things Open.

Escaping the trough of disillusionment for virtual and augmented reality [TechCrunch]: S. Somasegar writes about AR/VR's long road to mass adoption, stating, "Gartner has placed VR within its tech hype cycle as precariously struggling out of the trough of disillusionment, described as a period of waning interest as'experiments and implementations fail to deliver.'" However, while Somasegar says mainstream adoption is still likely three to five years away, "We still believe that in twenty years, VR will be a ubiquitous force and as pervasive and transformative as the internet was in the 90s or the smartphone was in the 2000s. Every 2D interface will be re-imagined and re-architected for 3D." He goes on to outline some of the big opportunities in AR/VR just waiting to be tapped by "those brave enough to weather the trough of disillusionment." Google artificial intelligence guru says A.I. won't kill jobs [Fortune]: Mustafa Suleyman, co-founder of artificial intelligence startup DeepMind, recently addressed some common concerns around AI at an O'Reilly event, and Jonathan Vanian recapped the highlights in Fortune this week.