There are two main challenges in solving AFT: 1) efficient feature transformation in a massive discrete search space; 2) robust feature transformation in an open learning environment.

Large multimodal models (LMMs) have demonstrated significant potential as generalists in vision-language (VL) tasks. However, there remains a significant gap between state-of-the-art LMMs and human performance when it comes to complex tasks that require a combination of fundamental VL capabilities, as well as tasks involving the grounding of complex instructions. To thoroughly investigate the human-LMM gap and its underlying causes, we propose MOAT, a diverse benchmark with complex real-world VL tasks that are challenging for LMMs. Specifically, the tasks in MOAT require LMMs to engage in generalist problem solving by integrating fundamental VL capabilities such as reading text, counting, understanding spatial relations, grounding textual and visual instructions, etc. All these abilities fit into a taxonomy proposed by us that contains 10 fundamental VL capabilities, enabling MOAT to provide a fine-grained view of LMMs' strengths and weaknesses. Besides, MOAT is the first benchmark to explicitly evaluate LMMs' ability to ground complex text and visual instructions, which is essential to many real-world applications. We evaluate over 20 proprietary and open source LMMs, as well as humans, on MOAT, and found that humans achieved 82.7% accuracy while the best performing LMM (OpenAI o1) achieved only 38.8%. To guide future model development, we analyze common trends in our results and discuss the underlying causes of observed performance gaps between LMMs and humans, focusing on which VL capability forms the bottleneck in complex tasks, whether test time scaling improves performance on MOAT, and how tiling harms LMMs' capability to count. Code and data are available at https://cambrian-yzt.github.io/MOAT.

Feature transformation aims to generate new pattern-discriminative feature space from original features to improve downstream machine learning (ML) task performances. However, the discrete search space for the optimal feature explosively grows on the basis of combinations of features and operations from low-order forms to high-order forms. Existing methods, such as exhaustive search, expansion reduction, evolutionary algorithms, reinforcement learning, and iterative greedy, suffer from large search space. Overly emphasizing efficiency in algorithm design usually sacrifices stability or robustness. To fundamentally fill this gap, we reformulate discrete feature transformation as a continuous space optimization task and develop an embedding-optimization-reconstruction framework. This framework includes four steps: 1) reinforcement-enhanced data preparation, aiming to prepare high-quality transformation-accuracy training data; 2) feature transformation operation sequence embedding, intending to encapsulate the knowledge of prepared training data within a continuous space; 3) gradient-steered optimal embedding search, dedicating to uncover potentially superior embeddings within the learned space; 4) transformation operation sequence reconstruction, striving to reproduce the feature transformation solution to pinpoint the optimal feature space.

Do large language models create a moat? In this given hype, what type of businesses should you invest your time and money in? While the technology of Large Language Models (LLMs) is new, the approach to analyzing the business moat is still the same. AI-driven businesses have a combination of either a model moat, a data moat, or the brand moat. The new LLMs, like OpenAI's model, can give an advantage in all areas, but it is by no means a given.

We've all come to appreciate that AI and machine learning are the magic sauce powering large-scale consumer internet properties. Facebook, Amazon and Instacart boast enormous datasets and huge user counts. Common wisdom suggests that this scale advantage is a powerful competitive moat; it enables far better personalization, recommendations and ultimately, a better user experience. In this article, I will show you that this moat is shallower than it seems; and that alternative approaches to personalization can produce outstanding outcomes without relying on billions of data points. How do Instagram and TikTok understand what you like and don't like?

Today, companies everywhere are generating unprecedented amounts of data. While data has always grown naturally as a byproduct of economic and business activity, these days, as more and more of our personal and work lives take place online, humans are creating an abundance of data daily. In fact, 90% of all the world's internet data has been created since 2016. For more than a decade, only the so-called FAANG companies (Facebook, Apple, Amazon, Netflix, and Google) were in the position to take advantage of collecting vast amounts of data at scale. For these companies, data is the prime product and inherent to their value proposition, so they invested early in AI teams, servers, network infrastructure, and more.

Pop Quiz, which company on the planet has the most number of customers? With 302 million customers, it's towering above the next largest one which is Apple iTunes with 225 million customers. Does business growth, revenue, and brand depend on the number of customers an organization has? The more customers you reach the more robust is the moat you build for your business. So if your organization has to grow large and accumulate a lot of customers and build a moat that is impenetrable around your business, you need to start acquiring more customers.