Instruction-following is particularly crucial for large language models (LLMs) to support diverse user requests. While existing work has made progress in aligning LLMs with human preferences, evaluating their capabilities on instruction following remains a challenge due to complexity and diversity of real-world user instructions. While existing evaluation methods focus on general skills, they suffer from two main shortcomings, i.e., lack of fine-grained task-level evaluation and reliance on singular instruction expression. To address these problems, this paper introduces DINGO, a fine-grained and diverse instruction-following evaluation dataset that has two main advantages: (1) DINGO is based on a manual annotated, fine-grained and multi-level category tree with 130 nodes derived from real-world user requests; (2) DINGO includes diverse instructions, generated by both GPT-4 and human experts. Through extensive experiments, we demonstrate that DINGO can not only provide more challenging and comprehensive evaluation for LLMs, but also provide task-level fine-grained directions to further improve LLMs.

Managing the semantic quality of the categorization in large textual datasets, such as Wikipedia, presents significant challenges in terms of complexity and cost. In this paper, we propose leveraging transformer models to distill semantic information from texts in the Wikipedia dataset and its associated categories into a latent space. We then explore different approaches based on these encodings to assess and enhance the semantic identity of the categories. Our graphical approach is powered by Convex Hull, while we utilize Hierarchical Navigable Small Worlds (HNSWs) for the hierarchical approach. As a solution to the information loss caused by the dimensionality reduction, we modulate the following mathematical solution: an exponential decay function driven by the Euclidean distances between the high-dimensional encodings of the textual categories. This function represents a filter built around a contextual category and retrieves items with a certain Reconsideration Probability (RP). Retrieving high-RP items serves as a tool for database administrators to improve data groupings by providing recommendations and identifying outliers within a contextual framework.

Hello! My name is Alex, and I work at Ozon in the Product Matching team. Ozon is an e-commerce company that offers customers to buy goods from different sellers. Every day we deal with dozens of millions of products, and our task is to identify and compare similar offers (find matches) on our site, in order to collect different sellers' offers into one product card. Every product has the following information: pictures, title, description and additional attributes. We want to retrieve and process all this information for dealing with different tasks, while it is especially important for the product matching team.

Hello! My name is Alex, and I work at Ozon in the Product Matching team. Ozon is an e-commerce company that offers customers to buy goods from different sellers. Every day we deal with dozens of millions of products, and our task is to identify and compare similar offers (find matches) on our site, in order to collect different sellers' offers into one product card. Every product has the following information: pictures, title, description and additional attributes. We want to retrieve and process all this information for dealing with different tasks, while it is especially important for the product matching team.

This paper presents a clustering algorithm that is an extension of the Category Trees algorithm. Category Trees is a clustering method that creates tree structures that branch on category type and not feature. The development in this paper is to consider a secondary order of clustering that is not the category to which the data row belongs, but the tree, representing a single classifier, that it is eventually clustered with. Each tree branches to store subsets of other categories, but the rows in those subsets may also be related. This paper is therefore concerned with looking at that second level of clustering between the other category subsets, to try to determine if there is any consistency over it. It is argued that Principal Components may be a related and reciprocal type of structure, and there is an even bigger question about the relation between exemplars and principal components, in general. The theory is demonstrated using the Portugal Forest Fires dataset as a case study. The distributed nature of that dataset can artificially create the tree categories and the output criterion can also be determined in an automatic and arbitrary way, leading to a flexible and dynamic clustering mechanism.

Numerical combination of LDA and NMF cascaded with W2V to categorize 1M multi-lingual records into a 275-node, 5-level deep category tree. Mixed were the feelings when I got to know the Top 3 position in Hackathon was a narrow miss by 1 mark. While it was kinda re-assuring to stand 4th among Finalist Teams, despite myself participating alone, the void of one passionate team partner, was quite disquieting. The Hackathon challenge was to multi-categorize 1M multi-lingual articles with meta-information, at high precision. The meta-information denotes the category-tree information hidden in the URL text sequence and also the title of each article.

This is the second part in a series where we analyze thousands of articles from tech news sites in order to get insights and trends about startups. So, if a sample mentions an IoT pacemaker startup, it should get the IoT tag in addition to the Health tag. Tagging the data was a similar process to the previous classifier, except that this time we took special care in tagging every sample with all the relevant categories. At this point, we are ready to repeat the same experiment we did in the previous post: classifying 100 articles and seeing what happens.

Building a quality machine learning model for text classification can be a challenging process. You need to build a training dataset, test different parameters for your model, fix the confusions, among other things. On this post, we will describe the process on how you can successfully train text classifiers with machine learning using MonkeyLearn. What are the categories or tags that you want to assign to your texts? This is the first question you need to answer when you start working on your text classifier.

This is the second part in a series where we analyze thousands of articles from tech news sites in order to get insights and trends about startups. Last time around we scraped all the articles ever published in TechCrunch, VentureBeat and Recode using Scrapy. We then filtered out all the articles that weren't about startups, so we now have only the publications relevant to our analysis. Finally, we'll combine these classifiers to be ready to analyze all of our data. For the first part of this analysis, it'd be great to know for each piece of startup news what "event" it is describing.