pantheon
SHADE: Semantic Hypernym Annotator for Domain-specific Entities -- DnD Domain Use Case
Peiris, Akila, de Silva, Nisansa
Manual data annotation is an important NLP task but one that takes considerable amount of resources and effort. In spite of the costs, labeling and categorizing entities is essential for NLP tasks such as semantic evaluation. Even though annotation can be done by non-experts in most cases, due to the fact that this requires human labor, the process is costly. Another major challenge encountered in data annotation is maintaining the annotation consistency. Annotation efforts are typically carried out by teams of multiple annotators. The annotations need to maintain the consistency in relation to both the domain truth and annotation format while reducing human errors. Annotating a specialized domain that deviates significantly from the general domain, such as fantasy literature, will see a lot of human error and annotator disagreement. So it is vital that proper guidelines and error reduction mechanisms are enforced. One such way to enforce these constraints is using a specialized application. Such an app can ensure that the notations are consistent, and the labels can be pre-defined or restricted reducing the room for errors. In this paper, we present SHADE, an annotation software that can be used to annotate entities in the high fantasy literature domain. Specifically in Dungeons and Dragons lore extracted from the Forgotten Realms Fandom Wiki.
Exploring Concept Depth: How Large Language Models Acquire Knowledge at Different Layers?
Jin, Mingyu, Yu, Qinkai, Huang, Jingyuan, Zeng, Qingcheng, Wang, Zhenting, Hua, Wenyue, Zhao, Haiyan, Mei, Kai, Meng, Yanda, Ding, Kaize, Yang, Fan, Du, Mengnan, Zhang, Yongfeng
Large language models (LLMs) have shown remarkable performances across a wide range of tasks. However, the mechanisms by which these models encode tasks of varying complexities remain poorly understood. In this paper, we explore the hypothesis that LLMs process concepts of varying complexities in different layers, introducing the idea of "Concept Depth" to suggest that more complex concepts are typically acquired in deeper layers. Specifically, we categorize concepts based on their level of abstraction, defining them in the order of increasing complexity within factual, emotional, and inferential tasks. We conduct extensive probing experiments using layer-wise representations across various LLM families (Gemma, LLaMA, QWen) on various datasets spanning the three domains of tasks. Our findings reveal that models could efficiently conduct probing for simpler tasks in shallow layers, and more complex tasks typically necessitate deeper layers for accurate understanding. Additionally, we examine how external factors, such as adding noise to the input and quantizing the model weights, might affect layer-wise representations. Our findings suggest that these factors can impede the development of a conceptual understanding of LLMs until deeper layers are explored. We hope that our proposed concept and experimental insights will enhance the understanding of the mechanisms underlying LLMs. Our codes are available at https://github.com/Luckfort/CD.
Guided Distant Supervision for Multilingual Relation Extraction Data: Adapting to a New Language
Plum, Alistair, Ranasinghe, Tharindu, Purschke, Christoph
Relation extraction is essential for extracting and understanding biographical information in the context of digital humanities and related subjects. There is a growing interest in the community to build datasets capable of training machine learning models to extract relationships. However, annotating such datasets can be expensive and time-consuming, in addition to being limited to English. This paper applies guided distant supervision to create a large biographical relationship extraction dataset for German. Our dataset, composed of more than 80,000 instances for nine relationship types, is the largest biographical German relationship extraction dataset. We also create a manually annotated dataset with 2000 instances to evaluate the models and release it together with the dataset compiled using guided distant supervision. We train several state-of-the-art machine learning models on the automatically created dataset and release them as well. Furthermore, we experiment with multilingual and cross-lingual experiments that could benefit many low-resource languages.
How Sam Altman is pushing OpenAI into the 'Big Tech' pantheon
In May, the company began a hiring spree, poaching executives from Meta, Apple and Amazon Web Services. Last month, the company expanded its footprint in San Francisco, subleasing nearly 445,000 square feet of office space from Uber, purchased when then-CEO Travis Kalanick was still the most envied founder in the Valley.
Best Automation Solutions: Time to take Healthtech to Hospitals
We are in a grave situation where our healthcare infrastructure is under intense pain. We have tried all means to tackle the growing needs of the people but the available infrastructure is not good enough for the same. In these limited resources what can help us is the right kind of "Automation". Automation not only in the process but also in the technology which handles patients. Product Brief: Best Kiosk provides a self-service channel for patients to register, check in for consultations, book appointments, and make payments.
Don't trust AI until we build systems that earn trust
To judge from the hype, artificial intelligence is inches away from ripping through the economy and destroying everyone's jobs--save for the AI scientists who build the technology and the baristas and yoga instructors who minister to them. But one critic of that view comes from within the tent of AI itself: Gary Marcus. From an academic background in psychology and neuroscience--rather than computer science--Mr Marcus has long been an AI gadfly. He relishes poking holes in the popular AI technique of deep-learning because of its inability to perform abstractions even as it does an impressive job at pattern-matching. Yet his unease with the state of the art didn't prevent him from advancing the art with his own AI startup, Geometric Intelligence, which he sold to Uber in 2016.
Don't trust AI until we build systems that earn trust
To judge from the hype, artificial intelligence is inches away from ripping through the economy and destroying everyone's jobs--save for the AI scientists who build the technology and the baristas and yoga instructors who minister to them. But one critic of that view comes from within the tent of AI itself: Gary Marcus. From an academic background in psychology and neuroscience--rather than computer science--Mr Marcus has long been an AI gadfly. He relishes poking holes in the popular AI technique of deep-learning because of its inability to perform abstractions even as it does an impressive job at pattern-matching. Yet his unease with the state of the art didn't prevent him from advancing the art with his own AI startup, Geometric Intelligence, which he sold to Uber in 2016.
Saving Big Data from Big Mouths
SA Forum is an invited essay from experts on topical issues in science and technology. It has become fashionable to bad-mouth big data. In recent weeks the New York Times, Financial Times, Wired and other outlets have all run pieces bashing this new technological movement. To be fair, many of the critiques have a point: There has been a lot of hype about big data and it is important not to inflate our expectations about what it can do. But little of this hype has come from the actual people working with large data sets.