lesson learned
Matching Pairs: Attributing Fine-Tuned Models to their Pre-Trained Large Language Models
Foley, Myles, Rawat, Ambrish, Lee, Taesung, Hou, Yufang, Picco, Gabriele, Zizzo, Giulio
The wide applicability and adaptability of generative large language models (LLMs) has enabled their rapid adoption. While the pre-trained models can perform many tasks, such models are often fine-tuned to improve their performance on various downstream applications. However, this leads to issues over violation of model licenses, model theft, and copyright infringement. Moreover, recent advances show that generative technology is capable of producing harmful content which exacerbates the problems of accountability within model supply chains. Thus, we need a method to investigate how a model was trained or a piece of text was generated and what their pre-trained base model was. In this paper we take the first step to address this open problem by tracing back the origin of a given fine-tuned LLM to its corresponding pre-trained base model. We consider different knowledge levels and attribution strategies, and find that we can correctly trace back 8 out of the 10 fine tuned models with our best method.
Using Machine Learning to Understand Top-Down Effects in an Ecosystem: Opportunities, Challenges, and Lessons Learned
Talbert, Douglas A. (Tennessee Technological University) | Tinker, Paul (Tennessee Technological University) | Crowther, Tom (Netherlands Institute of Ecology) | Walker, Donald (Tennessee Technological University)
The soil decomposer community is a primary driver of carbon cycling in forest ecosystems. Understanding the processes that regulate this community is critical to our understanding of the global carbon cycle and fungal mediated impact on climate change. Inadequate statistical strength in traditional soil food web studies has limited our capacity to disentangle the cascading effect of top-level predators on the composition of complex fungal communities. We hypothesize that machine learning can help with this complex problem. This paper examines the opportunities for machine learning in this domain, presents initial results from such analysis, identifies challenges encountered with initial effort, and charts a path forward.
Crowdsourcing Multimodal Dialog Interactions: Lessons Learned from the HALEF Case
Ramanarayanan, Vikram (Educational Testing Service) | Suendermann-Oeft, David (Educational Testing Service) | Molloy, Hillary (Educational Testing Service) | Tsuprun, Eugene (Educational Testing Service) | Lange, Patrick (Educational Testing Service) | Evanini, Keelan (Educational Testing Service)
The advent of multiple study on crowdsourcing for speech applications concluded crowdsourcing vendors and software infrastructure has that "although the crowd sometimes approached the level greatly helped this effort. Several providers also offer integrated of the experts, it never surpassed it" (Parent and Eskenazi filtering tools that allow users to customize different 2011)). This is exacerbated during multimodal dialog data aspects of their data collection, including target population, collections, where it becomes harder to quality-control for geographical location, demographics and sometimes usable audio-video data, due to a variety of factors including even education level and expertise. Managed crowdsourcing poor visual quality caused by variable lighting, position, providers extend these options by offering further customization or occlusions, participant or administrator error, or technical and end-to-end management of the entire data issues with the system or network (McDuff, Kaliouby, and collection operation.
Machine Learning – A Lesson Learned - DZone Big Data
Google self-driving car finally causes an accident -- a lesson in AI. According to The Verge, Google had recently performed a software update that changed the behavior to be more human like. "So several weeks ago we began giving the self-driving car the capabilities it needs to do what human drivers do: hug the rightmost side of the lane." The truth is that one of the complaints about self-driving cars is that they are too cautious so Google adapted the software so the car would move to the far right of the lane so two cars could fit in the single wide lane. This is what a regular, old fashioned, human being does so cars can move more fluidly though the heavily congested streets of California.
Competitive Benchmarking: Lessons Learned from the Trading Agent Competition
Ketter, Wolfgang (Erasmus University) | Symeonidis, Andreas (Aristotle University of Thessaloniki)
In many real-life domains, such as trading environments, selfinterested entities need to operate subject to limited time and information. Additionally, the web has mediated an ever broader range of transactions, urging participants to concurrently trade across multiple markets. All these have generated the need for technologies that empower prompt investigation of large volumes of data and rapid evaluation of numerous alternative strategies in the face of constantly changing market conditions (Bichler, Gupta, and Ketter 2010). AI and machine-learning techniques, including neural networks and genetic algorithms, are continuously gaining ground in the support of such trading scenarios. User modeling, price forecasting, market equilibrium prediction, and strategy optimization are typical cases where AI typically provides reliable solutions. Yet, the adoption and deployment of AI practices in real trading environments remains limited, since the proprietary nature of markets precludes open benchmarking, which is critical for further scientific progress.
Why Programming-By-Demonstration Systems Fail: Lessons Learned for Usable AI
Lau, Tessa (IBM Almaden Research Center)
Programming by demonstration systems have long attempted to make it possible for people to program computers without writing code. However, while these systems have resulted in many publications in AI venues, none of the technologies have yet achieved widespread.adoption. Usability remains a critical barrier to their success. On the basis of lessons learned from three different programming by demonstration systems, we present a set of guidelines to consider when designing usable AI-based systems.