Williams, Thomas
Automating Code Adaptation for MLOps -- A Benchmarking Study on LLMs
Patel, Harsh, Ramanan, Buvaneswari A., Khan, Manzoor A., Williams, Thomas, Friedman, Brian, Drabeck, Lawrence
This paper explores the possibilities of the current generation of Large Language Models for incorporating Machine Learning Operations (MLOps) functionalities into ML training code bases. We evaluate the performance of OpenAI (gpt-3.5-turbo) and WizardCoder (open-source, 15B parameters) models on the automated accomplishment of various MLOps functionalities in different settings. We perform a benchmarking study that assesses the ability of these models to: (1) adapt existing code samples (Inlining) with component-specific MLOps functionality such as MLflow and Weights & Biases for experiment tracking, Optuna for hyperparameter optimization etc., and (2) perform the task of Translation from one component of an MLOps functionality to another, e.g., translating existing GitPython library based version control code to Data Version Control library based. We also propose three different approaches that involve teaching LLMs to comprehend the API documentation of the components as a reference while accomplishing the Translation tasks. In our evaluations, the gpt-3.5-turbo model significantly outperforms WizardCoder by achieving impressive Pass@3 accuracy in model optimization (55% compared to 0% by WizardCoder), experiment tracking (100%, compared to 62.5% by WizardCoder), model registration (92% compared to 42% by WizardCoder) and hyperparameter optimization (83% compared to 58% by WizardCoder) on average, in their best possible settings, showcasing its superior code adaptability performance in complex MLOps tasks.
Models for Capturing Temporal Smoothness in Evolving Networks for Learning Latent Representation of Nodes
Saha, Tanay Kumar, Williams, Thomas, Hasan, Mohammad Al, Joty, Shafiq, Varberg, Nicholas K.
In a dynamic network, the neighborhood of the vertices evolve across different temporal snapshots of the network. Accurate modeling of this temporal evolution can help solve complex tasks involving real-life social and interaction networks. However, existing models for learning latent representation are inadequate for obtaining the representation vectors of the vertices for different time-stamps of a dynamic network in a meaningful way. In this paper, we propose latent representation learning models for dynamic networks which overcome the above limitation by considering two different kinds of temporal smoothness: (i) retrofitted, and (ii) linear transformation. The retrofitted model tracks the representation vector of a vertex over time, facilitating vertex-based temporal analysis of a network. On the other hand, linear transformation based model provides a smooth transition operator which maps the representation vectors of all vertices from one temporal snapshot to the next (unobserved) snapshot-this facilitates prediction of the state of a network in a future time-stamp. We validate the performance of our proposed models by employing them for solving the temporal link prediction task. Experiments on 9 real-life networks from various domains validate that the proposed models are significantly better than the existing models for predicting the dynamics of an evolving network.
Learning to Recognize Novel Objects in One Shot through Human-Robot Interactions in Natural Language Dialogues
Krause, Evan A. (Tufts University) | Zillich, Michael (Technical University Vienna) | Williams, Thomas (Tufts University) | Scheutz, Matthias (Tufts University)
Being able to quickly and naturally teach robots new knowledge is critical for many future open-world human-robot interaction scenarios. In this paper we present a novel approach to using natural language context for one-shot learning of visual objects, where the robot is immediately able to recognize the described object. We describe the architectural components and demonstrate the proposed approach on a robotic platform in a proof-of-concept evaluation.
Novel Mechanisms for Natural Human-Robot Interactions in the DIARC Architecture
Scheutz, Matthias (Tufts University) | Briggs, Gordon (Tufts University) | Cantrell, Rehj (Indiana University) | Krause, Evan (Tufts University) | Williams, Thomas (Tufts University) | Veale, Richard (Indiana University)
Natural human-like human-robot interactions require many functional capabilities from a robot that have to be reflected in architectural components in the robotic control architecture. In particular, various mechanisms for producing social behaviors , goal-oriented cognition , and robust intelligence are required. In this paper, we present an overview of the most recent version of our DIARC architecture and show how several novel algorithms attempt to address these three areas, leading to more natural interactions with humans, while also extending the overall capability of the integrated system.