Materials
LLM-Assisted Question-Answering on Technical Documents Using Structured Data-Aware Retrieval Augmented Generation
Sobhan, Shadman, Haque, Mohammad Ariful
Large Language Models (LLMs) are capable of natural language understanding and generation. But they face challenges such as hallucination and outdated knowledge. Fine-tuning is one possible solution, but it is resource-intensive and must be repeated with every data update. Retrieval-Augmented Generation (RAG) offers an efficient solution by allowing LLMs to access external knowledge sources. However, traditional RAG pipelines struggle with retrieving information from complex technical documents with structured data such as tables and images. In this work, we propose a RAG pipeline, capable of handling tables and images in documents, for technical documents that support both scanned and searchable formats. Its retrieval process combines vector similarity search with a fine-tuned reranker based on Gemma-2-9b-it. The reranker is trained using RAFT (Retrieval-Augmented Fine-Tuning) on a custom dataset designed to improve context identification for question answering. Our evaluation demonstrates that the proposed pipeline achieves a high faithfulness score of 94% (RAGas) and 96% (DeepEval), and an answer relevancy score of 87% (RAGas) and 93% (DeepEval). Comparative analysis demonstrates that the proposed architecture is superior to general RAG pipelines in terms of table-based questions and handling questions outside context.
Transfer Learning for Assessing Heavy Metal Pollution in Seaports Sediments
Lai, Tin, Farid, Farnaz, Kuan, Yueyang, Zhang, Xintian
Detecting heavy metal pollution in soils and seaports is vital for regional environmental monitoring. The Pollution Load Index (PLI), an international standard, is commonly used to assess heavy metal containment. However, the conventional PLI assessment involves laborious procedures and data analysis of sediment samples. To address this challenge, we propose a deep-learning-based model that simplifies the heavy metal assessment process. Our model tackles the issue of data scarcity in the water-sediment domain, which is traditionally plagued by challenges in data collection and varying standards across nations. By leveraging transfer learning, we develop an accurate quantitative assessment method for predicting PLI. Our approach allows the transfer of learned features across domains with different sets of features. We evaluate our model using data from six major ports in New South Wales, Australia: Port Yamba, Port Newcastle, Port Jackson, Port Botany, Port Kembla, and Port Eden. The results demonstrate significantly lower Mean Absolute Error (MAE) and Mean Absolute Percentage Error (MAPE) of approximately 0.5 and 0.03, respectively, compared to other models. Our model performance is up to 2 orders of magnitude than other baseline models. Our proposed model offers an innovative, accessible, and cost-effective approach to predicting water quality, benefiting marine life conservation, aquaculture, and industrial pollution monitoring.
A Deep Learning framework for building damage assessment using VHR SAR and geospatial data: demonstration on the 2023 Turkiye Earthquake
Russo, Luigi, Tapete, Deodato, Ullo, Silvia Liberata, Gamba, Paolo
Building damage identification shortly after a disaster is crucial for guiding emergency response and recovery efforts. Although optical satellite imagery is commonly used for disaster mapping, its effectiveness is often hampered by cloud cover or the absence of pre-event acquisitions. To overcome these challenges, we introduce a novel multimodal deep learning (DL) framework for detecting building damage using single-date very high resolution (VHR) Synthetic Aperture Radar (SAR) imagery from the Italian Space Agency (ASI) COSMO SkyMed (CSK) constellation, complemented by auxiliary geospatial data. Our method integrates SAR image patches, OpenStreetMap (OSM) building footprints, digital surface model (DSM) data, and structural and exposure attributes from the Global Earthquake Model (GEM) to improve detection accuracy and contextual interpretation. Unlike existing approaches that depend on pre and post event imagery, our model utilizes only post event data, facilitating rapid deployment in critical scenarios. The framework effectiveness is demonstrated using a new dataset from the 2023 earthquake in Turkey, covering multiple cities with diverse urban settings. Results highlight that incorporating geospatial features significantly enhances detection performance and generalizability to previously unseen areas. By combining SAR imagery with detailed vulnerability and exposure information, our approach provides reliable and rapid building damage assessments without the dependency from available pre-event data. Moreover, the automated and scalable data generation process ensures the framework's applicability across diverse disaster-affected regions, underscoring its potential to support effective disaster management and recovery efforts. Code and data will be made available upon acceptance of the paper.
Mic-hackathon 2024: Hackathon on Machine Learning for Electron and Scanning Probe Microscopy
Pratiush, Utkarsh, Houston, Austin, Barakati, Kamyar, Raghavan, Aditya, Yoon, Dasol, KP, Harikrishnan, Baraissov, Zhaslan, Ma, Desheng, Welborn, Samuel S., Jakowski, Mikolaj, Barhorst, Shawn-Patrick, Pattison, Alexander J., Manganaris, Panayotis, Madugula, Sita Sirisha, Ayyagari, Sai Venkata Gayathri, Kennedy, Vishal, Bulanadi, Ralph, Wang, Michelle, Pang, Kieran J., Addison-Smith, Ian, Menacho, Willy, Guzman, Horacio V., Kiefer, Alexander, Furth, Nicholas, Kolev, Nikola L., Petrov, Mikhail, Liu, Viktoriia, Ilyev, Sergey, Rairao, Srikar, Rodani, Tommaso, Pinto-Huguet, Ivan, Chen, Xuli, Cruañes, Josep, Torrens, Marta, Pomar, Jovan, Su, Fanzhi, Vedanti, Pawan, Lyu, Zhiheng, Wang, Xingzhi, Yao, Lehan, Taqieddin, Amir, Laskowski, Forrest, Yin, Xiangyu, Shao, Yu-Tsun, Fein-Ashley, Benjamin, Jiang, Yi, Kumar, Vineet, Mishra, Himanshu, Paul, Yogesh, Bazgir, Adib, Madugula, Rama chandra Praneeth, Zhang, Yuwen, Omprakash, Pravan, Huang, Jian, Montufar-Morales, Eric, Chawla, Vivek, Sethi, Harshit, Huang, Jie, Kurki, Lauri, Guinan, Grace, Salvador, Addison, Ter-Petrosyan, Arman, Van Winkle, Madeline, Spurgeon, Steven R., Narasimha, Ganesh, Wu, Zijie, Liu, Richard, Liu, Yongtao, Slautin, Boris, Lupini, Andrew R, Vasudevan, Rama, Duscher, Gerd, Kalinin, Sergei V.
Microscopy is a primary source of information on materials structure and functionality at nanometer and atomic scales. The data generated is often well-structured, enriched with metadata and sample histories, though not always consistent in detail or format. The adoption of Data Management Plans (DMPs) by major funding agencies promotes preservation and access. However, deriving insights remains difficult due to the lack of standardized code ecosystems, benchmarks, and integration strategies. As a result, data usage is inefficient and analysis time is extensive. In addition to post-acquisition analysis, new APIs from major microscope manufacturers enable real-time, ML-based analytics for automated decision-making and ML-agent-controlled microscope operation. Yet, a gap remains between the ML and microscopy communities, limiting the impact of these methods on physics, materials discovery, and optimization. Hackathons help bridge this divide by fostering collaboration between ML researchers and microscopy experts. They encourage the development of novel solutions that apply ML to microscopy, while preparing a future workforce for instrumentation, materials science, and applied ML. This hackathon produced benchmark datasets and digital twins of microscopes to support community growth and standardized workflows. All related code is available at GitHub: https://github.com/KalininGroup/Mic-hackathon-2024-codes-publication/tree/1.0.0.1
Cannot See the Forest for the Trees: Invoking Heuristics and Biases to Elicit Irrational Choices of LLMs
Yang, Haoming, Ma, Ke, Jia, Xiaojun, Sun, Yingfei, Xu, Qianqian, Huang, Qingming
Despite the remarkable performance of Large Language Models (LLMs), they remain vulnerable to jailbreak attacks, which can compromise their safety mechanisms. Existing studies often rely on brute-force optimization or manual design, failing to uncover potential risks in real-world scenarios. To address this, we propose a novel jailbreak attack framework, ICRT, inspired by heuristics and biases in human cognition. Leveraging the simplicity effect, we employ cognitive decomposition to reduce the complexity of malicious prompts. Simultaneously, relevance bias is utilized to reorganize prompts, enhancing semantic alignment and inducing harmful outputs effectively. Furthermore, we introduce a ranking-based harmfulness evaluation metric that surpasses the traditional binary success-or-failure paradigm by employing ranking aggregation methods such as Elo, HodgeRank, and Rank Centrality to comprehensively quantify the harmfulness of generated content. Experimental results show that our approach consistently bypasses mainstream LLMs' safety mechanisms and generates high-risk content, providing insights into jailbreak attack risks and contributing to stronger defense strategies.
Inside a plan to use AI to amplify doubts about the dangers of pollutants
An industry-backed researcher who has forged a career sowing doubt about the dangers of pollutants is attempting to use artificial intelligence (AI) to amplify his perspective. Louis Anthony "Tony" Cox Jr, a Denver-based risk analyst and former Trump adviser who once reportedly claimed there is no proof that cleaning air saves lives, is developing an AI application to scan academic research for what he sees as the false conflation of correlation with causation. Cox has described the project as an attempt to weed "propaganda" out of epidemiological research and perform "critical thinking at scale" in emails to industry researchers, which were obtained via Freedom of Information Act requests by the Energy and Policy Institute, a non-profit advocacy group, and exclusively reviewed by the Guardian. He has long leveled accusations of flimsiness at research linking exposure to chemical compounds with health dangers, including on behalf of polluting interests such as cigarette manufacturer Philip Morris and the American Petroleum Institute – a fossil fuel lobbying group he has even allowed to "copy edit" his findings. Both the tobacco and oil industries have a history of weaponizing scientific uncertainty, experts say, with some arguing that similar tactics drive the Trump administration's current deregulatory efforts. The president's May "gold standard" science order, for instance, empowered his appointees to "correct scientific information" and "discipline" those who breach the administration's views, prompting outrage from some scientists. Cox has obtained funding to develop the new AI reviewer from the American Chemistry Council (ACC), the nation's largest chemical industry advocacy group, which counts oil and chemical giants such as Exxon and DuPont as members.
A Comprehensive Dataset for Underground Miner Detection in Diverse Scenario
Addy, Cyrus, Gurumadaiah, Ajay Kumar, Gao, Yixiang, Awuah-Offei, Kwame
Underground mining operations face significant safety challenges that make emergency response capabilities crucial. While robots have shown promise in assisting with search and rescue operations, their effectiveness depends on reliable miner detection capabilities. Deep learning algorithms offer potential solutions for automated miner detection, but require comprehensive training datasets, which are currently lacking for underground mining environments. This paper presents a novel thermal imaging dataset specifically designed to enable the development and validation of miner detection systems for potential emergency applications. We systematically captured thermal imagery of various mining activities and scenarios to create a robust foundation for detection algorithms. To establish baseline performance metrics, we evaluated several state-of-the-art object detection algorithms including YOLOv8, YOLOv10, YOLO11, and RT-DETR on our dataset. While not exhaustive of all possible emergency situations, this dataset serves as a crucial first step toward developing reliable thermal-based miner detection systems that could eventually be deployed in real emergency scenarios. This work demonstrates the feasibility of using thermal imaging for miner detection and establishes a foundation for future research in this critical safety application.
Neural networks for the prediction of peel force for skin adhesive interface using FEM simulation
Masarkar, Ashish, Gupta, Rakesh, Dingari, Naga Neehar, Rai, Beena
Studying the peeling behaviour of adhesives on skin is vital for advancing biomedical applications such as medical adhesives and transdermal patches. Traditional methods like experimental testing and finite element method (FEM), though considered gold standards, are resource-intensive, computationally expensive and time-consuming, particularly when analysing a wide material parameter space. In this study, we present a neural network-based approach to predict the minimum peel force (F_min) required for adhesive detachment from skin tissue, limiting the need for repeated FEM simulations and significantly reducing the computational cost. Leveraging a dataset generated from FEM simulations of 90 degree peel test with varying adhesive and fracture mechanics parameters, our neural network model achieved high accuracy, validated through rigorous 5-fold cross-validation. The final architecture was able to predict a wide variety of skin-adhesive peeling behaviour, exhibiting a mean squared error (MSE) of 3.66*10^-7 and a R^2 score of 0.94 on test set, demonstrating robust performance. This work introduces a reliable, computationally efficient method for predicting adhesive behaviour, significantly reducing simulation time while maintaining accuracy. This integration of machine learning with high-fidelity biomechanical simulations enables efficient design and optimization of skin-adhesive systems, providing a scalable framework for future research in computational dermato-mechanics and bio-adhesive material design.
ARChemist: Autonomous Robotic Chemistry System Architecture
Fakhruldeen, Hatem, Pizzuto, Gabriella, Glowacki, Jakub, Cooper, Andrew Ian
-- Automated laboratory experiments have the potential to propel new discoveries, while increasing reproducibility and improving scientists' safety when handling dangerous materials. However, many automated laboratory workflows have not fully leveraged the remarkable advancements in robotics and digital lab equipment. As a result, most robotic systems used in the labs are programmed specifically for a single experiment, often relying on proprietary architectures or using unconventional hardware. In this work, we tackle this problem by proposing a novel robotic system architecture specifically designed with and for chemists, which allows the scientist to easily reconfigure their setup for new experiments. Specifically, the system's strength is its ability to combine together heterogeneous robotic platforms with standard laboratory equipment to create different experimental setups. Finally, we show how the architecture can be used for specific laboratory experiments through case studies such as solubility screening and crystallisation. I. INTRODUCTION Accelerating the discovery of new materials is important for industrial applications such as healthcare and energy production. This can be achieved through running long-term experiments autonomously, for example by increasing the use of robotic platforms in laboratories. In practice, this would accumulate more experiments in less time, and potentially minimise the scientists' exposure to harmful chemicals, reducing their repetitive tasks.
Multimodal Behaviour Trees for Robotic Laboratory Task Automation
Fakhruldeen, Hatem, Nambiar, Arvind Raveendran, Veeramani, Satheeshkumar, Tailor, Bonilkumar Vijaykumar, Juneghani, Hadi Beyzaee, Pizzuto, Gabriella, Cooper, Andrew Ian
Laboratory robotics offer the capability to conduct experiments with a high degree of precision and reproducibility, with the potential to transform scientific research. Trivial and repeatable tasks; e.g., sample transportation for analysis and vial capping are well-suited for robots; if done successfully and reliably, chemists could contribute their efforts towards more critical research activities. Currently, robots can perform these tasks faster than chemists, but how reliable are they? Improper capping could result in human exposure to toxic chemicals which could be fatal. To ensure that robots perform these tasks as accurately as humans, sensory feedback is required to assess the progress of task execution. To address this, we propose a novel methodology based on behaviour trees with multimodal perception. Along with automating robotic tasks, this methodology also verifies the successful execution of the task, a fundamental requirement in safety-critical environments. The experimental evaluation was conducted on two lab tasks: sample vial capping and laboratory rack insertion. The results show high success rate, i.e., 88% for capping and 92% for insertion, along with strong error detection capabilities. This ultimately proves the robustness and reliability of our approach and that using multimodal behaviour trees should pave the way towards the next generation of robotic chemists.