Multi-Modal Fusion of In-Situ Video Data and Process Parameters for Online Forecasting of Cookie Drying Readiness

Food drying is essential for food production, extending shelf life, and reducing transportation costs. Accurate real-time forecasting of drying readiness is crucial for minimizing energy consumption, improving productivity, and ensuring product quality. However, this remains challenging due to the dynamic nature of drying, limited data availability, and the lack of effective predictive analytical methods. To address this gap, we propose an end-to-end multi-modal data fusion framework that integrates in-situ video data with process parameters for real-time food drying readiness forecasting. Our approach leverages a new encoder-decoder architecture with modality-specific encoders and a transformer-based decoder to effectively extract features while preserving the unique structure of each modality. We apply our approach to sugar cookie drying, where time-to-ready is predicted at each timestamp. Experimental results demonstrate that our model achieves an average prediction error of only 15 seconds, outperforming state-of-the-art data fusion methods by 65.69% and a video-only model by 11.30%. The proposed model is extensible to various other industrial modality fusion tasks for online decision-making. Introduction Drying is a fundamental process in the food industry that plays a critical role in both food production and preservation. By removing moisture, it transforms raw ingredients into their final, consumable forms while enhancing texture, flavor, and structural integrity [1]. However, food drying is a highly time-and energy-intensive process which accounts for 15% of energy consumption in U.S. industrial processes [2]. As a result, advancing drying technologies and improving product quality are key strategies for minimizing waste and enhancing energy efficiency [3].