Transformer-Based Spatio-Temporal Association of Apple Fruitlets

-- In this paper, we present a transformer-based method to spatio-temporally associate apple fruitlets in stereo-images collected on different days and from different camera poses. State-of-the-art association methods in agriculture are dedicated towards matching larger crops using either high-resolution point clouds or temporally stable features, which are both difficult to obtain for smaller fruit in the field. T o address these challenges, we propose a transformer-based architecture that encodes the shape and position of each fruitlet, and propagates and refines these features through a series of transformer encoder layers with alternating self and cross-attention. We demonstrate that our method is able to achieve an F1-score of 92.4% on data collected in a commercial apple orchard and outperforms all baselines and ablations. The global food supply is constantly under increasing pressure as a result of climate change, population growth, and increased labor shortages. To keep up with demand, agriculturalists are turning to computer vision-based systems that can automate a variety of laborious and time-intensive tasks such as harvesting [1], pruning [2], counting [3], and crop modeling [4]. These automated solutions not only improve efficiency, but also help mitigate the challenges posed by labor shortages and increasing food demand, ensuring that critical agricultural tasks can be performed reliably at scale. One particularly important but challenging task to automate is monitoring the growth and development of individual plants and fruits. Monitoring plant and fruit growth is important because it enables agricultural specialists to make more informed real-time crop management decisions and helps with downstream tasks such as phenotyping [5], disease management [6], and yield prediction [7].