Scheurer, Erik
Lossy Neural Compression for Geospatial Analytics: A Review
Gomes, Carlos, Wittmann, Isabelle, Robert, Damien, Jakubik, Johannes, Reichelt, Tim, Martone, Michele, Maurogiovanni, Stefano, Vinge, Rikard, Hurst, Jonas, Scheurer, Erik, Sedona, Rocco, Brunschwiler, Thomas, Kesselheim, Stefan, Batic, Matej, Stier, Philip, Wegner, Jan Dirk, Cavallaro, Gabriele, Pebesma, Edzer, Marszalek, Michael, Belenguer-Plomer, Miguel A, Adriko, Kennedy, Fraccaro, Paolo, Kienzler, Romeo, Briq, Rania, Benassou, Sabrina, Lazzarini, Michele, Albrecht, Conrad M
Over the past decades, there has been an explosion in the amount of available Earth Observation (EO) data. The unprecedented coverage of the Earth's surface and atmosphere by satellite imagery has resulted in large volumes of data that must be transmitted to ground stations, stored in data centers, and distributed to end users. Modern Earth System Models (ESMs) face similar challenges, operating at high spatial and temporal resolutions, producing petabytes of data per simulated day. Data compression has gained relevance over the past decade, with neural compression (NC) emerging from deep learning and information theory, making EO data and ESM outputs ideal candidates due to their abundance of unlabeled data. In this review, we outline recent developments in NC applied to geospatial data. We introduce the fundamental concepts of NC including seminal works in its traditional applications to image and video compression domains with focus on lossy compression. We discuss the unique characteristics of EO and ESM data, contrasting them with "natural images", and explain the additional challenges and opportunities they present. Moreover, we review current applications of NC across various EO modalities and explore the limited efforts in ESM compression to date. The advent of self-supervised learning (SSL) and foundation models (FM) has advanced methods to efficiently distill representations from vast unlabeled data. We connect these developments to NC for EO, highlighting the similarities between the two fields and elaborate on the potential of transferring compressed feature representations for machine--to--machine communication. Based on insights drawn from this review, we devise future directions relevant to applications in EO and ESM.
Detection Defenses: An Empty Promise against Adversarial Patch Attacks on Optical Flow
Scheurer, Erik, Schmalfuss, Jenny, Lis, Alexander, Bruhn, Andrés
Adversarial patches undermine the reliability of optical flow predictions when placed in arbitrary scene locations. Therefore, they pose a realistic threat to real-world motion detection and its downstream applications. Potential remedies are defense strategies that detect and remove adversarial patches, but their influence on the underlying motion prediction has not been investigated. In this paper, we thoroughly examine the currently available detect-and-remove defenses ILP and LGS for a wide selection of state-of-the-art optical flow methods, and illuminate their side effects on the quality and robustness of the final flow predictions. In particular, we implement defense-aware attacks to investigate whether current defenses are able to withstand attacks that take the defense mechanism into account. Our experiments yield two surprising results: Detect-and-remove defenses do not only lower the optical flow quality on benign scenes, in doing so, they also harm the robustness under patch attacks for all tested optical flow methods except FlowNetC. As currently employed detect-and-remove defenses fail to deliver the promised adversarial robustness for optical flow, they evoke a false sense of security. The code is available at https://github.com/cv-stuttgart/DetectionDefenses.