Government
ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time
Ding, Yi, Li, Bolian, Zhang, Ruqi
Vision Language Models (VLMs) have become essential backbones for multimodal intelligence, yet significant safety challenges limit their real-world application. While textual inputs are often effectively safeguarded, adversarial visual inputs can easily bypass VLM defense mechanisms. Existing defense methods are either resource-intensive, requiring substantial data and compute, or fail to simultaneously ensure safety and usefulness in responses. To address these limitations, we propose a novel two-phase inference-time alignment framework, Evaluating Then Aligning (ETA): 1) Evaluating input visual contents and output responses to establish a robust safety awareness in multimodal settings, and 2) Aligning unsafe behaviors at both shallow and deep levels by conditioning the VLMs' generative distribution with an interference prefix and performing sentence-level best-of-N to search the most harmless and helpful generation paths. Extensive experiments show that ETA outperforms baseline methods in terms of harmlessness, helpfulness, and efficiency, reducing the unsafe rate by 87.5% in cross-modality attacks and achieving 96.6% win-ties in GPT-4 helpfulness evaluation. The code is publicly available at https://github.com/DripNowhy/ETA.
Viewers don't trust candidates who use generative AI in political ads, study finds
Artificial intelligence is expected to have an impact on the upcoming US election in November. States have been trying to protect against misinformation by passing laws that require political advertisements to disclose when they have used generative AI. Twenty states now have rules on the books, and according to new research, voters have a negative reaction to seeing those disclaimers. That seems like a pretty fair response: If a politician uses generative AI to mislead voters, then voters don't appreciate that. The study was conducted by New York University's Center on Technology Policy and first reported by The Washington Post.
Elon Musk Is Offering People Cash To Identify Trump Voters. What Could Go Wrong?
Elon Musk, one of the world's richest men, has gone "Deep MAGA." The billionaire owner of X--the "everything app"--endorsed Donald Trump seconds after the former president was nearly assassinated in July, and within a few days was hatching plans to spend 45 million to get Trump elected. On Saturday, Musk joined the former president on stage in Butler, Pennsylvania, where he did his signature hop (which is vaguely in the shape of an X), and predicted that if Kamala Harris is elected there will be no more elections in the United States. Musk has a habit of making predictions that don't amount to much. It is one of the defining facets of his personality, up there with scientific racism and being a guy with a signature hop. You can go online and watch a supercut of him promising fully autonomous self-driving cars every year since 2014.
Reviews: Robust Detection of Adversarial Attacks by Modeling the Intrinsic Properties of Deep Neural Networks
The paper presents an unsupervised learning approach to the problem of adversarial attack detection in the context of deep neural networks. The authors model the intrinsic properties of the networks to detect adversarial inputs. To do so, they employ a Gaussian Mixture Model (GMM) to approximate the hidden state distribution, in practice the state of the fully connected hidden layers, and detect adversarial samples by simply checking that their likelihood is lower than a given threshold. Exhaustive experimental results in different show that the proposed method achieves state-of-the-art performance compared to unsupervised methods while generalizing better than supervised approaches. The paper reads well and is technically sound.
Decoding the role Big Tech plays in the war in Gaza
In pulverized Gaza, life has become almost medieval for traumatized residents, but the Israeli offensive that obliterated their homes and killed their neighbors is defiantly modern and has raised profound questions over the role of Big Tech in war. More than any other major conflict this century, the Israel-Hamas war has spotlighted how artificial intelligence and machine learning can be used on the battlefield, and what responsibility the makers of these tools should bear. "From the early days of the war, the campaign has been framed as an opportunity to test and refine how AI is used in the field of war," said Sophia Goodfriend, a post-doctoral fellow at Harvard University who studies Israel's use of AI and automation in war.
A New Architecture for Neural Enhanced Multiobject Tracking
Wei, Shaoxiu, Liang, Mingchao, Meyer, Florian
Multiobject tracking (MOT) is an important task in robotics, autonomous driving, and maritime surveillance. Traditional work on MOT is model-based and aims to establish algorithms in the framework of sequential Bayesian estimation. More recent methods are fully data-driven and rely on the training of neural networks. The two approaches have demonstrated advantages in certain scenarios. In particular, in problems where plenty of labeled data for the training of neural networks is available, data-driven MOT tends to have advantages compared to traditional methods. A natural thought is whether a general and efficient framework can integrate the two approaches. This paper advances a recently introduced hybrid model-based and data-driven method called neural-enhanced belief propagation (NEBP). Compared to existing work on NEBP for MOT, it introduces a novel neural architecture that can improve data association and new object initialization, two critical aspects of MOT. The proposed tracking method is leading the nuScenes LiDAR-only tracking challenge at the time of submission of this paper.
Counterfactual Causal Inference in Natural Language with Large Language Models
Gendron, Gaรซl, Roลพanec, Joลพe M., Witbrock, Michael, Dobbie, Gillian
Causal structure discovery methods are commonly applied to structured data where the causal variables are known and where statistical testing can be used to assess the causal relationships. By contrast, recovering a causal structure from unstructured natural language data such as news articles contains numerous challenges due to the absence of known variables or counterfactual data to estimate the causal links. Large Language Models (LLMs) have shown promising results in this direction but also exhibit limitations. This work investigates LLM's abilities to build causal graphs from text documents and perform counterfactual causal inference. We propose an end-to-end causal structure discovery and causal inference method from natural language: we first use an LLM to extract the instantiated causal variables from text data and build a causal graph. We merge causal graphs from multiple data sources to represent the most exhaustive set of causes possible. We then conduct counterfactual inference on the estimated graph. The causal graph conditioning allows reduction of LLM biases and better represents the causal estimands. We use our method to show that the limitations of LLMs in counterfactual causal reasoning come from prediction errors and propose directions to mitigate them. We demonstrate the applicability of our method on real-world news articles.
A Deep Learning Approach to Estimate Canopy Height and Uncertainty by Integrating Seasonal Optical, SAR and Limited GEDI LiDAR Data over Northern Forests
Castro, Jose B., Rogers, Cheryl, Sothe, Camile, Cyr, Dominic, Gonsamo, Alemu
Accurate forest canopy height estimation is essential for evaluating aboveground biomass and carbon stock dynamics, supporting ecosystem monitoring services like timber provisioning, climate change mitigation, and biodiversity conservation. However, despite advancements in spaceborne LiDAR technology, data for northern high latitudes remain limited due to orbital and sampling constraints. This study introduces a methodology for generating spatially continuous, high-resolution canopy height and uncertainty estimates using Deep Learning Regression models. We integrate multi-source, multi-seasonal satellite data from Sentinel-1, Landsat, and ALOS-PALSAR-2, with spaceborne GEDI LiDAR as reference data. Our approach was tested in Ontario, Canada, and validated with airborne LiDAR, demonstrating strong performance. The best results were achieved by incorporating seasonal Sentinel-1 and Landsat features alongside PALSAR data, yielding an R-square of 0.72, RMSE of 3.43 m, and bias of 2.44 m. Using seasonal data instead of summer-only data improved variability by 10%, reduced error by 0.45 m, and decreased bias by 1 m. The deep learning model's weighting strategy notably reduced errors in tall canopy height estimates compared to a recent global model, though it overestimated lower canopy heights. Uncertainty maps highlighted greater uncertainty near forest edges, where GEDI measurements are prone to errors and SAR data may encounter backscatter issues like foreshortening, layover, and shadow. This study enhances canopy height estimation techniques in areas lacking spaceborne LiDAR coverage, providing essential tools for forestry, environmental monitoring, and carbon stock estimation.
Empowering Cognitive Digital Twins with Generative Foundation Models: Developing a Low-Carbon Integrated Freight Transportation System
Li, Xueping, Xu, Haowen, Tupayachi, Jose, Omitaomu, Olufemi, Wang, Xudong
Effective monitoring of freight transportation is essential for advancing sustainable, low-carbon economies. Traditional methods relying on single-modal data and discrete simulations fall short in optimizing intermodal systems holistically. These systems involve interconnected processes that affect shipping time, costs, emissions, and socio-economic factors. Developing digital twins for real-time awareness, predictive analytics, and urban logistics optimization requires extensive efforts in knowledge discovery, data integration, and multi-domain simulation. Recent advancements in generative AI offer new opportunities to streamline digital twin development by automating knowledge discovery and data integration, generating innovative simulation and optimization solutions. These models extend digital twins' capabilities by promoting autonomous workflows for data engineering, analytics, and software development. This paper proposes an innovative paradigm that leverages generative AI to enhance digital twins for urban research and operations. Using freight decarbonization as a case study, we propose a conceptual framework employing transformer-based language models to enhance an urban digital twin through foundation models. We share preliminary results and our vision for more intelligent, autonomous, and general-purpose digital twins for optimizing integrated freight systems from multimodal to synchromodal paradigms.
Automating Bibliometric Analysis with Sentence Transformers and Retrieval-Augmented Generation (RAG): A Pilot Study in Semantic and Contextual Search for Customized Literature Characterization for High-Impact Urban Research
Xu, Haowen, Li, Xueping, Tupayachi, Jose, Jianming, null, Lian, null, Omitaomu, Femi
Bibliometric analysis is essential for understanding research trends, scope, and impact in urban science, especially in high-impact journals, such Nature Portfolios. However, traditional methods, relying on keyword searches and basic NLP techniques, often fail to uncover valuable insights not explicitly stated in article titles or keywords. These approaches are unable to perform semantic searches and contextual understanding, limiting their effectiveness in classifying topics and characterizing studies. In this paper, we address these limitations by leveraging Generative AI models, specifically transformers and Retrieval-Augmented Generation (RAG), to automate and enhance bibliometric analysis. We developed a technical workflow that integrates a vector database, Sentence Transformers, a Gaussian Mixture Model (GMM), Retrieval Agent, and Large Language Models (LLMs) to enable contextual search, topic ranking, and characterization of research using customized prompt templates. A pilot study analyzing 223 urban science-related articles published in Nature Communications over the past decade highlights the effectiveness of our approach in generating insightful summary statistics on the quality, scope, and characteristics of papers in high-impact journals. This study introduces a new paradigm for enhancing bibliometric analysis and knowledge retrieval in urban research, positioning an AI agent as a powerful tool for advancing research evaluation and understanding.