Personal use of this material is permitted. The A TLAS of Traffic Lights: A Reliable Perception Framework for Autonomous Driving Rupert Polley 1, Nikolai Polley 2, Dominik Heid 1, Marc Heinrich 1, Sven Ochs 1, and J. Marius Z ollner 1, 2 Abstract -- Traffic light perception is an essential component of the camera-based perception system for autonomous vehicles, enabling accurate detection and interpretation of traffic lights to ensure safe navigation through complex urban environments. In this work, we propose a modularized perception framework that integrates state-of-the-art detection models with a novel real-time association and decision framework, enabling seamless deployment into an autonomous driving stack. T o address the limitations of existing public datasets, we introduce the ATLAS dataset, which provides comprehensive annotations of traffic light states and pictograms across diverse environmental conditions and camera setups. We train and evaluate several state-of-the-art traffic light detection architectures on ATLAS, demonstrating significant performance improvements in both accuracy and robustness. Finally, we evaluate the framework in real-world scenarios by deploying it in an autonomous vehicle to make decisions at traffic light-controlled intersections, highlighting its reliability and effectiveness for real-time operation. I NTRODUCTION Perception of traffic lights plays a pivotal role in ensuring the safe navigation of urban environments for autonomous driving (AD). To operate reliably, autonomous vehicles must not only detect and classify traffic lights accurately but also interpret their relevance to the vehicle's current context and programmed trajectory. Complex intersections, occlusions, and environmental conditions such as rain or nighttime visibility remain a challenge. Unlike other perception tasks, such as object detection, where LiDAR can complement vision-based approaches, traffic light recognition primarily relies on real-time camera-based perception. While V ehicle-to-Everything (V2X) communication has the potential to provide traffic light state information, its deployment remains sparse, making vision-based detection the only widely available method.

In recent years, diffusion models [1, 2], particularly latent diffusion models [3], have spearheaded high quality textto-image generation, and have been widely adopted by researchers and the general public alike. Trained on massive datasets like LAION-5B [4], these models have developed a broad understanding of visual concepts, enabling new creative and practical applications. Notably, tools like Stable Diffusion [3, 5] have been made readily accessible for general use. Building on this foundation, efficient adaptation methods such as parameter efficient fine-tuning (PEFT) [6, 7, 8], guidance based approaches [9, 10, 11], and pseudo-word generation [12] enable users to leverage this extensive pretraining for customizing models that can specialize on downstream tasks with smaller datasets. However, the rapid adoption of diffusion models has also raised significant privacy, ethical and legal concerns. One critical issue is the vulnerability of these models to privacy attacks, from membership inference [13], where an attacker determines whether a specific data point was used to train a particular model, to data extraction [14], which enables an attacker to reconstruct particular images from the training dataset. This issue is even more severe during the fine-tuning phase where the model is fine-tuned on smaller specialized datasets from a possibly different domain and each data record has more impact on the final model. This risk underscores the importance of privacy-preserving technologies, particularly as diffusion models often rely on vast datasets scraped from the internet without explicit consent from content owners.

Effective traffic light detection is a critical component of the perception stack in autonomous vehicles. This work introduces a novel deep-learning detection system while addressing the challenges of previous work. Utilizing a comprehensive dataset amalgamation, including the Bosch Small Traffic Lights Dataset, LISA, the DriveU Traffic Light Dataset, and a proprietary dataset from Karlsruhe, we ensure a robust evaluation across varied scenarios. Furthermore, we propose a relevance estimation system that innovatively uses directional arrow markings on the road, eliminating the need for prior map creation. On the DriveU dataset, this approach results in 96% accuracy in relevance estimation. Finally, a real-world evaluation is performed to evaluate the deployment and generalizing abilities of these models. For reproducibility and to facilitate further research, we provide the model weights and code: https://github.com/KASTEL-MobilityLab/traffic-light-detection.

Adversarial examples arose as a challenge for machine learning. To hinder them, most defenses alter how models are trained (e.g., adversarial training) or inference is made (e.g., randomized smoothing). Still, while these approaches markedly improve models' adversarial robustness, models remain highly susceptible to adversarial examples. Identifying that, in certain domains such as traffic-sign recognition, objects are implemented per standards specifying how artifacts (e.g., signs) should be designed, we propose a novel approach for improving adversarial robustness. Specifically, we offer a method to redefine standards, making minor changes to existing ones, to defend against adversarial examples. We formulate the problem of artifact design as a robust optimization problem, and propose gradient-based and greedy search methods to solve it. We evaluated our approach in the domain of traffic-sign recognition, allowing it to alter traffic-sign pictograms (i.e., symbols within the signs) and their colors. We found that, combined with adversarial training, our approach led to up to 25.18\% higher robust accuracy compared to state-of-the-art methods against two adversary types, while further increasing accuracy on benign inputs.

Individuals with complex communication needs (CCN) often rely on augmentative and alternative communication (AAC) systems to have conversations and communique their wants. Such systems allow message authoring by arranging pictograms in sequence. However, the difficulty of finding the desired item to complete a sentence can increase as the user's vocabulary increases. This paper proposes using BERTimbau, a Brazilian Portuguese version of BERT, for pictogram prediction in AAC systems. To finetune BERTimbau, we constructed an AAC corpus for Brazilian Portuguese to use as a training corpus. We tested different approaches to representing a pictogram for prediction: as a word (using pictogram captions), as a concept (using a dictionary definition), and as a set of synonyms (using related terms). We also evaluated the usage of images for pictogram prediction. The results demonstrate that using embeddings computed from the pictograms' caption, synonyms, or definitions have a similar performance. Using synonyms leads to lower perplexity, but using captions leads to the highest accuracies. This paper provides insight into how to represent a pictogram for prediction using a BERT-like model and the potential of using images for pictogram prediction.

Augmentative and alternative communication (AAC) is a practical means of communication for people with language disabilities. In this study, we propose PicTalky, which is an AI-based AAC system that helps children with language developmental disabilities to improve their communication skills and language comprehension abilities. PicTalky can process both text and pictograms more accurately by connecting a series of neural-based NLP modules. Moreover, we perform quantitative and qualitative analyses on the essential features of PicTalky. It is expected that those suffering from language problems will be able to express their intentions or desires more easily and improve their quality of life by using this service. We have made the models freely available alongside a demonstration of the Web interface. Furthermore, we implemented robotics AAC for the first time by applying PicTalky to the NAO robot.

This paper presents a discussion about assistive and augmentative natural language processing designed for certain disabled persons unable to communicate. Several approaches have been proposed, according to abilities of the writer. Here we distinguish two cases in the writer’s capacities: the writer knows alphabetic writing, or (s)he does not know it. In the first case, the idea is to assist the writer by completing the words or the group of words which are initially written. In the second case, pictograms are used instead of characters, but it must be decided if these pictograms represent concepts or words in a new writing system. If the pictograms represent concepts, the produced text may not correspond exactly to the wishes of the writer; whereas when the pictograms represent words, the writer has to change his (her) mental approach to write the words that (s)he has chosen in another way.

This paper intends to demonstrate how Applicative and Combinatory Categorial Grammar (ACCG) can be drawn on to design powerful software applications for the teaching of languages. To this end, we present some modules from our “pictographic translator”, a software that performs syntactical analysis of sentences in natural language directly written by the user, and then dynamically displays series of pictograms that illustrate the words and structure of the user’s sentences. After a short presentation of our application and an introduction to ACCG, we will examine how this formalism enables the building of several high-level functions in our system, such as disambiguation, structure exhibition and grammatical correction/validation. We finally open a short discussion concerning the potential (and limits) of this architecture with regards to multilingualism.

For certain disable persons unable to communicate, we present a palliative aid which consists of a virtual pictographic keyboard associated to a text processing from a pictographic scripture. Words and the grammar are given as pictograms. The pictographic lexicon must be organized following the mental lexicon of the user to propose the pictograms of grammar in order to facilitate his (her) task of writing. We discuss the utility of ontologies in the organization of lexicons and in the building of texts.