We present AstroLLaVA, a vision language model for astronomy that enables interaction with astronomical imagery through natural dialogue. By fine-tuning the LLaVA model on a diverse dataset of $\sim$30k images with captions and question-answer pairs sourced from NASA's `Astronomy Picture of the Day', the European Southern Observatory, and the NASA/ESA Hubble Space Telescope, we create a model capable of answering open-ended questions about astronomical concepts depicted visually. Our two-stage fine-tuning process adapts the model to both image captioning and visual question answering in the astronomy domain. We demonstrate AstroLLaVA's performance on an astronomical visual question answering benchmark and release the model weights, code, and training set to encourage further open source work in this space. Finally, we suggest a roadmap towards general astronomical data alignment with pre-trained language models, and provide an open space for collaboration towards this end for interested researchers.

Galaxy morphology analysis involves classifying galaxies by their shapes and structures. For this task, directly training domain-specific models on large, annotated astronomical datasets is effective but costly. In contrast, fine-tuning vision foundation models on a smaller set of astronomical images is more resource-efficient but generally results in lower accuracy. To harness the benefits of both approaches and address their shortcomings, we propose GalaxAlign, a novel method that fine-tunes pre-trained foundation models to achieve high accuracy on astronomical tasks. Specifically, our method extends a contrastive learning architecture to align three types of data in fine-tuning: (1) a set of schematic symbols representing galaxy shapes and structures, (2) textual labels of these symbols, and (3) galaxy images. This way, GalaxAlign not only eliminates the need for expensive pretraining but also enhances the effectiveness of fine-tuning. Extensive experiments on galaxy classification and similarity search demonstrate that our method effectively fine-tunes general pre-trained models for astronomical tasks by incorporating domain-specific multi-modal knowledge.

Reflected or scattered light produce artefacts in astronomical observations that can negatively impact the scientific study. Hence, automated detection of these artefacts is highly beneficial, especially with the increasing amounts of data gathered. Machine learning methods are well-suited to this problem, but currently there is a lack of annotated data to train such approaches to detect artefacts in astronomical observations. In this work, we present a dataset of images from the XMM-Newton space telescope Optical Monitoring camera showing different types of artefacts. We hand-annotated a sample of 1000 images with artefacts which we use to train automated ML methods. We further demonstrate techniques tailored for accurate detection and masking of artefacts using instance segmentation. We adopt a hybrid approach, combining knowledge from both convolutional neural networks (CNNs) and transformer-based models and use their advantages in segmentation. The presented method and dataset will advance artefact detection in astronomical observations by providing a reproducible baseline. All code and data are made available (https://github.com/ESA-Datalabs/XAMI-model and https://github.com/ESA-Datalabs/XAMI-dataset).

ABSTRACT Vision-Language multimodal Models (VLMs) offer the possibility for zero-shot classification in astronomy: i.e. classification via natural language prompts, with no training. We investigate two models, GPT-4o and LLaVA-NeXT, for zero-shot classification of low-surface brightness galaxies and artifacts, as well as morphological classification of galaxies. We show that with natural language prompts these models achieved significant accuracy (above 80 percent typically) without additional training/fine tuning. We discuss areas that require improvement, especially for LLaVA-NeXT, which is an open source model. Our findings aim to motivate the astronomical community to consider VLMs as a powerful tool for both research and pedagogy, with the prospect that future custom-built or fine-tuned models could perform better. INTRODUCTION Multi-modal foundation models have the potential to impact diverse scientific fields.

Big data has become the norm in astronomy, making it an ideal domain for computer science research. Astronomers typically classify galaxies based on their morphologies, a practice that dates back to Hubble (1936). With small datasets, classification could be performed by individuals or small teams, but the exponential growth of data from modern telescopes necessitates automated classification methods. In December 2013, Winton Capital, Galaxy Zoo, and the Kaggle team created the Galaxy Challenge, which tasked participants with developing models to classify galaxies. The Kaggle Galaxy Zoo dataset has since been widely used by researchers. This study investigates the impact of colour space transformation on classification accuracy and explores the effect of CNN architecture on this relationship. Multiple colour spaces (RGB, XYZ, LAB, etc.) and CNN architectures (VGG, ResNet, DenseNet, Xception, etc.) are considered, utilizing pre-trained models and weights. However, as most pre-trained models are designed for natural RGB images, we examine their performance with transformed, non-natural astronomical images. We test our hypothesis by evaluating individual networks with RGB and transformed colour spaces and examining various ensemble configurations. A minimal hyperparameter search ensures optimal results. Our findings indicate that using transformed colour spaces in individual networks yields higher validation accuracy, and ensembles of networks and colour spaces further improve accuracy. This research aims to validate the utility of colour space transformation for astronomical image classification and serve as a benchmark for future studies.

Coadded astronomical images are created by stacking multiple single-exposure images. Because coadded images are smaller in terms of data size than the single-exposure images they summarize, loading and processing them is less computationally expensive. However, image coaddition introduces additional dependence among pixels, which complicates principled statistical analysis of them. We present a principled Bayesian approach for performing light source parameter inference with coadded astronomical images. Our method implicitly marginalizes over the single-exposure pixel intensities that contribute to the coadded images, giving it the computational efficiency necessary to scale to next-generation astronomical surveys. As a proof of concept, we show that our method for estimating the locations and fluxes of stars using simulated coadds outperforms a method trained on single-exposure images.

We present a new probabilistic method for detecting, deblending, and cataloging astronomical sources called the Bayesian Light Source Separator (BLISS). BLISS is based on deep generative models, which embed neural networks within a Bayesian model. For posterior inference, BLISS uses a new form of variational inference known as Forward Amortized Variational Inference. The BLISS inference routine is fast, requiring a single forward pass of the encoder networks on a GPU once the encoder networks are trained. BLISS can perform fully Bayesian inference on megapixel images in seconds, and produces highly accurate catalogs. BLISS is highly extensible, and has the potential to directly answer downstream scientific questions in addition to producing probabilistic catalogs.

There are a few major observatories in the world, and lots of science to do. In order to observe with one of the big telescopes, astronomers have to propose for time, explaining what they want to do and why it's important. This is a pretty competitive process, and scheduling observations is a hard problem since these observatories are oversubscribed. Telescope time is expensive, too--around $25,000 USD a night or more, depending on which telescope you're using! But what if we could use machine learning to help with this problem of scheduling, making observations shorter or more efficient so we could get through more of them in a night?

But what if we could use machine learning to help with this problem of scheduling, making observations shorter or more efficient so we could get …

Researchers have developed a model for generating pixel-level morphological classifications of astronomical sources. Morpheus can analyze astronomical image data pixel-by-pixel to identify and classify all of the galaxies and stars in large data sets from astronomy surveys. Morphology represents the structural end state of the galaxy formation process, and astronomers have long connected the morphological character of galaxies to the physics of their formation. Therefore, being able to measure such morphologies is a very important task in observational astronomy. There are a number of models that have addressed many of these requirements in complimentary ways.