We present a handcrafted neural network that, without training, solves the seemingly difficult problem of encoding an arbitrary set of integers into a single numerical variable, and then recovering the original elements. While using only standard neural network operations -- weighted sums with biases and identity activation -- we make design choices that challenge common notions in this area around representation, continuity of domains, computation, learnability and more. For example, our construction is designed, not learned; it represents multiple values using a single one by simply concatenating digits without compression, and it relies on hardware-level truncation of rightmost digits as a bit-manipulation mechanism. This neural net is not intended for practical application. Instead, we see its resemblance to -- and deviation from -- standard trained autoencoders as an invitation to examine assumptions that may unnecessarily constrain the development of systems and models based on autoencoding and machine learning. Motivated in part by our research on a theory of biological evolution centered around natural autoencoding of species characteristics, we conclude by refining the discussion with a biological perspective.

Climate-economic modeling under uncertainty presents significant computational challenges that may limit policymakers' ability to address climate change e ff ectively. This paper explores neural network-based approaches for solving high-dimensional optimal control problems arising from models that incorporate ambiguity aversion in climate mitigation decisions. We develop a continuous-time endogenous-growth economic model that accounts for multiple mitigation pathways, including emission-free capital and carbon intensity reductions. Given the inherent complexity and high dimensionality of these models, traditional numerical methods become computationally intractable. We benchmark several neural network architectures against finite-di fference generated solutions, evaluating their ability to capture the dynamic interactions between uncertainty, technology transitions, and optimal climate policy. Our findings demonstrate that appropriate neural architecture selection significantly impacts both solution accuracy and computational e fficiency when modeling climate-economic systems under uncertainty. These methodological advances enable more sophisticated modeling of climate policy decisions, allowing for better representation of technology transitions and uncertainty--critical elements for developing effective mitigation strategies in the face of climate change.

In the realm of activity detection for massive machine-type communications, intelligent reflecting surfaces (IRS) have shown significant potential in enhancing coverage for devices lacking direct connections to the base station (BS). However, traditional activity detection methods are typically designed for a single type of channel model, which does not reflect the complexities of real-world scenarios, particularly in systems incorporating IRS. To address this challenge, this paper introduces a novel approach that combines model-driven deep unfolding with a mixture of experts (MoE) framework. By automatically selecting one of three expert designs and applying it to the unfolded projected gradient method, our approach eliminates the need for prior knowledge of channel types between devices and the BS. Simulation results demonstrate that the proposed MoE-augmented deep unfolding method surpasses the traditional covariance-based method and black-box neural network design, delivering superior detection performance under mixed channel fading conditions.

Neural network performance has made great strides in recent years by incorporating key assumptions, often referred to as inductive biases, about data domains into specialized model structures. The designs of popular neural network architectures such as recurrent neural networks, convolutional neural networks, graph neural networks, and transformers all incorporate aspects of their respective task-specific domains into the operations, weight sharing, and connections of their underlying network structure [1, 3, 4, 9, 12]. That specialization, has, in turn, yielded improved efficiency and performance over the more general, fully connected design. Note, however, when implemented, these neural networks tend to be treated as entirely separate architectures, with no explicit connections between them, despite their similar underlying assumptions. Not only does this practice obscures some of the core theoretical similarities between these models, but it can also make modifying the architecture cumbersome when any of those original assumptions about the task domain change even slightly. There exist several well-established methods for describing and reasoning from logical knowledge bases that could trivially describe both the assumptions made on a task's domain and the graphical structure of the neural network itself. Nonetheless, simply using deterministic logic on its own to define that structure, through any given logical programming language, does not immediately align with the constrained structure of the neural network and the uncertainty present in said network's predictions.

The automation of this field supported the development of novel Deep Learning (DL) [62] architectures, especially Convolutional Neural Networks (CNN) [61], that competed with previous state-of-the-art handcrafted models. Since the introduction of CNNs with LeNet [61] and the beginning of Deep Learning with AlexNet [60], most improvements in the field (e.g., deepening the architecture or adding residual connections) were driven by empiricism. NAS aims to put an end to this trial-and-error practice and bring a formal way to smoothen the progress in deep learning architecture design. Moreover, automatically discovering more efficient architectures is particularly relevant in the ecological transition context (i.e., green Deep Learning [119]). The role of manual feature engineering and model development has gradually decreased ever since.

Neural network architectures are most often conceptually designed and described in visual terms, but are implemented by writing error-prone code. PrototypeML is a machine learning development environment that bridges the dichotomy between the design and development processes: it provides a highly intuitive visual neural network design interface that supports (yet abstracts) the full capabilities of the PyTorch deep learning framework, reduces model design and development time, makes debugging easier, and automates many framework and code writing idiosyncrasies. In this paper, we detail the deep learning development deficiencies that drove the implementation of PrototypeML, and propose a hybrid approach to resolve these issues without limiting network expressiveness or reducing code quality. We demonstrate the real-world benefits of a visual approach to neural network design for research, industry and teaching.

This tutorial is based on the Neural Network Module, available on ATOMS. This Neural Network Module is based on the book "Neural Network Design" book by Martin T. Hagan. A function is implemented in neural network module to simplify the plotting of 2 groups of data points. First, we split the data to the source (P), and target (T). We transpose the data to match the format required by the module.

Using the tools of complexity theory, Stephen Judd develops a formal description of associative learning in connectionist networks. He rigorously exposes the computational difficulties in training neural networks and explores how certain design principles will or will not make the problems easier. Judd looks beyond the scope of any one particular learning rule, at a level above the details of neurons. There he finds new issues that arise when great numbers of neurons are employed and he offers fresh insights into design principles that could guide the construction of artificial and biological neural networks. The first part of the book describes the motivations and goals of the study and relates them to current scientific theory.