impulse control
AgentZero++: Modeling Fear-Based Behavior
Malhotra, Vrinda, Li, Jiaman, Pisupati, Nandini
We present AgentZero++, an agent-based model that integrates cognitive, emotional, and social mechanisms to simulate decentralized collective violence in spatially distributed systems. Building on Epstein's Agent\_Zero framework, we extend the original model with eight behavioral enhancements: age-based impulse control; memory-based risk estimation; affect-cognition coupling; endogenous destructive radius; fight-or-flight dynamics; affective homophily; retaliatory damage; and multi-agent coordination. These additions allow agents to adapt based on internal states, previous experiences, and social feedback, producing emergent dynamics such as protest asymmetries, escalation cycles, and localized retaliation. Implemented in Python using the Mesa ABM framework, AgentZero++ enables modular experimentation and visualization of how micro-level cognitive heterogeneity shapes macro-level conflict patterns. Our results highlight how small variations in memory, reactivity, and affective alignment can amplify or dampen unrest through feedback loops. By explicitly modeling emotional thresholds, identity-driven behavior, and adaptive networks, this work contributes a flexible and extensible platform for analyzing affective contagion and psychologically grounded collective action.
Stochastic Games with Minimally Bounded Action Costs
In many multi-player interactions, players incur strictly positive costs each time they execute actions e.g. 'menu costs' or transaction costs in financial systems. Since acting at each available opportunity would accumulate prohibitively large costs, the resulting decision problem is one in which players must make strategic decisions about when to execute actions in addition to their choice of action. This paper analyses a discrete-time stochastic game (SG) in which players face minimally bounded positive costs for each action and influence the system using impulse controls. We prove SGs of two-sided impulse control have a unique value and characterise the saddle point equilibrium in which the players execute actions at strategically chosen times in accordance with Markovian strategies. We prove the game respects a dynamic programming principle and that the Markov perfect equilibrium can be computed as a limit point of a sequence of Bellman operations. We then introduce a new Q-learning variant which we show converges almost surely to the value of the game enabling solutions to be extracted in unknown settings. Lastly, we extend our results to settings with budgetory constraints.
Timing is Everything: Learning to Act Selectively with Costly Actions and Budgetary Constraints
Mguni, David, Sootla, Aivar, Ziomek, Juliusz, Slumbers, Oliver, Dai, Zipeng, Shao, Kun, Wang, Jun
Many real-world settings involve costs for performing actions; transaction costs in financial systems and fuel costs being common examples. In these settings, performing actions at each time step quickly accumulates costs leading to vastly suboptimal outcomes. Additionally, repeatedly acting produces wear and tear and ultimately, damage. Determining \textit{when to act} is crucial for achieving successful outcomes and yet, the challenge of efficiently \textit{learning} to behave optimally when actions incur minimally bounded costs remains unresolved. In this paper, we introduce a reinforcement learning (RL) framework named \textbf{L}earnable \textbf{I}mpulse \textbf{C}ontrol \textbf{R}einforcement \textbf{A}lgorithm (LICRA), for learning to optimally select both when to act and which actions to take when actions incur costs. At the core of LICRA is a nested structure that combines RL and a form of policy known as \textit{impulse control} which learns to maximise objectives when actions incur costs. We prove that LICRA, which seamlessly adopts any RL method, converges to policies that optimally select when to perform actions and their optimal magnitudes. We then augment LICRA to handle problems in which the agent can perform at most $k<\infty$ actions and more generally, faces a budget constraint. We show LICRA learns the optimal value function and ensures budget constraints are satisfied almost surely. We demonstrate empirically LICRA's superior performance against benchmark RL methods in OpenAI gym's \textit{Lunar Lander} and in \textit{Highway} environments and a variant of the Merton portfolio problem within finance.
Deep combinatorial optimisation for optimal stopping time problems and stochastic impulse control. Application to swing options pricing and fixed transaction costs options hedging
Deschatre, Thomas, Mikael, Joseph
American-style options are used not only by traditional asset managers but also by energy companies to hedge "optimised assets" by finding optimal decisions to optimise their P&L and find their value. A common modelling of a power plant unit P&L is done using swing options which are American options allowing to exercise at most l times the option with possibly a constraint on the delay between two exercise dates (see Carmona and Touzi (2008) or Warin (2012) for gas storage modelling). Formally, for T 0, we are given a stochastic processes ( X t) t 0 defined on a probability space (โฆ, F, F ( F t) t 0, P) and one wants to find an increasing sequence of F stopping times ฯ ( ฯ 1,ฯ 2,...,ฯ l) that maximises the expectation of some objective function f E Pnull l null i 1f ( ฯ i,X ฯ i) 1 ฯ i Tnull . Numerical methods to solve the optimal stopping problem when l 1,f ( x,t) e rt g (x) and X is Markovian include: - Dynamic programming equation: the option price P 0 is computed using the following backward discrete scheme over a grid t 0 0 t 1 ... t N T: P t N g ( X T), P t i max( g ( X t i),e r (t i 1 t i) E P( P t i 1 F t i)), i 0,...,N 1 . One then needs to perform regression to compute the conditional expectations, see Longstaff and Schwartz (2001) or Bouchard and Warin (2012).
The Philosopher Who Says We Should Play God - Issue 72: Quandary
Australian bioethicist Julian Savulescu has a knack for provocation. He says most of us would readily accept it if it benefited us. As for eugenics--creating smarter, stronger, more beautiful babies--he believes we have an ethical obligation to use advanced technology to select the best possible children. A protรฉgรฉ of the philosopher Peter Singer, Savulescu is a prominent moral philosopher at the University of Oxford, where he directs the Uehiro Centre for Practical Ethics. He sees nothing wrong with doping to help cyclists climb those steep mountains in the Tour de France. Some elite athletes will always cheat to boost their performance, so instead of trying to enforce rules that will be broken, he claims we'd be better off with a system that allows low-dose doping. So does Savulescu just get off being outrageous? "I actually think of myself as the voice of common sense," he says, though he admits to receiving his share of hate mail.
Can Your Genes Make You Kill?
It was a fall night in 2006, when Bradley Waldroup walked out of his rural trailer in southeastern Tennessee, carrying his .22 His estranged wife and her friend, Leslie Bradshaw, had just pulled up to drop off the Waldroups' four children. Waldroup began arguing with his wife and Bradshaw, who was unloading the car. He used a knife to cut her head open. He then chased his wife with the knife and a machete, managing to slice off one of her pinkies before dragging her into the trailer. There, he told their frightened children, "Come tell your mama goodbye," because it was the last time they'd ever see her. Miraculously, his wife managed to slip his grasp and escape.