In this short paper we address issues related to building multimodal AI systems for human performance support in manufacturing domains. We make two contributions: we first identify challenges of participatory design and training of such systems, and secondly, to address such challenges, we propose the ACE paradigm: "Action and Control via Explanations". Specifically, we suggest that LLMs can be used to produce explanations in the form of human interpretable "semantic frames", which in turn enable end users to provide data the AI system needs to align its multimodal models and representations, including computer vision, automatic speech recognition, and document inputs. ACE, by using LLMs to "explain" using semantic frames, will help the human and the AI system to collaborate, together building a more accurate model of humans activities and behaviors, and ultimately more accurate predictive outputs for better task support, and better outcomes for human users performing manual tasks.

Traditional studies of memory for meaningful narratives focus on specific stories and their semantic structures but do not address common quantitative features of recall across different narratives. We introduce a statistical ensemble of random trees to represent narratives as hierarchies of key points, where each node is a compressed representation of its descendant leaves, which are the original narrative segments. Recall is modeled as constrained by working memory capacity from this hierarchical structure. Our analytical solution aligns with observations from large-scale narrative recall experiments. Specifically, our model explains that (1) average recall length increases sublinearly with narrative length, and (2) individuals summarize increasingly longer narrative segments in each recall sentence. Additionally, the theory predicts that for sufficiently long narratives, a universal, scale-invariant limit emerges, where the fraction of a narrative summarized by a single recall sentence follows a distribution independent of narrative length.

Interpreting and assessing goal driven actions is vital to understanding and reasoning over complex events. It is important to be able to acquire the knowledge needed for this understanding, though doing so is challenging. We argue that such knowledge can be elicited through a participant achievement lens. We analyze a complex event in a narrative according to the intended achievements of the participants in that narrative, the likely future actions of the participants, and the likelihood of goal success. We collect 6.3K high quality goal and action annotations reflecting our proposed participant achievement lens, with an average weighted Fleiss-Kappa IAA of 80%. Our collection contains annotated alternate versions of each narrative. These alternate versions vary minimally from the "original" story, but can license drastically different inferences. Our findings suggest that while modern large language models can reflect some of the goal-based knowledge we study, they find it challenging to fully capture the design and intent behind concerted actions, even when the model pretraining included the data from which we extracted the goal knowledge. We show that smaller models fine-tuned on our dataset can achieve performance surpassing larger models.

Learning a world model for model-free Reinforcement Learning (RL) agents can significantly improve the sample efficiency by learning policies in imagination. However, building a world model for Multi-Agent RL (MARL) can be particularly challenging due to the scalability issue in a centralized architecture arising from a large number of agents, and also the non-stationarity issue in a decentralized architecture stemming from the inter-dependency among agents. To address both challenges, we propose a novel world model for MARL that learns decentralized local dynamics for scalability, combined with a centralized representation aggregation from all agents. We cast the dynamics learning as an auto-regressive sequence modeling problem over discrete tokens by leveraging the expressive Transformer architecture, in order to model complex local dynamics across different agents and provide accurate and consistent long-term imaginations. As the first pioneering Transformer-based world model for multi-agent systems, we introduce a Perceiver Transformer as an effective solution to enable centralized representation aggregation within this context. Results on Starcraft Multi-Agent Challenge (SMAC) show that it outperforms strong model-free approaches and existing model-based methods in both sample efficiency and overall performance.

Understanding Transformer-based models has attracted significant attention, as they lie at the heart of recent technological advances across machine learning. While most interpretability methods rely on running models over inputs, recent work has shown that a zero-pass approach, where parameters are interpreted directly without a forward/backward pass is feasible for some Transformer parameters, and for two-layer attention networks. In this work, we present a theoretical analysis where all parameters of a trained Transformer are interpreted by projecting them into the embedding space, that is, the space of vocabulary items they operate on. We derive a simple theoretical framework to support our arguments and provide ample evidence for its validity. First, an empirical analysis showing that parameters of both pretrained and fine-tuned models can be interpreted in embedding space. Second, we present two applications of our framework: (a) aligning the parameters of different models that share a vocabulary, and (b) constructing a classifier without training by ``translating'' the parameters of a fine-tuned classifier to parameters of a different model that was only pretrained. Overall, our findings open the door to interpretation methods that, at least in part, abstract away from model specifics and operate in the embedding space only.

Let me introduce you to Marie. Marie is a 28-year-old professional and while on her way home from work is talking to a TikTok follower about the French elections. This follower has an uncanny ability to touch on subjects that mean the most to her. Almost overnight, Marie's social media feeds become increasingly filled with political themes, until on election day, her vote has already been heavily influenced. The trouble is the TikTok follower is not a person, but an artificial intelligence-driven bot, exploiting personal but publicly available data about Marie to manipulate her opinion.

When driving to the office, Marie, CIO of a large bank in France, often has this quote from Bill Gates on her mind: "We always overestimate the change that will occur in the next two years and underestimate the change that will occur in the next 10. Don't let yourself be lulled into inaction." This morning the thought is particularly persistent. Her company has recently embarked on a digital transformation program and she has been looking to recruit employees with different skill sets in areas such as artificial intelligence (AI), cybersecurity and the Internet of Things (IoT). Marie knows that the secret to digital is analog.

The new season of Black Mirror is out, so say goodbye to your faith in humanity. Charlie Brooker's bleak sci-fi anthology series asks the tough questions about technology and how it fucks with us humans--most of the time, it transforms us into huge assholes, or lets us show our inherent assholery in new and frightening ways. Season four is no different. The overall theme is consciousness, with episodes showing the spooky side of everything from artificial intelligence to brain implants, dating apps, and video games. This gives Brooker and his team of writers, actors, and directors--including Jodie Foster!--plenty of room to explore what happens when extremely shitty people encounter technology that society hasn't caught up with yet.

The hexapod robot in the foreground was constructed by Larry Watkins and Todd Heinze. In the background, a robot constructed by Ben Greer ambles along. The challenge of the Autonomous Vehicle Competition, hosted by hobbyist electronics vendor SparkFun at its Boulder, Colorado, headquarters, seems simple enough: Build a robot that can navigate itself around the company's parking lot. Though the AVC course is dotted with small obstacles, it's really just one lap -- a distance of less than 900 feet. But for the majority of competitors, it feels more like the path into Mordor.

Lara Croft, who turns 20 today, has been described as all of these. Born at the height of Britpop, the female protagonist of computer game Tomb Raider became one of the pillars of Cool Britannia - but also provoked the ire of feminists who criticised her sexualised image. Her journey took in two Hollywood films, numerous magazine covers and advertising campaigns but began in the comparatively unglamorous English city of Derby. Tomb Raider was created by a small team of people working for Core Design, a video game developer founded in the city in 1988. "The story goes that within the industry it wasn't easy to sell a female heroine," says Heather Gibson, one of the six developers who created the original game.