Given a sequence of tokens generated by a language model, we may want to identify the preceding tokens that influence the model to generate this sequence. Performing such token attribution is expensive; a common approach is to ablate preceding tokens and directly measure their effects. To reduce the cost of token attribution, we revisit attention weights as a heuristic for how a language model uses previous tokens. Naive approaches to attribute model behavior with attention (e.g., averaging attention weights across attention heads to estimate a token's influence) have been found to be unreliable. To attain faithful attributions, we propose treating the attention weights of different attention heads as features. This way, we can learn how to effectively leverage attention weights for attribution (using signal from ablations). Our resulting method, Attribution with Attention (AT2), reliably performs on par with approaches that involve many ablations, while being significantly more efficient. To showcase the utility of AT2, we use it to prune less important parts of a provided context in a question answering setting, improving answer quality. We provide code for AT2 at https://github.com/MadryLab/AT2 .

On some of the creative tool pages, tiny emojis are hidden, offering silly dialogue when you find them. A bacon brush lets you paint ribbons of rashers. A button lets you offer your zine to The Void, scrambling the screen and spewing back a piece of randomly generated glitch art. The software's bright layout and playful text is disarming – and when our guard is down, we can really make interesting things. My first run of zines made through this software were printed on bright yellow paper, and featured images of bananas cut to look like dolphins. There was some text inside, detailing a little of how burned out I felt, how exhaustion had drained the laughter out of me, and how these dolphins had brought some of it back.

Every day, our kids are swept through the world by algorithms. YouTube algorithms decide what videos they watch, GPS algorithms map what route they take to school, Spotify algorithms select what songs they hear, and personal assistants like Siri and Alexa advise them -- all of it driven by artificial intelligence. Kids (and adults!) leave these passive engagements with AI without any physical product -- just an endless stream of passive consumption. Instead of getting carried away by these digital currents, teachers, parents, and caregivers should show kids how to experiment with powerful tools like machine learning and neural networks. We must raise kids who are capable of working alongside artificial intelligence in the workplace.

But we do have to pay the bills. That's why, if your company is looking for a journalistic-quality content blog or you need world-class editors, writers, videographers, designers or HTML jockeys, you've come to the right place. And we can do it for you. Check out our sample content site to get an idea of what our team of 100 well-known journalists, editors, photographers, video producers, animators, cartoonists, web designers and social experts can do for you. Bmod is based on the zine we created from scratch for leading smartphone maker, HTC.