Goto

Collaborating Authors

 macgyver


Measuring LLM Novelty As The Frontier Of Original And High-Quality Output

Padmakumar, Vishakh, Yueh-Han, Chen, Pan, Jane, Chen, Valerie, He, He

arXiv.org Artificial Intelligence

As large language models (LLMs) are increasingly used for ideation and scientific discovery, it is important to evaluate their ability to generate novel output. Prior work evaluates novelty as originality with respect to model training data, but original outputs may be of low quality. In contrast, non-expert judges more reliably score quality but may favor memorized outputs, limiting the reliability of human preference as a metric. We introduce a new novelty metric for LLM generations that balances originality and quality -- the harmonic mean of the fraction of \ngrams unseen during training and a task-specific quality score. Using this framework, we identify trends that affect the novelty of generations from three families of open-data models (OLMo, OLMo-2, and Pythia) on three creative tasks: story completion, poetry writing, and creative tool use. We find that model-generated text from some base LLMs is less novel than human-written text from the internet. However, increasing model scale and post-training reliably improves novelty due to improvements in output quality. We also find that improving the base model at the same scale (\eg OLMo 7B to OLMo-2 7B) leads to higher novelty due to higher originality. Finally, we observe that inference-time methods, such as prompting and providing novel in-context examples, have a much smaller effect on novelty, often increasing originality at the expense of quality. This highlights the need for further research into more effective elicitation strategies as we use models for creative applications.


AI Needs To Gain Famed MacGyver Shrewdness, Including For AI Self-Driving Cars

#artificialintelligence

MacGyver-like savviness is needed for AI, including self-driving cars. Who or what is a MacGyver, you might wonder? Well, most people have heard of MacGyver, the TV series and main character that manages to always find a clever means to extricate himself from some puzzling predicament, using his wits to devise a solution out of rather everyday items. Fans know that he carries a Swiss Army knife, rather than a gun, since he believes that using his creativity and inventiveness will always allow him to deal with any untoward circumstance (the knife is handy when you need to defuse a bomb, or when you need to take apart a toaster and reuse its electronics for a completely different purpose and ultimately save your life accordingly). Turns out that you don't necessarily need to have ever seen the show or watched any YouTube clips and yet still nonetheless might know what it signifies to be a "MacGyver" in dealing with a thorny task (it has become part of our lexicon of speaking).


'MacGyver'-like robot can build own tools by assessing form, function of supplies

#artificialintelligence

Thanks to new technology that enables them to create simple tools, robots may be on the verge of their own version of the Stone Age. Using a novel capability to reason about shape, function, and attachment of unrelated parts, researchers have for the first time successfully trained an intelligent agent to create basic tools by combining objects. The concept may sound familiar. It's called "MacGyvering," based off the name of a 1980s--and recently rebooted--television series. In the series, the title character is known for his unconventional problem-solving ability using differing resources available to him.


Sci-Fi Author Robert Heinlein Was Basically MacGyver

WIRED

Robert Heinlein is the legendary author of such classic works as Starship Troopers, The Moon Is a Harsh Mistress, and Stranger in a Strange Land. His books have influenced generations of artists and scientists, including physicist and science fiction writer Gregory Benford. "He was one of the people who propelled me forward to go into the sciences," Benford says in Episode 348 of the Geek's Guide to the Galaxy podcast. "Because his depiction of the prospect of the future of science, engineering--everything--was so enticing. He was my favorite science fiction writer."


US Navy funds 'MacGyver' robot that can create tools - BBC News

AITopics Original Links

A US team aims to build a robot that can work out how to use nearby objects to solve problems or escape threats. The machine has been dubbed a MacGyver Bot, after the TV character who cobbled together devices to escape life-threatening situations. The challenge is to develop software that "understands" what objects are in order to deduce how they can be used. The US Navy is funding the project and says the machines might ultimately be deployed alongside humans. It is providing $900,000 (£562,000) to robotics researchers at the Georgia Institute of Technology to carry out the work.