Goto

Collaborating Authors

 dalian


VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applications

He, Wei, Sun, Yueqing, Hao, Hongyan, Hao, Xueyuan, Xia, Zhikang, Gu, Qi, Han, Chengcheng, Zhao, Dengchang, Su, Hui, Zhang, Kefeng, Gao, Man, Su, Xi, Cai, Xiaodong, Cai, Xunliang, Yang, Yu, Zhao, Yunke

arXiv.org Artificial Intelligence

As LLM-based agents are increasingly deployed in real-life scenarios, existing benchmarks fail to capture their inherent complexity of handling extensive information, leveraging diverse resources, and managing dynamic user interactions. To address this gap, we introduce VitaBench, a challenging benchmark that evaluates agents on versatile interactive tasks grounded in real-world settings. Drawing from daily applications in food delivery, in-store consumption, and online travel services, VitaBench presents agents with the most complex life-serving simulation environment to date, comprising 66 tools. Through a framework that eliminates domain-specific policies, we enable flexible composition of these scenarios and tools, yielding 100 cross-scenario tasks (main results) and 300 single-scenario tasks. Each task is derived from multiple real user requests and requires agents to reason across temporal and spatial dimensions, utilize complex tool sets, proactively clarify ambiguous instructions, and track shifting user intent throughout multi-turn conversations. Moreover, we propose a rubric-based sliding window evaluator, enabling robust assessment of diverse solution pathways in complex environments and stochastic interactions. Our comprehensive evaluation reveals that even the most advanced models achieve only 30% success rate on cross-scenario tasks, and less than 50% success rate on others. Overall, we believe VitaBench will serve as a valuable resource for advancing the development of AI agents in practical real-world applications. The code, dataset, and leaderboard are available at https://vitabench.github.io/



Ties in the time of Artificial Intelligence

#artificialintelligence

India's small and medium-sized software firms, wanting to ride the wave of Artificial Intelligence (AI) and the Internet of Things (IOT), could soon find a new home in China's northeast corner. Nasscom, India's IT industry body, has negotiated a deal with local authorities in Dalian, a famous port city on the Liaodong Peninsula, to establish an IT corridor at the Bio-diverse Emerging Science Technology (BEST) City, on the outskirts of the metropolis. In turn, it would allow small Indian cyber companies to register with the new IT cluster. The Chinese companies, who are not a part of the top 100 biggies, are also likely to benefit. "The story is a little more complicated," said Gagan Sabharwal, senior director with Nasscom, referring to his disruptive bottom-up model, which could script the next chapter of India's software story. "We want to marry Indian strengths, especially of small companies, in software with China's heft in hardware."


Ties in the time of Artificial Intelligence

#artificialintelligence

India's small and medium-sized software firms, wanting to ride the wave of Artificial Intelligence (AI) and the Internet of Things (IOT), could soon find a new home in China's northeast corner. Nasscom, India's IT industry body, has negotiated a deal with local authorities in Dalian, a famous port city on the Liaodong Peninsula, to establish an IT corridor at the Bio-diverse Emerging Science Technology (BEST) City, on the outskirts of the metropolis. In turn, it would allow small Indian cyber companies to register with the new IT cluster. The Chinese companies, who are not a part of the top 100 biggies, are also likely to benefit. "The story is a little more complicated," said Gagan Sabharwal, senior director with Nasscom, referring to his disruptive bottom-up model, which could script the next chapter of India's software story. "We want to marry Indian strengths, especially of small companies, in software with China's heft in hardware."


Indian, Chinese IT companies discuss avenues in artificial intelligence

#artificialintelligence

DALIAN: Several Indian and Chinese IT companies on Wednesday got together in China's port city of Dalian for cooperation in the field of artificial intelligence. On the first India-China Dalian IoT (Internet of Things) Conference, the government officials and company representatives from both sides agreed that a lot can be done if India's excellence in software technology and China's expertise in hardware are brought together. The event is being attended by 30 delegates representing the Indian government and companies, while 50 delegates from the Chinese industry are participating in the event. Representatives of Indian companies like Wipro, HCL, Infosys, Cognizant and CBSI Technologies were present. "Chinese hardware needs to be given soul that can come from India's software technology," said Sudhanshu Pandey, Joint Secretary, Department of Commerce (India).


AI Could Add $15 Trillion to Global Economy by 2030

#artificialintelligence

Artificial intelligence might not be so threatening after all. Amid warnings of the economic disruption that robots and automation could unleash on the world economy as traditional roles disappear, researchers are finding that new technologies will help fuel global growth as productivity and consumption soar. AI will contribute as much as $15.7 trillion to the world economy by 2030, according to a PwC report released Wednesday. That's more than the current combined output of China and India. Gains would be split between $6.6 trillion from increased productivity as businesses automate processes and augment their labor forces with new AI technology, and $9.1 trillion from consumption side-effects as shoppers snap up personalized and higher-quality goods, according to the report.


AI Will Add $15.7 Trillion to the Global Economy

#artificialintelligence

Artificial intelligence may not be so threatening after all. Amid warnings of the economic disruption that robots and automation could unleash on the world economy as traditional roles disappear, researchers are finding that new technologies will help fuel global growth as productivity and consumption soar. AI will contribute as much as $15.7 trillion to the world economy by 2030, according to a PwC report Wednesday. That's more than the current combined output of China and India. Gains would be split between $6.6 trillion from increased productivity as businesses automate processes and augment their labor forces with new AI technology, and $9.1 trillion from consumption side-effects as shoppers snap up personalized and higher-quality goods, according to the report.