Large Language Models (LLMs) are typically evaluated through general or domain-specific benchmarks testing capabilities that often lack grounding in the lived realities of end users. Critical domains such as healthcare require evaluations that extend beyond artificial or simulated tasks to reflect the everyday needs, cultural practices, and nuanced contexts of communities. We propose Samiksha, a community-driven evaluation pipeline co-created with civil-society organizations (CSOs) and community members. Our approach enables scalable, automated benchmarking through a culturally aware, community-driven pipeline in which community feedback informs what to evaluate, how the benchmark is built, and how outputs are scored. We demonstrate this approach in the health domain in India. Our analysis highlights how current multilingual LLMs address nuanced community health queries, while also offering a scalable pathway for contextually grounded and inclusive LLM evaluation.

As Conversational Artificial Intelligence (AI) becomes more integrated into everyday life, AI-powered chatbot mobile applications are increasingly adopted across industries, particularly in the healthcare domain. These chatbots offer accessible and 24/7 support, yet their collection and processing of sensitive health data present critical privacy concerns. While prior research has examined chatbot security, privacy issues specific to AI healthcare chatbots have received limited attention. Our study evaluates the privacy practices of 12 widely downloaded AI healthcare chatbot apps available on the App Store and Google Play in the United States. We conducted a three-step assessment analyzing: (1) privacy settings during sign-up, (2) in-app privacy controls, and (3) the content of privacy policies. The analysis identified significant gaps in user data protection. Our findings reveal that half of the examined apps did not present a privacy policy during sign up, and only two provided an option to disable data sharing at that stage. The majority of apps' privacy policies failed to address data protection measures. Moreover, users had minimal control over their personal data. The study provides key insights for information science researchers, developers, and policymakers to improve privacy protections in AI healthcare chatbot apps.

Existing medical text datasets usually take the form of ques- tion and answer pairs that support the task of natural language gener- ation, but lacking the composite annotations of the medical terms. In this study, we publish a Vietnamese dataset of medical questions from patients with sentence-level and entity-level annotations for the Intent Classification and Named Entity Recognition tasks. The tag sets for two tasks are in medical domain and can facilitate the development of task- oriented healthcare chatbots with better comprehension of queries from patients. We train baseline models for the two tasks and propose a simple self-supervised training strategy with span-noise modelling that substan- tially improves the performance. Dataset and code will be published at https://github.com/tadeephuy/ViMQ

Chatbots are fast emerging at the forefront of user engagement across industries. In 2021, healthcare is undoubtedly being touted as one of the most important industries due to the noticeable surge in demand amid the pandemic and its subsequent waves. The Global Healthcare Chatbots Market is expected to exceed over US$ 314.63 Million by 2024 at a CAGR of 20.58%. Chatbots are being seen as those with high potential to revolutionize healthcare. They act as the perfect support system to agents on the floor by providing the first-step resolution to the customer, in terms of understanding intent and need, boost efficiency, and also improve the accuracy of symptom detection and ailment identification, preventive care, feedback procedures, claim filing and processing and more.

Thanks to the advancements in natural language processing (NLP), including LaMDA, GPT-3, GPT-Neo, BERT, and large pre-trained transformer-based language models (PLM), conversational AI has achieved state-of-the-art performance on many tasks. In addition, it has fueled a paradigm shift in NLP. As per Mordor Intelligence, the global ChatBot (conversational AI or virtual assistant) market is expected to touch $100 billion by 2026, growing at a compound annual growth rate of 34.75 percent. Technology has seeped into almost every industry and is changing how businesses and individuals interact and perform everyday tasks. The top five industries that benefit from chatbots include banking and finance, travel, real estate, education, and healthcare.

Chatbots are fast emerging at the forefront of user engagement across industries. In 2021, healthcare is undoubtedly being touted as one of the most important industries due to the noticeable surge in demand amid the pandemic and its subsequent waves. The Global Healthcare Chatbots Market is expected to exceed over US$ 314.63 Million by 2024 at a CAGR of 20.58%. Chatbots are being seen as those with high potential to revolutionize healthcare. They act as the perfect support system to agents on the floor by providing the first-step resolution to the customer, in terms of understanding intent and need, boost efficiency, and also improve the accuracy of symptom detection and ailment identification, preventive care, feedback procedures, claim filing and processing and more.

While more healthcare providers turned to chatbots for patient triaging during the COVID-19 pandemic, researchers say the technologies powering these virtual assistants still need advancing before coming close to imitating a physician's brain, The Washington Post reported. Salt Lake City-based Intermountain Healthcare rolled out artificial intelligence chatbot Scout in March 2020 to navigate calls and concerns from people who were worried they had COVID-19. Health tech company Gyant created the chatbot. After a successful launch, the health system added a symptom-checker to the assistant in June 2020, which asks patients about 30 questions to narrow in on symptoms, such as how bad their headache is, and then set up an appointment or advise them to visit urgent care. Intermountain hopes to eventually add a way to have a "doctor's visit" with Scout, where the chatbot would provide care that is quickly reviewed and verified by a real, human physician, according to the report.

Sapio Smart Healthcare, a division of Indian government advisory firm Sapio Analytics, has developed a chatbot that assists patients from rural and remote areas in India. Based on a press statement, the AKS Sapio Med Bot was designed to understand the "local and personal" concerns of a patient, while assisting them in seeking treatment before getting medical consultations. The chatbot is named after the late Dr Ashok Kumar Srivastava, a surgeon in the tribal areas of Sahibganj and Pakur in the eastern state of Jharkhand, who had demonstrated the delivery of "empathetic healthcare". The company says the chatbot was created to emulate Dr Srivastava's care style following a large-scale evaluation of his mindset, behaviour and persona. Sapio Analytics seeks to build a comprehensive and accessible healthcare system in small towns and villages across India.

Abstract--Chatbots are emerging as a promising platform for accessing and delivering healthcare services. The evidence is in the growing number of publicly available chatbots aiming at taking an active role in the provision of prevention, diagnosis, and treatment services. This article takes a closer look at how these emerging chatbots address design aspects relevant to healthcare service provision, emphasizing the Human-AI interaction aspects and the transparency in AI automation and decision making. CONVERSATIONAL systems are entering The growing number of healthcare chatbots, partly our everyday lives, such as Amazon Alexa and due to the democratization of chatbot development, Google Assistant. Beyond such all-in-one systems, motivates a closer look at how the systems address there are growing demands for building aspects concerning user experience, adoption and trust conversational services in healthcare. The services are in automation, and healthcare provision.

Paddy Padmanabhan is founder and CEO of Damo Consulting, a growth strategy and digital transformation advisory firm that works with healthcare enterprises and global technology companies, and author of The Big Unlock: Harnessing Data and Growing Digital Health Businesses in a Value-Based Era. My first instinct on being hit with the flu (after consulting Dr. Google, of course) was: I'm fine, I don't need to go to an ER. I need to speak with my primary care physician (PCP), get him to prescribe some meds, have someone pick it up for me, stay hydrated and ride it out for the next few days. I typed out a message via the myChart app on my iPhone (I had also lost my voice, so I was in no shape for a phone conversation) to my PCP, gave him a brief run-down of my symptoms, and waited. I had to wait two days before I received an answer -- from an administrative staff member, who asked me to call their after-hours service.