Babylon Claims Its AI Chatbot Outperforms GPs in Medical Diagnostics Ahead of IPO

Babylon Announces Its Next-Generation Software Achieves Medical Exam Scores Comparable to, and Even Surpassing, the Average Performance of Human PhysiciansThe announcement has sent shockwaves through the healthcare sector, posing a clear threat to the UK’s National Health Service (NHS), which is built on the foundation of general practitioners. This development has sparked intense debate: Will AI encroach upon the domain of physicians, or will it serve as a complementary partner, working hand in hand with human doctors?

At an event held at the Royal College of Physicians in the UK, Babylon launched this artificial intelligence software.

Babylon Health is a leading digital healthcare company in the UK, dedicated to providing convenient and affordable health services to everyone, thereby democratizing healthcare. Currently, Babylon Health serves over 1.4 million users worldwide, spanning regions across Europe, Asia, and Africa.

Babylon Health’s artificial intelligence system communicates with users to understand their specific symptoms, providing health assessments and recommendations for healthy behaviors. For instance, when users describe their symptoms or physical condition to the system, the AI generates personalized health assessments based on its analysis and offers advice on whether to seek hospital care or purchase over-the-counter medications. Its flagship product is currently the telemedicine app “NHS GP at Hand.” Additionally, the app includes services such as health tracking and medication delivery. VCBeat has previously reported on this; for more details, please refer to:AI-powered healthcare platform Babylon Health brings AI doctors to users' smartphones, making medical consultations readily accessible.

The newly announced artificial intelligence system is set to transform the service level of the app. On Babylon’s existing NHS GP at Hand app, if the application suspects a medical issue, it refers the user to a general practitioner. In contrast, the new chatbot will perform its own diagnosis, offering several possible conditions and providing percentage-based probability estimates for each.

However, experts from another medical professional organization have expressed skepticism about the capabilities of artificial intelligence. A Royal College of General Practitioners member stated, “No application or algorithm can replicate everything that a general practitioner does. An application may pass an automated clinical knowledge test, but responses to clinical scenarios are not one-size-fits-all.”

“General practitioners have many factors to consider, need to manage numerous risks, and must be mindful of the potential emotional impact of a diagnosis on patients,” he added.

Of course, some view the challenges brought by artificial intelligence as progress. Malcolm Grant, Chairman of the NHS, who attended the conference that day, expressed optimism: “It is hard to imagine that the historical model of general practitioners, as the cornerstone of the NHS and medicine, would not progress and evolve. We are now at a tipping point in how healthcare services are delivered. Therefore, we have also been closely monitoring what companies like Babylon are doing.”

Babylon also demonstrated its chatbot’s voice-control technology integrated with Amazon smart speakers.

Professor Martin Marshall, Vice-Chair of the RCGP, stated: “Babylon’s software claims to replace doctors, which is our primary concern.”

However, Marshall refuted the notion that physicians would be marginalized. He explained that the purpose of AI is to enable doctors to further follow up on their diagnoses. Artificial intelligence is completely incapable of caring for patients independently; it requires physicians to provide complementary expertise. AI will never replace doctors but rather serve to assist them.

Babylon stated that its chatbot passed representative questions from the Membership of the Royal College of General Practitioners (MRCGP) examination. The MRCGP is a mandatory examination that every member must pass to join the College.

Babylon stated that its artificial intelligence achieved a score of 81% on its first attempt at the test, whereas the average score for human doctors from 2012 to 2017 was 72%.

However, the RCGP stated that it had not provided questions to Babylon and was unable to verify Babylon’s claims. “The university examination questions we actually use are not available in the public domain,” added Professor Martin Marshall, Vice Chair of the RCGP.

Babylon responded to BBC reporters that it used sample questions directly published by the college, some of which had indeed been made public.

“We would be delighted if they could formally share their exam papers with us, enabling us to accurately replicate the examination,” Ali Parsa, CEO of Babylon, told the BBC.

To further test artificial intelligence, Babylon collaborated with physicians from two U.S. institutions—Stanford Primary Care and Yale New Haven Health—as well as doctors from the Royal College of Physicians.

The company stated that it has developed 100 real-world scenarios to test artificial intelligence. It further indicated that the diagnostic capabilities of its chatbot are expected to improve as a result.

Babylon’s goal is to provide affordable healthcare services to people around the world. Since 2016, Babylon has been collaborating with the Rwandan government. The country’s healthcare system was severely devastated following the 1994 genocide, which claimed more than 800,000 lives. Babylon has registered two million users in Rwanda and has conducted tens of thousands of consultations.

In China, Babylon has partnered with tech giant Tencent. This year, the two parties reached a cooperation agreement to explore and conduct research in areas such as health assessments, leveraging Babylon Health’s artificial intelligence system.

VCBeat has previously reported on this matter; for details, please refer to:Tencent Partners with UK Digital Health Company Babylon Health to Conduct Health Assessments Using Artificial Intelligence。