Inside IBM Watson in Healthcare: Insights from Doctors, AI Experts, and Industry Stakeholders

By Li Yanyu and Wang Shiwei

A review of global corporate history reveals that the enduring success and century-long prosperity of leading companies stem from their ability to continuously launch milestone products and services, thereby generating successive growth curves.

For the tech giant IBM, such products are countless, including the System/360, laser vision correction surgery, RAM, PCs, and the well-known ThinkPad...

Currently, Watson is undoubtedly a product endowed with significant historical responsibility.IBM aims to leverage its cognitive platform, “Watson,” and the power of artificial intelligence to drive its transformation toward “cognitive computing.”

According to John Kelly, Senior Vice President of IBM Cognitive Solutions and IBM Research, the focus of IBM Watson Health is on applying next-generation AI technologies to cancer treatment. By betting on this direction, IBM Watson Health has already developed three unique cancer treatment solutions to assist physicians worldwide in diagnosing and treating patients:

Watson Product Line

Prior to 2017, Watson was widely regarded as a star project; its “breakup” with the Anderson Center brought it “down from its pedestal,” with negative news following in quick succession:

February 2017, the University of Texas, which operates the M.D. Anderson Center, announced the termination of its collaborative project with IBM, paying up to $39 million in compensation to IBM for a project initially valued at $2.4 million under the contract.

May 2017, Chamath Palihapitiya, a seasoned tech investor and founder of the venture capital firm Social Capital, even went so far as to slam it on CNBC in May, stating, “Watson is a joke.”

....

May 2018, IBM Watson Health was exposed to layoffs;

July 2018, IBM Watson Health was exposed by the media for recommending "unsafe and incorrect" cancer treatments;

October 2018, Deborah DiSanzo, head of IBM Watson Health, announced her departure.

A review of media perspectives reveals that skepticism toward Watson products can essentially be categorized into the following four types:

Deng Kan, Chief AI Expert at Fosun Pharma and CTO of Dashu Yida, recently shared his views on Watson with VCBeat. He believes that Watson’s current “predicament” is mainly due to four reasons:

First, excessive promotion claiming that Watson could replace or surpass physicians—possessing cognitive capabilities beyond those of doctors and generating treatment plans for complex and refractory diseases—rapidly inflated external expectations for IBM Watson. In the absence of unified industry testing standards and with the final clinical efficacy of the product yet to be evaluated, such overly exaggerated marketing is detrimental to the product’s long-term healthy development;

Second, Watson’s current theoretical framework is not yet sufficiently robust to support machine reading capabilities;

Third, during the R&D process, there is an insufficient number of personnel involved in product development and research, and the volume of real-world medical records is limited;

Fourth, IBM’s internal strategic planning was flawed. After acquiring the medical imaging company Merge Healthcare for $1 billion in August 2015, IBM failed to achieve notable success in the imaging sector, owing to unclear roadmaps for resource integration.

According to publicly available information, Watson is capable of supporting the following areas, including but not limited to:

· Understanding Natural Language

· Understanding and Analysis of Big Data

· Dynamically analyze various hypotheses and issues

· Sophisticated personalized analysis capabilities

· Optimize question answering based on relevant data

· Distill insights and identify new operational patterns within a short timeframe

·Learn through iteration and explore optimized solutions

According toVCBeatresearch, Watson’s processing logic is an application of open-domain question-answering technologies, including natural language processing, information retrieval, knowledge representation, automated reasoning, and machine learning, based on DeepQA, which was developed for hypothesis generation and large-scale evidence collection, analysis, and evaluation.

In a lengthy article by Professor James Hendler translated by Leiphone, the implementation process of Watson, built upon “associative knowledge,” is revealed. In brief, after physicians input information regarding a patient’s medical condition, the application recommends treatment options by analyzing potentially relevant published studies. The operational workflow of IBM Watson for Oncology includes analyzing patient medical records, providing treatment plans, and ranking them:

1. Analyze patient medical records, including structured and unstructured data;

2. Provide treatment plan options; by analyzing various medical data, IBM Watson for Oncology tailors recommendations for each patient

Provide several treatment options for the physician to select from.

3. Treatment Plan Ranking: Rank various treatment options and specify their medical evidence

Watson's Diagnostic Process, VCBeat Mapping

Watson “learns” by continuously adjusting its internal processing algorithms to yield high-probability correct answers for certain tasks, such as identifying cancer in radiological images. The correct answers must be known in advance so that the system can be informed of when it has performed correctly and when it has erred. The more training cases the system processes, the higher its accuracy rate becomes.

By analyzing options and ranking them by optimality, the system ultimately provides patients with a comprehensive report of recommended cancer treatment plans. This report includes recommended options, alternatives for consideration, and non-recommended options. For each recommendation, Watson “Doctor” cites sources and evidence, ranked by credibility, for the attending physician’s reference. If a specific treatment plan is selected, the system also provides relevant information such as survival rates, incidence of adverse reactions, and drug interactions, thereby assisting physicians in comprehensively evaluating the efficacy and risks of the chosen regimen.

In fact, most of the criticism directed at Watson pertains to its exaggerated marketing and overly optimistic statements regarding its prospects.

If Watson has yet to achieve significant breakthroughs, one of the most obvious obstacles is its need for specific types of data for “training,” which are often either scarce or difficult to access. This is not a problem unique to Watson; it is a common challenge facing the entire field of medical machine learning.

Although the scarcity of data has slowed Watson’s development, it has had an even greater impact on IBM’s competitors. In the training of algorithms and models for medical AI, the most effective way to obtain data is through close collaboration with large healthcare institutions, which tend to be highly conservative in their adoption of technology.

Due to the rigorous nature of medical practice, AI products require extensive clinical data validation to gain recognition. Although major hospitals currently maintain an open attitude toward medical AI, competition among similar products is exceptionally fierce. It is common for a single department to have multiple comparable products available for physicians’ use, posing certain challenges in acquiring clinical data.

The most distinctive feature of Watson “Doctor” is its ability to learn and improve rapidly. In 2017, Watson “Doctor” added four new cancer types and six new treatment options, with continuous upgrades and improvements across all performance metrics. By 2018, Watson “Doctor” had expanded its treatment recommendations to cover 13 cancer types: breast cancer, lung cancer, rectal cancer, colon cancer, gastric cancer, cervical cancer, ovarian cancer, prostate cancer, bladder cancer, liver cancer, thyroid cancer, esophageal cancer, and endometrial cancer. According to reports, the research papers and data used for Watson “Doctor”’s learning now include publications by experts from Hong Kong.

On one side, there are doubts about IBM and Watson; on the other, IBM has directly responded to the continuous wave of negative public opinion.

Just three days after The Wall Street Journal published its report on Watson’s predicament, John Kelly, Senior Vice President of IBM Cognitive Solutions and IBM Research, swiftly responded:

“IBM has much to be proud of, including its pioneering research in Watson Health. Regrettably, some media reports, including an article published by The Wall Street Journal on August 11, have distorted and overlooked certain facts, implying that IBM has not made ‘sufficient’ progress in applying the advantages of artificial intelligence to the healthcare sector. It is imperative to clarify the truth.”

According to data provided by The Wall Street Journal, the largest AI product in the Watson Health portfolio is Watson for Oncology, for which IBM typically charges $200 to $1,000 per patient, with consultation fees required in certain cases.

Since 2012, Memorial Sloan Kettering Cancer Center in New York has been assisting IBM in training the software (which is not used for patient care). Experts from the hospital have collaborated with IBM engineers to rank relevant features of medical histories, such as tumor location and comorbidities, and to evaluate medical research on specific therapies. Watson’s ability to match test cases with treatments was then assessed, and engineers adjusted the output until it aligned with physicians’ judgments.

Reports issued by IBM indicate that Watson is working closely with top cancer research institutions, including Memorial Sloan Kettering Cancer Center and the Mayo Clinic, to jointly develop and refine cognitive solutions. These solutions are currently deployed in 230 hospitals and healthcare institutions worldwide. By the end of June 2018, the number of patients served had reached 84,000, nearly doubling the figure recorded at the end of 2017.

In response to questions about real-world cases, John Kelly presented a series of Watson use cases and reported data:

A report presented by physicians from the Mayo Clinic at the American Society of Clinical Oncology (ASCO) Annual Meeting stated that, following the implementation of the Watson for Clinical Trial Matching solution, the enrollment rate for breast cancer trials increased by 80% (rising to 6.3 patients per month, compared with 3.5 patients per month during the preceding 18 months).

Dr. Thaddeus Beck and the research team at Highland Oncology Group reported that the Watson for Clinical Trial Matching solution reduced the time required for clinical trial matching by 78%.

Dr. Somashekhar and Manipal Hospitals stated earlier this year in *Annals of Oncology* that the concordance rate between breast cancer treatment plans recommended by the Watson for Oncology solution and those proposed by the hospital’s multidisciplinary tumor board reached 93%. More recently, they reported that they have applied the Watson for Oncology solution to all complex cases reviewed by the multidisciplinary tumor board, resulting in changes to the recommended treatment plans for 9%–11% of patient cases.

Dr. Michael Kelley and the Department of Veterans Affairs have just renewed the contract for Watson Genomics Solutions. To date, nearly 3,000 veterans with Stage IV cancer have received treatment supported by this solution.

Dr. William Kim and the University of North Carolina Lineberger Comprehensive Cancer Center published a study showing that Watson for Genomics identified novel, actionable gene mutations in 32% of patients.

In IBM’s response, John Kelly emphasized that the role of technology is to help physicians provide better care and treatment for patients. Regarding this product, the core issue IBM aims to address is: “Can Watson help oncologists develop more effective treatment plans for their patients?” The key lies in “assisting” rather than “replacing.”

This perspective aligns somewhat with the views shared by certain medical AI companies during discussions with VCBeat. Industry insiders have noted that, as startups, some companies look to Watson as a model to learn from its strengths. Furthermore, they believe that most of the current issues surrounding Watson stem from misleading media publicity. IBM did not intend for this product to make diagnoses on behalf of physicians; rather, artificial intelligence should serve only as an assistant to doctors. Moreover, for a technology-driven company like IBM, its products are unlikely to stray far from clinical practice during the research and development process.

Although Dr. Deng Kan believes that IBM’s products have made certain missteps in terms of marketing, technology, and strategy, he has also positively acknowledged Watson’s “historical contribution” to the application of artificial intelligence in the healthcare sector—namely, transforming the methodology of medical practice.

Taking the Clinical Decision Support System (CDSS) as an example, research in this field was first initiated around 1970 by researchers at the University of Pittsburgh in the United States. At that time, the primary method employed involved extracting medical rules from medical textbooks and literature, with these rules expressed in “if-then” formal logic. By inputting patient symptoms to identify the corresponding “if” conditions, the system would then infer the diagnosed disease based on the “then” conclusions.

Two years later, in 1972, professors at Stanford University also began conducting similar research under the project name MYCIN. MYCIN was primarily based on an if-then rule base, which later acquired a prominent new designation: “Expert System.”

If-then rules are binary and clearly distinct. Later, probability was introduced into the rules, and a network structure was used to interconnect numerous rules. This gave rise to Bayesian networks, also known as causal networks, which became highly popular in the 1990s. Although Bayesian networks are mathematically elegant, their practical implementation is extremely complex, and effective real-world applications have remained elusive. Consequently, after a period of heightened interest, Bayesian networks have now fallen out of favor.

It was not until 2011 that IBM Watson emerged. Initially, IBM Watson was a research project at IBM Research, with the team focusing on natural language processing since 2006. They trained machines to extract statements such as “The Portuguese explorer Vasco da Gama arrived in Calicut on May 20, 1498” from literature. They also extracted other statements, such as “Calicut is located in southwestern India,” from additional sources. By linking these two statements, the system inferred the conclusion: “The Portuguese landed in India in 1498.”

The most famous outcome of this project was Watson’s participation in the American quiz show *Jeopardy!* in 2011, where it defeated human contestants. This competition was highly significant, effectively heralding the advent of the era of AI applications and marking a major milestone in the history of artificial intelligence. On the path to transforming technological achievements into profitable products, IBM ultimately chose to focus on AI after thorough evaluation.

Choosing healthcare was the right strategic pivot for IBM, as the medical services market offers immense potential. The approach of mining and organizing clinical diagnostic and treatment experience from human physicians within massive volumes of medical records is based on inductive reasoning. In contrast, the previous method of extracting rules from medical literature relied on deductive reasoning. IBM Watson has transformed the methodology employed in AI-driven healthcare.

The history of modern science demonstrates that shifts in methodology can lead to transformative changes. IBM Watson extracts clinical diagnostic insights from vast volumes of medical records, rather than extracting and reasoning through medical rules from literature—a fundamental methodological shift. In essence, IBM Watson has spearheaded a cognitive revolution.

Currently, many people regard Google’s Google Medical Brain project as the industry leader in artificial intelligence for healthcare. The Google Medical Brain project also mines and organizes clinical pathways of human physicians from vast amounts of medical records. In April this year, it published a paper in Nature magazine that systematically outlined the overall project roadmap of Google Brain. Although Google Medical Brain has slight advantages in certain details, its methodology is consistent with that of Watson.

Shanghai Tenth People’s Hospital and Zhoukou Traditional Chinese Medicine Hospital introduced Watson in August 2017 and February 2018, respectively, primarily implementing it within their oncology departments. VCBeat interviewed Dr. Xu Qing, Director of the Oncology Department at Shanghai Tenth People’s Hospital, and Dr. Zhang Yueqiang, Director of the Oncology Department at Zhoukou Traditional Chinese Medicine Hospital, to gain insights into the utilization of Watson in their respective departments.

Xu Qing told VCBeat that since the Department of Oncology at Shanghai Tenth People’s Hospital introduced Watson, it has completed auxiliary decision-making for nearly 650 cancer patients, accounting for approximately 50% of all outpatient cases. The cancers involved include colorectal cancer, gastric cancer, lung cancer, and other common types.

Doctors typically recommend using Watson when a patient’s condition is relatively complex. However, since Watson-assisted decision-making is not yet covered by medical insurance, patients can only use it if they can afford the cost. “Some patients also seek out Watson specifically due to its reputation,” said Xu Qing. The introduction of Watson has, to some extent, enhanced the appeal of the Oncology Department at Shanghai Tenth People’s Hospital to patients.

"The proportion of oncology patients at Zhoukou Traditional Chinese Medicine Hospital using Watson is relatively lower. Zhang Yueqiang told VCBeat that only about 10% of patients have used Watson. Zhang Yueqiang believes that whether patients use Watson is strongly correlated with the complexity of their disease and their financial capacity."

In terms of accuracy, both Xu Qing and Zhang Yueqiang stated that Watson is essentially an auxiliary decision-making and treatment tool, and its accuracy should be measured by the level of consistency between its treatment recommendations and clinical disease guidelines. Since Watson has learned a vast amount of clinical disease guidelines and medical literature during data training, its accuracy is quite high. Zhang Yueqiang indicated that, according to his estimates, Watson’s accuracy can reach 90%. “Watson’s treatment plans are based on a massive volume of the latest research findings; sometimes, the treatment plans it provides are even more reasonable,” Zhang Yueqiang added.

However, Watson does not fully meet physicians’ expectations. Xu Qing and Zhang Yueqiang stated that Watson’s current level of localization is insufficient to fully satisfy clinical needs. The primary manifestation of its poor adaptation in Chinese hospitals is that it recommends medications not yet marketed in China.

As early as 2017, VCBeat interviewed doctors who were among the first in China to use Watson. Gu Xidong, an attending physician in the Breast Surgery Department of Zhejiang Provincial Hospital of Traditional Chinese Medicine, stated that for physicians,There are four applications of Watson.: (Original text: "Exclusive Interview with One of China’s First Physicians to Use Watson: He Identifies 4 Major Applications and 2 Shortcomings of AI》）

The first application is to select the optimal treatment regimen based on evidence-based research.

The second application is to reduce misdiagnosis by physicians;

The third use is to provide physicians with novel treatment options for reference;

The fourth use is to assist in the training of young doctors.

Regarding Watson's shortcomings,Gu Xidong stated that, first, Watson itself is positioned as a tool to assist physicians and cannot adjust to the real-life circumstances of patients. It can only recommend treatment plans based on objective pathological indicators. However, oncology treatment is highly complex; the optimal medical regimen is not always one that patients can accept. In many cases, physicians must adjust treatments according to the patient’s actual condition and provide persuasion and emotional support—tasks that Watson is incapable of performing.

Second, Watson is currently unable to integrate traditional Chinese medicine (TCM) with Western medicine. Gu Xidong stated that TCM is gradually gaining recognition, and some Chinese physicians incorporate TCM principles to varying degrees when managing patients’ conditions; however, Watson does not yet possess this capability.

Therefore, studying more domestic clinical cases is essential to better localize Watson.

Wang Dong, Director of the Robot-Assisted Minimally Invasive Surgery Center at Sichuan Provincial People’s Hospital, has also stated publicly that physicians aspire to achieve genuine AI integration in every stage of clinical care. Currently, the Watson system is relatively mature in the diagnostic phase.

Professor Miao Xiaohui from Changzheng Hospital, affiliated with the Second Military Medical University, shared his perspectives on AI from the standpoint of a clinician and user. First, AI does not involve fabrication; it is merely immature at present. Tumor chemotherapy regimens have relied solely on Western guidelines and the experience of select experts. A key limitation of guideline-directed therapy is that it remains theoretical—"armchair strategy"—and future approaches must incorporate broader expert clinical experience. Second, over the past few years, AI’s diagnostic and therapeutic capabilities have been dismissed by “senior experts.” Even as these experts utilize the da Vinci Surgical Robot, they have not perceived AI as a threat. Third, AI development will inevitably experience ups and downs, requiring adaptation and adjustment throughout this process. “The era of AI in medicine is already upon us. Whether you like it or not, it is here and will sooner or later take center stage. While its emergence is driven by human will, its ultimate trajectory is beyond human control.”

Additionally, Zhang Yueqiang stated in the interview that artificial intelligence represents a major trend, and hospitals and physicians should maintain an open attitude toward the integration of medical AI products into clinical departments. Given that medical AI is still in its early stages, it is believed that future medical AI products will become more intelligent and practical.

To assess the actual implementation of Watson in real-world medical scenarios in China and to verify whether external skepticism was justified, VCBeat made multiple attempts to contact Zhang Wenming, General Manager for Greater China and the Asia-Pacific Region at Watson Health. He did not respond to our inquiries. Subsequently, we reached out to Baiyang Intelligent Technology (hereinafter referred to as “Baiyang”), Watson’s general agent in China. Wang Biquan, Chief Marketing Officer of Baiyang, accepted our interview request.

Wang Biquan believes that many of the external doubts and accusations against Watson are unfounded and inconsistent with the facts. He emphasizes that, first, Watson is not a robot but an artificial intelligence system capable of understanding, reasoning, analysis, learning, and interaction. Second, Watson does not have prescription authority; it merely provides decision support to oncologists, enabling them to save substantial time to care for more patients. “Just as Watson was to Sherlock Holmes, Watson is the doctor’s best ‘assistant.’”

“They have never used Watson for Oncology, never conducted on-site research with real hospital users, and never clarified the actual usage scenarios of Watson for Oncology with IBM or official operators. Instead, they merely gathered some online information and opinions, and then took things out of context to distort its meaning in a seemingly professional manner. We strongly disapprove of such behavior, as it may, to some extent, impact Watson.” This also marks Watson’s first public response.

Furthermore, Wang Biquan responded to the previously circulated online incident regarding “Watson prescribing the wrong medication.”

The incident was triggered when the U.S. medical media outlet STAT released confidential internal documents from IBM, which recorded strong criticisms of Watson by clinicians and demonstrated through case studies that there were serious issues with both the process by which Watson provided medical recommendations and its underlying technology.

In one case, a 65-year-old male was diagnosed with lung cancer accompanied by severe hemorrhage. Watson recommended chemotherapy and the anticancer drug bevacizumab. However, bevacizumab is known to increase the risk of bleeding as one of its side effects.

Subsequently, MSK responded that the “65-year-old male lung cancer patient” was a fictional case provided by oncologists at the cancer center during Watson’s training, intended solely to enhance Watson’s clinical decision-support capabilities. Wang Biquan pointed out that in real-world practice, bevacizumab is a prescription drug and must be prescribed, dispensed, and used only with a physician’s prescription. Furthermore, when entering patient information into the Watson system, physicians are required to indicate whether the patient has hemoptysis; if “yes” is selected, the system automatically filters out medications that may increase the risk of bleeding.

Wang Biquan stated that since Watson’s deployment in China, hospitals have demonstrated clear demand for the system. They require tools to train young physicians, study advanced international cases, and validate treatment plans—all needs that Watson can fulfill.

Meanwhile, cancer treatment in China also requires Watson to provide standardized, evidence-based reference recommendations, sparing patients the hardship of long-distance travel. Furthermore, hospitals are highly receptive to this solution, as real-world needs align well with Watson’s actual product value.

“The difficulties that exceeded our expectations were mainly reflected in the public’s insufficient understanding of Watson and the negative coverage by foreign media.” Wang Biquan appeared somewhat helpless. “AI is an entirely new field, giving rise to a wide variety of perceptions. Some people have exceptionally high expectations for AI, while others harbor feelings of fear and resistance. Therefore, when a novel technology is introduced to the market, controversies of one kind or another are inevitable.”

Wang Biquan added that although medical AI has received support at the national level, the biggest pain point currently is that billing items in some cities have not yet been clarified, which is unfavorable for the promotion of new technologies and products.

Regarding the widely discussed mass layoffs at Watson, IBM previously responded publicly that after a period of significant acquisitions and migration to IBM Cloud, it needed to rationalize its business operations, leading to workforce adjustments. This is normal from a business perspective. IBM still has hundreds of job openings and continues to recruit in key areas such as data management, analytics, and artificial intelligence.

In March 2017, Watson Health signed an agreement with Baheal Medical, marking the official entry of Watson for Oncology into the Chinese market.

It is reported that Watson, developed by IBM, has been trained since 2011 by the Memorial Sloan Kettering Cancer Center (MSKCC), a world-leading cancer treatment center. To date, it has processed data from over 330 medical specialty journals, more than 250 medical textbooks, and 27 million research papers. In addition to these research datasets, Watson also accumulates data through real-world evidence and clinical cases.

Currently, Watson is the only application-level AI tool and the only AI tool capable of providing a second treatment opinion. Watson’s second treatment opinion is based on standardized treatment recommendations derived from the MSK decision-making process.

Watson enables physicians to grasp the latest literature in less time. Meanwhile, with its robust capabilities in reasoning, analysis, and interaction, it provides evidence-based, personalized, and prioritized treatment recommendations, citing sources and rationale for each suggestion to guide treating physicians—all within no more than 10 seconds.

As of press time, Watson has been deployed in nearly 80 hospitals across 43 cities in 22 provinces throughout China.“Feedback from hospitals, physicians using Watson, and patients indicates recognition of Watson’s auxiliary advisory capabilities, with many acknowledging its significant utility in assisting clinical diagnosis and treatment as well as in educating junior physicians,” Wang Biquan told VCBeat.

According to Wang Biquan, Watson has been deeply integrated into clinical workflows at its deployed hospitals, facilitating disciplinary development and enhancing the efficiency of clinical decision-making. Specifically, in real-world medical settings, Watson provides treatment recommendations based on evidence-based medicine, participates in multidisciplinary team (MDT) discussions, and rapidly delivers comprehensive materials to support medical decision-making.

According to reports, Baheal Medical and IBM are actively collaborating with local medical institutions to promote the localization of Watson. In the future, Watson will learn more from domestic clinical guidelines, literature, and real-world cases, thereby better aligning with the diagnostic and treatment needs of oncologists in China.

One thing is certain: regardless of how Watson’s future unfolds, humanity’s exploration of leveraging AI to drive a revolution in medical technology will continue.