138 Global Radiology Experts Gather in Nanjing to Evaluate Real-World AI Diagnostic Performance Through Competitive Lung Nodule Reading Challenge

According to VCBeat (WeChat ID: vcbeat), the Annual Conference of the Radiologists Branch of the Jiangsu Medical Doctor Association, jointly held with the Neuroimaging Annual Conference of the Radiologists Branch of the Chinese Medical Doctor Association and the Summit of the Imaging Diagnosis Professional Committee of the Bethune Public Welfare Foundation, was successfully convened in Nanjing from August 3 to 5, 2018. The conference invited 138 domestic and international experts in the field of medical imaging, including Professor Jin Zhengyu, Professor Feng Xiaoyuan, Professor Tian Jie, and Professor Sanjiv Sam Gambhir. It also attracted 1,153 registered medical imaging professionals from across China. Gathering in Nanjing, the attendees engaged in in-depth discussions and exchanges on the development of medical imaging in China, residency training, and diagnostic approaches and clinical pathways for various diseases, particularly those affecting the central nervous system. Additionally, practical value assessments were conducted on medical imaging AI, a topic of significant public interest.

Opening Ceremony of the Conference

Whether AI-assisted medical diagnosis can truly serve clinical practice has been one of the hot topics in the medical community in recent years. Currently, society at large places high expectations on AI-based diagnostic systems for medical imaging, particularly hoping that AI will demonstrate practical application value in diagnosing certain common yet serious diseases. Pulmonary nodules and ground-glass opacities are currently the most common imaging manifestations associated with lung cancer, attracting widespread public attention while posing significant challenges for qualitative diagnosis. To address this major clinical issue, during this annual conference, the Chinese Medical Doctor Association’s Radiologist Branch and the Jiangsu Provincial Medical Doctor Association’s Radiologist Branch jointly organized a public-welfare competition titled “AI Empowers My Practice: 100-Participant CT Image Reading Competition for Pulmonary Nodules.” The aim was to evaluate the value of AI in enhancing physicians’ ability to differentiate between benign and malignant pulmonary nodules through imaging diagnosis.

Medical AI Empowers Me: The On-Site Scene of the 100-Participant CT Lung Nodule Reading Competition

Professor Lu Guangming, Director of the Department of Medical Imaging at Nanjing General Hospital of the PLA and the organizer of this image interpretation competition, stated, “We selected 100 CT scans of pulmonary nodules with confirmed pathological results. On one hand, we used an AI-assisted medical diagnostic system to interpret these CT scans; on the other hand, 126 radiologists from tertiary Grade A hospitals and primary care hospitals in Jiangsu Province, holding junior, intermediate, and senior professional titles, reviewed the same 100 CT scans via an AI cloud platform to determine whether the pulmonary lesions were benign or malignant. Through this setup, we aim to compare the overall diagnostic accuracy between physicians and AI, examine whether there are similarities or differences in problem-solving capabilities among physicians from hospitals of different levels and with different professional titles versus AI, and assess to what extent AI can help us address practical clinical issues.”

“Meanwhile, we also aim to identify the current limitations of AI in medical imaging, empowering AI with our physicians’ knowledge and experience to refine these technologies and enhance their clinical value. By leveraging AI assistance, we seek to improve the likelihood of earlier and more accurate diagnosis of lung cancer, thereby benefiting patients and enhancing their quality of life. In clinical practice, 10%–30% of benign pulmonary nodules undergo surgical resection. Although surgical removal of lesions can alleviate patients’ fear of lung cancer and reduce psychological stress, it causes physical harm, wastes valuable medical resources, and increases healthcare costs. One of the primary objectives of this initiative is to determine whether artificial intelligence can help our physicians make definitive diagnoses regarding the benign or malignant nature of pulmonary nodules, thereby bolstering clinicians’ confidence in differential diagnosis, improving work efficiency, alleviating psychological burden for some patients, and facilitating optimal treatment decision-making.”

Professor Lu Guangming, Director of the Department of Medical Imaging at Nanjing General Hospital of the PLA, Accepts an Interview

At the closing ceremony on August 5, Professor Lu Guangming announced the results of the competition. To enable a more precise evaluation of diagnostic outcomes for the same cases among radiologists at different levels and between radiologists and artificial intelligence systems, the competition conducted comparisons across four metrics: area under the curve (AUC), accuracy (ACC), sensitivity (Sen), and specificity (Spe). The competition results were confirmed by Chief Judge Professor Lu Guangming, Deputy Chief Judges Professor Zhang Bing, and Professor Ju Shenghong.

Overall, the data indicate that artificial intelligence (AI) outperformed the average performance of physicians across all senior, intermediate, and junior professional title groups. Nevertheless, physicians still achieved commendable results. A detailed examination of individual metrics for each participating physician revealed that the top-ranked physician surpassed the AI system in all four evaluated indicators. These findings demonstrate that, through deep learning and core algorithms, AI is already capable of providing evidence to support clinical decision-making. Experienced physicians can also empower AI by continuously refining it, thereby enhancing its utility in clinical practice. There is significant complementarity between humans and AI in medical imaging diagnosis. AI systems are well-suited to assume simple, repetitive tasks, while physicians can contribute information and insights to facilitate continuous machine learning and improvement, thus maximizing the advantages of AI to better serve clinical needs. While one should not expect AI to replace physicians, its substantial potential must not be overlooked. Physicians should adopt a proactive attitude toward recognizing, developing, and applying AI technologies at an early stage.

Professor Lu Guangming, Director of the Department of Medical Imaging at Nanjing General Hospital of the PLA, Announced the Competition Results

At the competition, Deepwise’s Dr. Wise™ AI-assisted diagnostic system competed alongside 126 radiologists. Meanwhile, radiologists utilized the Dr. Wise™ Cloud intelligent imaging platform to review CT scans and rapidly determine whether lesions were benign or malignant.

Radiologists at the competition reviewed images on iPads using the Deepwise Dr. Wise™ Cloud intelligent imaging cloud platform.

Following the competition, participating physicians remained highly enthusiastic, engaging in case reviews and discussions at the Deepwise booth, fostering a strong academic atmosphere. After reviewing the correct answers, some contestants acknowledged that the competition, combined with the post-event discussions, revealed gaps in their accumulated experience and a lack of confidence in diagnosing difficult and complex cases. They expressed that the future implementation of AI-assisted diagnosis would boost their diagnostic confidence, enabling them to better serve and benefit patients.

After the competition, participants reviewed the case studies and held discussions using the Dr. Wise™ AI-powered medical auxiliary diagnostic system by Deepwise.

Regarding the results of this competition, Qiao Xin, Co-founder and CEO of Deepwise AI, stated, “AI-based detection of pulmonary nodules employs computer vision technology to examine suspicious lesions in medical images. It is one of the earliest AI technologies implemented in medical imaging and is relatively mature. Physicians are more eager to see breakthroughs in AI-assisted differential diagnosis between benign and malignant lung cancers, enabling them to provide more accurate diagnostic recommendations. Differentiating between benign and malignant lung cancers poses greater scientific challenges, requiring the integration of computer vision with other AI technologies to better align with clinicians’ daily diagnostic workflows. This competition has elevated the application of AI in the medical field to new heights. Deepwise AI is currently the first team bold enough to confront this challenge head-on and publicly release detailed results.”

“We are deeply grateful to the competition organizers for providing us with this opportunity. By recreating the work scenarios of 126 radiologists, our system underwent a rigorous and fair evaluation across various dimensional metrics. Dr. Jin, who secured first place in this competition, made thorough preparations beforehand and delivered an outstanding performance, achieving an excellent Area Under the Curve (AUC) of 0.932 and an accuracy (ACC) of 94%. Ultimately, artificial intelligence is designed to serve physicians. Clinicians can leverage AI insights to make final diagnostic decisions, while the continuous improvement of AI relies on empowerment by physicians’ rich clinical experience and knowledge. Human–AI collaboration is the direction we are striving toward. In the future, we will engage in deeper and closer cooperation with physicians to jointly expand the medical application scenarios of artificial intelligence. Deepwise Doctor: Making healthcare access easier.”