PathOrchestra: China's First Pathology Foundation Model Unveiled at WAIC 2024

On July 5, during the 2024 World Artificial Intelligence Conference, SenseTime Medical, in collaboration with the team of Associate Chairman Wang Zhe from the Pathology Branch of the Chinese Medical Association and the team of Professor He Yonghong from Tsinghua University, jointly released PathOrchestra, China’s first large pathology model. Trained on the largest digital pathology image dataset in China, the model enables the broadest range of clinical tasks globally, marking a leapfrog breakthrough in the field of pathology AI from “single-model, single-disease” to “one-model, multi-disease.” SenseTime Medical has deeply integrated this achievement into its industry-leading comprehensive solution for intelligent digital pathology departments, helping hospitals establish “one-stop” large-model-driven digital smart pathology departments and ushering in a “new era” of pathological diagnosis.

Caption: Wang Zhe (center), Vice Chairman of the Pathology Branch of the Chinese Medical Association and Vice President of the Pathologists Branch of the Chinese Medical Doctor Association; He Yonghong (left), Professor of Biomedical Engineering and Optics at the Academy of Biomedical and Health Engineering, Shenzhen International Graduate School, Tsinghua University; and Zhang Shaoting (right), Vice President of SenseTime, jointly launched PathOrchestra, China’s first large pathology model.

Pathological diagnosis is hailed as the “gold standard” for disease diagnosis, yet this field has long grappled with challenges such as the prolonged training cycle for pathologists and the uneven distribution of high-quality pathological diagnostic resources. The application of artificial intelligence (AI) technology has ushered in a wave of digital and intelligent transformation in pathology departments both domestically and internationally. However, due to the high resolution of digital pathology slides and the wide variety of diseases involved, it is nearly an “impossible mission” to perform precise annotation-based training for each disease under the traditional AI model training paradigm of “big data + fine-grained annotation.”

“Pathological images exhibit immense diversity, making AI-assisted diagnosis extremely challenging; thus, pathological image processing is often referred to as the ‘crown jewel’ of image analysis. Large pathology models are precisely the key to breaking through the bottlenecks in digital pathology,” said Wang Zhe at the launch ceremony.

Figure Caption: Wang Zhe delivers a keynote presentation at the press conference.

The newly released PathOrchestra, a large pathology model, integrates visual models with large language models. It was trained on China’s largest dataset of nearly 300,000 whole-slide digital pathology images (approximately 300 TB), incorporating multimodal training data such as text and video. Through self-supervised learning on massive datasets, the model can generalize to analyze pathological images of various organs without requiring extensive finely annotated data. Currently covering more than 20 organs, including the lung, breast, liver, and esophagus, it empowers over 100 clinical tasks, such as pan-cancer classification, lesion identification and detection, multi-cancer subtype classification, and biomarker assessment. The model has achieved accuracy rates exceeding 95% in nearly 50 tasks, including multi-organ pan-cancer classification, lymphoma subtype diagnosis, and bladder cancer screening, making it the pathology large model with the broadest clinical application scope globally.

Figure: The PathOrchestra pathology foundation model can empower over 100 clinical tasks

During the training process, the R&D team specifically incorporated data on cancers with high incidence rates in China, such as esophageal and breast cancer, to better align the model’s capabilities with the actual clinical needs in the country. Wang Zhe stated, “PathOrchestra is a concerto performed jointly by visual models, large language models, and multimodal data. We hope this concerto will produce harmonious melodies and provide better services for pathological diagnosis of patients in China and around the world.”

The release of PathOrchestra, a large pathology model, provides robust foundational technical support for the development of digital and intelligent pathology departments. To promote the application of large model technology in pathological diagnosis, SenseTime Medical has developed a comprehensive solution for digital and intelligent pathology departments driven by large models. This solution integrates the entire workflow of smart pathological diagnosis, including digital slide scanning, high-performance slide review, AI-assisted intelligent diagnosis, and data storage.

To meet the regional demands for intelligent pathological analysis across multi-campus hospitals and medical consortia, SenseTime Medical assists hospitals in building an intelligent digital slide management system and a high-performance digital pathology reading platform. This system delivers exceptional performance, enabling "second-level, lag-free access" even under concurrent access by thousands of users. Building on this foundation, and leveraging the underlying capabilities of large pathological models, SenseTime Medical has expanded its AI-assisted diagnostic solutions to cover over one hundred clinical pathology tasks. This effectively alleviates the excessive workload of pathologists and significantly improves diagnostic efficiency.

SenseTime Medical will continue to explore the application potential of large model technologies in the medical field, providing greater empowerment for the digital and intelligent transformation and standardized development of pathology departments in China, thereby supporting the high-quality development of the nation’s healthcare sector.