DeepMind's Streams App Faces Regulatory Hurdles Amid Scrutiny Over Medical Device Classification and Patient Data Use

As previously reported by VCBeat (WeChat ID: vcbeat), in February this year, DeepMind, the AI company under Google, announced its full-scale entry into the healthcare sector through a data access partnership with the UK’s National Health Service (NHS). However, to be frank, DeepMind Health has not yet operated entirely smoothly and continues to face numerous regulatory controversies. (Related article:AI Robot AlphaGo Enters the Healthcare Sector）

Previously, DeepMind stated that it would no longer use a warning system app designed in collaboration with the NHS to identify patients at high risk of acute kidney injury, before the software was registered as a medical device by the UK’s Medicines and Healthcare products Regulatory Agency (MHRA). Notably, this application, named Streams, had already been deployed for patient care in multiple NHS hospitals. The Royal Free London NHS Foundation Trust previously disclosed that the app had been used by up to six clinicians during three “user tests” at its London hospital. In other words, hospitals were using unregistered medical devices, drawing profit-driven commercial entities into this contentious trial.

In May this year, following controversies surrounding the privacy of health data for 1.6 million patients in its partnership with DeepMind, Royal Free NHS Trust discontinued the use of the problematic app, Streams. One of the reasons leading to this outcome was the disagreement over whether the app should be registered as a medical device application. (Related article:Google DeepMind Secures Data on 1.6 Million Patients from Three London Hospitals to Develop Early-Warning App)

A spokesperson for the regulatory authority told reporters, “According to the MHRA, Streams is developed based on the UK NHS AKI alert platform and is likely to be classified as a Class I medical device. For Class I medical devices, manufacturers must hold all relevant testing and validation data (including software) to support the device’s intended purpose. The competent authorities within the EU (in the UK, this is the MHRA) may request access to all documentation relating to the device at any time.”

Clearly, before accelerating the rollout of more “user tests” leveraging the NHS dataset, DeepMind needs to reflect on its approach and engage in face-to-face discussions with relevant healthcare regulatory authorities in advance.

Earlier this month, Mustafa Suleyman, co-founder of DeepMind, wrote on the content publishing platform Medium that respecting NHS health data is what truly matters. He went on to point out: “Different regulatory bodies grant different types of approval and oversight for the use of NHS data, involving institutions such as the HSCIC, HRA, MHRA, ICO, and Caldicott Guardians. We are committed to cooperating with all these organizations and are confident that, with their support, we can do an excellent job.”

“Regarding the latest progress in our collaboration with the MHRA, a DeepMind spokesperson told reporters: ‘We are still in the stage of developing the prototype for the Streams app. Before final completion, we must ensure that it complies with all applicable EU and UK medical device regulations, and we are currently working with the MHRA on this basis. Only after the Streams application has obtained full certification, received the CE mark, and been registered will we market it as a medical device.’”

“DeepMind is currently working with the MHRA to ensure that the device complies with all relevant legal requirements for medical devices before it is placed on the market,” added a Royal Free spokesperson, who confirmed that the trust continues to have confidence in Stream.

Furthermore, both Royal Free and DeepMind have insisted that they have not yet conducted clinical trials on the relevant software programs, nor will they proceed with full-scale market deployment, as both actions would likely require them to obtain additional regulatory approvals beforehand. Nevertheless, Royal Free is obligated to disclose under what circumstances user testing would be considered a clinical trial, and how the number of patients involved in such “user testing” is determined.

DeepMind’s initial NHS collaboration with Royal Free also sparked early controversy, centering on the scope of data sharing agreed upon by both parties.

The exceptionally broad data-sharing agreement encompasses five years of historical hospital inpatient data and the medical records of millions of potential NHS patients, without requiring patient consent. The scope of this data-sharing agreement has raised concerns among health data privacy organizations such as MedConfidential, which have questioned why DeepMind was granted access to such a large volume of identifiable patient data.

DeepMind and Royal Free claimed that they required extensive patient data to enable the Streams app to rapidly achieve the capability of identifying acute kidney injury. However, the Caldicott Rules clearly state that Streams constitutes “indirect care” affecting patient populations, rather than a “direct care” relationship that relies on access to personally identifiable patient data.

Another NHS collaboration by DeepMind was conducted with Moorfields Eye Hospital in London. Although the partnership did not involve identifiable patient data, the nature of the shared data (detailed optical biometric scans) meant that there was a high risk of personal data breaches. In this instance, approximately one million eye scans were shared with DeepMind to train machine learning models and develop algorithms capable of accelerating the identification of specific eye conditions. 

However, following scrutiny of the data-sharing agreement between DeepMind and Royal Free, an investigation by the Information Commissioner’s Office (ICO) still rated it as “evolving.” Clearly, DeepMind has advanced rapidly in the health sector, even outpacing certain NHS regulatory bodies. Meanwhile, Suleyman confirmed that DeepMind intends to monetize these tools and systems in the future, stating that it is exploring potential payment models based on performance outcomes.