The Ethical Dilemma of Medical AI: Navigating Bias, Accountability, and Privacy in the Age of Augmented Intelligence

2,400 Years After the Hippocratic Oath Laid the Foundations of Medical Ethics, the Emergence of Artificial Intelligence May Pose the Greatest Challenge in the History of Medical Ethics.

Experts predict that by 2024, the AI healthcare market will reach nearly $20 billion. Artificial intelligence is poised to become a boon for medical practice, enhancing diagnostic accuracy, enabling personalized treatment, and facilitating the timely identification of future public health risks.

Even so, the technology still raises a series of thorny ethical dilemmas. What issues arise when AI systems make erroneous decisions? If problems occur, who should be held accountable? How can clinicians verify or even understand the contents of the AI “black box”? And how can they mitigate bias in AI systems while safeguarding patient privacy?

In June 2018, the American Medical Association (AMA) released its first guidelines on the development, use, and regulation of artificial intelligence. Notably, the association refers to this technology as “augmented intelligence” rather than the widely recognized term “artificial intelligence.” This indicates that the AMA views the role of such technology as augmenting, rather than replacing, the work of physicians.

Although the AMA states in its guidelines that artificial intelligence should be designed to identify and address bias, safeguard the needs of vulnerable populations, ensure process transparency, and protect patient privacy, these requirements are difficult to meet in practical implementation. The following are the most urgent ethical challenges facing medical practitioners, researchers, and medical ethicists.

In 2017, the data analytics team at the University of Chicago Medicine (UCM) employed artificial intelligence to predict patients’ potential length of stay. The objective was to identify patients eligible for early discharge, thereby freeing up hospital resources and making room for new admissions. Subsequently, the hospital assigned a case manager to assist patients with insurance matters, ensure their timely return home, and facilitate their early discharge.

When testing the system, the research team found that the most accurate predictor of patients’ length of hospital stay was their ZIP code, which immediately raised red flags for the researchers. They recognized that ZIP codes are closely correlated with patients’ race and socioeconomic status. Relying on ZIP codes for predictions would adversely affect African Americans in Chicago’s poorest communities, who tend to have longer hospital stays. Consequently, the team concluded that using this algorithm to assign case managers would be biased and unethical.

“If you were to implement this algorithm in practice, you would arrive at a paradoxical result: allocating more (case management) resources to wealthier white patients,” said Marshall Chin, an internist at UCM and professor of medical ethics.

The data analysis team ultimately removed zip code as a predictive factor. The algorithm is still under development, and the new model has not yet been tested.

This case highlights the weaknesses of AI-based healthcare tools: algorithms often reflect existing racial or gender health disparities. If this issue is not addressed, it could lead to long-term bias and entrench existing inequalities in the healthcare sector.

Bias can also affect the treatment of rare or novel diseases, for which therapeutic data are limited. Artificial intelligence systems may directly propose general treatment regimens without considering individual patient circumstances, rendering the AI-recommended treatments ineffective.

Recently, Danton Char, an Assistant Professor of Anesthesiology at Stanford University, pointed out in a paper on machine learning that because the survival rates for patients with severe brain injury or extremely premature infants are very low, doctors often withdraw care. Even if individual patients have a favorable prognosis, machine learning algorithms may still directly conclude that all similar cases are fatal and recommend withdrawing treatment.

The second ethical challenge is that, in most cases, researchers do not understand how AI systems arrive at their results—a phenomenon known as the “black box” problem. Advanced machine learning techniques can ingest vast amounts of data and identify statistical patterns without explicit instructions, a process that is particularly difficult for humans to validate. Physicians who blindly follow such systems may inadvertently harm patients.

“It is often difficult for us to understand the ‘thought’ process of algorithms,” said Eleonore Pauwels, a researcher on emerging network technologies at the United Nations University Institute for New Technologies.

A 2015 study highlighted this issue. In the study, researchers compared the performance of different AI models in predicting mortality risk among patients with pneumonia. Based on the predictions, high-risk patients would be hospitalized, while low-risk patients could be managed in outpatient settings.

One of the models is a “rule-based” system whose decision-making process is transparent to researchers, yet it yields counterintuitive results: patients with both pneumonia and asthma have a higher survival rate than those with pneumonia alone, suggesting that treatment for patients with both conditions could be delayed. It is evident that healthcare professionals can clearly recognize that patients with both conditions face a higher risk of mortality, whereas the algorithm cannot. Relying solely on such an algorithm would mean that the most critically ill patients would not receive the necessary treatment in a timely manner.

Another model that employs neural networks and machine learning algorithms yielded more accurate results, but its reasoning process is opaque, preventing researchers from promptly identifying potential issues. Richard Caruana, a researcher at Microsoft and the lead author of the study, concluded that neural network models pose too great a risk to enter clinical trials, as there is no way to determine whether they have made similar errors.

According to the AMA’s fundamental principles of medical ethics, physicians must assume full responsibility for their patients. However, when artificial intelligence enters the equation, how should liability be allocated? The answer to this question is still being formulated by ethicists, researchers, and regulatory bodies.

Artificial intelligence has broken down the traditional boundaries of healthcare provider groups, enabling individuals who are not traditionally bound by medical ethics—such as data scientists—to deliver medical services to patients. Furthermore, as illustrated by the “black box” problem, it is not always possible to determine exactly how an AI system arrives at a diagnosis or prescribes a treatment regimen. Flawed algorithms may cause significant harm to patients, thereby leading to medical malpractice.

Stanford anesthesiologist Char likens artificial intelligence to prescription medication. Char stated that while clinicians cannot be expected to understand every biochemical detail of the drugs they prescribe, they must at least know, based on their clinical experience and knowledge of medical literature, that these medications are safe and effective. As for AI systems, he would not use them unless thorough research convinced him that they were the optimal choice. “When you do not have a sufficient understanding of a tool, you are unwilling to put any patient’s life at risk,” Char said.

The American Medical Association has issued a warning: Artificial intelligence must safeguard the privacy and security of patient information. The commitment to physician-patient confidentiality has been the cornerstone of medical ethics since the Hippocratic Oath.

However, to make accurate predictions, machine learning systems must have access to vast amounts of patient data. Without individual medical records, artificial intelligence will be unable to provide accurate diagnoses or effective treatment plans, let alone deliver more personalized care. More importantly, if millions of patients withhold their medical data, critical public health trends may go unnoticed, resulting in a loss for everyone.

A potential solution is to protect patient privacy by separately removing personally identifiable information from medical records. However, a recent study led by the University of California indicates that current anonymization techniques are not yet mature enough to guarantee the effective removal of such data. Nevertheless, more sophisticated data collection methods could be developed in the future to better safeguard privacy.

Regardless of technical capabilities, medical experts recommend that the medical community reconsider the entire concept of patient privacy. As healthcare systems become increasingly complex, more institutions will have legitimate and reasonable needs to access sensitive patient information. “The implementation of machine learning systems means that we need to re-examine medical data privacy and other core principles of professional ethics,” wrote Char in the paper.

In practice, hospitals and institutions need to earn the trust of patients. Patients have the right to know how their medical privacy data is used, and whether the data will benefit themselves or only future patients.

Nathan Lea, a senior research fellow at the UCL Institute of Health Informatics, stated, “If patients had a better understanding of how artificial intelligence is improving individual and public health, they might be more willing to relinquish traditional notions of privacy. Privacy itself is not absolute; we cannot use the protection of patient privacy as an excuse to reject the substantial value embedded in data.”