AI in Drug Discovery: Rethinking Success Beyond Efficiency Gains

Over the past three years, we have witnessed and been captivated by the third wave of artificial intelligence (AI) resurgence. This same trend has swept through the relatively conservative pharmaceutical industry, even penetrating the highly technical barrier-laden sector of drug development. In the blueprints of some startup teams, leveraging machine learning algorithms and virtual R&D business models can revolutionize the current inefficient “2 Billions, 20 Years” drug development process, bringing new drugs to patients faster and at lower cost.

But if we shift our gaze away from the future and look back at history, we will find that AI is no stranger to drug development. Since the 1980s, a host of industry giants have been promoting Computer-Aided Drug Design (CADD). Many may be aware that Merck has sponsored several Kaggle Challenges focused on using machine learning to predict P450 enzyme activity. However, the company actually began related efforts more than 30 years ago, even featuring in Fortune magazine in 1981.

In that era, early computational chemists were already using algorithms such as KNN and SVM to perform unsupervised clustering of moieties in drug molecules, attempting to distill the features associated with specific pharmacological effects. Some molecular dynamics programs also sought to leverage machine learning to optimize convergence speed, particularly when addressing the search space in protein conformation docking. These artificial intelligence approaches had long been integrated into commercial QSAR (Quantitative Structure-Activity Relationship) software.

So, what sets apart the current AI resurgence, epitomized by deep learning? If it is indeed different this time, can AI make drug development—a process that has become increasingly frustrating—“different” as well?

Before discussing AI-driven drug discovery, let us outline the entire drug development process. It is typically divided into three stages:

Drug Discovery: Understanding disease mechanisms (or discovering “miracles”) and identifying druggable targets within these mechanisms, which I refer to as the “biology” phase;

Drug Design: Screen for lead small molecules or biological macromolecules suitable for the target, design and optimize the structure of the leads, and then conduct preclinical assays on the molecules, which constitutes the broad “chemistry” phase;

Drug Development: The three phases of clinical trials verify safety and efficacy, followed by post-marketing surveillance and expansion of indications; this can be referred to as the "medical" phase.

Across these three distinct stages, R&D teams operate with different objective functions, giving rise to varying technical requirements and corresponding methodological approaches to address them. Consequently, the application of AI at each stage presents markedly different levels of difficulty and utility.

In the most ideal scenario, biologists can explain disease phenomena in terms of underlying mechanisms and then identify targets to address these mechanisms accordingly. In more practical situations, specific clinical cases or heuristic reasoning can be used to directionally identify druggable targets. In other words, even if the complete network of disease mechanisms is not fully understood, targeting certain hubs within these networks can effectively subdue the disease. This represents the closest approximation to what can be termed a “Magic Bullet.”

In current pharmaceutical R&D, biological researchers face two widening gaps: one is relatively obvious and easier to understand, while the other is more subtle but truly thorny.

A more pronounced gap lies in the translation of biological mechanisms into druggable targets, a process that is far less straightforward than ideally envisioned. We have encountered numerous cases where well-understood mechanisms failed to yield viable drugs, because therapies targeting a single mechanism are often interfered with by other, poorly understood pathways. This can render ostensibly effective treatments completely ineffective or lead to unforeseen adverse effects. Consequently, a significant number of R&D projects stall in the early stages, or, as seen with the tau protein, fail only after entering clinical trials.

Behind this obvious gap lies an even more formidable one: extracting comprehensible mechanisms from the inherent complexity of biological phenomena is far more difficult than being given an electron micrograph of a CPU and asked to reverse-engineer its computational circuitry. The complexity of human bodily operations at the molecular level likely transcends the bounds of human understanding. In a sense, all mechanisms uncovered through biological discovery can only be local mechanisms—“special cases that are more general than mere exceptions.”

This does not imply that these “more general special cases” are incorrect, but their number will be limited. We have already harvested many of the low-hanging fruits, and as we proceed to investigate the underlying mechanisms, we will face a divergence between interpretability and predictability; the decline in predictability may cause confusion in clinical practice.

Existing paradigms in biological research have failed to recognize this difficulty, which makes it understandable why the reproducibility of biological studies has declined significantly since the beginning of this century (excluding the impact of academic misconduct). If one acknowledges the inherent complexity of biological phenomena, it becomes clear that “biological interpreters” have indeed done their best; however, they have been misled by the combined effects of misapplied statistics (such as the recent debate over P < 0.05) and network complexity, thereby failing to derive mechanistic insights that are both deterministic and comprehensible.

And this may be the most fundamental reason for the declining efficiency of drug development, which we will set aside for now.

When AI researchers encounter this term, they are bound to feel a sense of quiet satisfaction. Indeed, screening drug molecules when the target is already known is the most intuitive task for AI to empower. In fact, this has been a battlefield cultivated by computational chemistry and Computer-Aided Drug Design (CADD) for decades, with substantial accumulation of expertise and data. Most AI-driven new drug companies also focus on the “chemistry” stage. Whether engaging in molecular simulation, conformational prediction, QSAR analysis, crystal form optimization, or even further downstream tasks such as retrosynthetic route deduction and production process optimization, these efforts all start from existing prior knowledge to search for and optimize lead compounds.

For this relatively mature process, the potential challenges do not lie in the algorithms, as computational chemists have consistently kept pace with and applied the latest machine learning algorithms. The most pressing issue is that the dimensionality and scale of existing biochemical databases are somewhat limited, yet these are critical factors on which deep learning methods heavily rely. Existing libraries such as DeepChem and ZINC are insufficient to drive deep learning forward in the manner that ImageNet did. Worse still, the reliability of data sources is inadequate; after all, we all know the principle of “garbage in, garbage out.”

Until databases in omics and structural biology are fully refined and corrected, it is difficult to achieve sufficiently reliable virtual screening. Therefore, I am highly optimistic about the large-scale application of mass spectrometry and cryo-electron microscopy, which serves as a foundational infrastructure akin to the nationwide fiber-optic network construction prior to the advent of the internet. Of course, while ab initio computation of novel molecular structures may supplement empirical data, this approach remains constrained by the data structures inherent in existing molecular libraries.

What is even more perplexing is that the tasks AI algorithms aim to accomplish can also be achieved with high efficiency by chemists using heuristic approaches, as they fail to recognize the disruptive value AI could bring to their work. For instance, the most puzzling case is the application of deep learning to retrosynthetic analysis. AI has not demonstrated superiority over human intuition, because the successful execution of synthetic routes relies more on wet-lab practice than on theoretical deduction. The areas AI attempts to optimize are precisely those where manual operations are already most mature and least in need of AI intervention. Applying AI in these stages constitutes involutionary competition, leaving very limited room for meaningful innovation.

Moreover, the “chemistry” stage is not a rate-determining step in the overall drug development process. Any improvements at this stage yield only marginal gains; unless applied to me-too, me-better, or first-to-file generic drugs—where subtle differences in speed are more critical—the resulting benefits are limited. From this perspective, AI algorithms serve primarily as tools in the “chemistry” stage and can generate greater value only when combined with novel mechanisms that have not yet been developed into drugs or with targets that are challenging for drug development.

We do observe that some teams choose to build their own biology units, developing lead candidates from a mechanistic perspective all the way through to acquisition by pharmaceutical companies. However, the challenge lies in the necessity of having a robust biological mechanism as a starting point—a critical requirement that is independent of one’s own machine learning expertise. This creates an awkward predicament.

However, I believe that as an unsupervised learning process devoid of prior knowledge, the most disruptive aspect of machine learning lies in its ability to identify drug molecules capable of tackling diseases even when the underlying mechanisms are unknown to humans, without necessarily providing interpretable mechanistic insights. This is highly challenging, yet it represents the most worthwhile area of exploration. When mechanisms are clearly understood, drug discovery becomes merely a question of speed; whereas, when mechanisms remain unknown, it becomes a challenge of creating something from nothing. It is the latter that truly hinders the efficiency of drug R&D today. To adapt Bill Clinton’s famous campaign slogan, I would say: “It’s biology, stupid!”

However, biology is a "Hard Problem," and currently, few AI drug discovery teams set their sights on it.

If the goal of the chemical phase is to “Be better,” then the objective of the medicinal chemistry phase appears to narrow further, becoming “Be usable.” However, the reality is that most drug molecules fail to clear this hurdle. In drug development, there is a proverb: “Fail fast, fail early.” This is actually a desirable outcome, because failure at later stages—such as Phase II or even Phase III clinical trials—would result in substantial losses capable of devastating a company’s market valuation.

This may seem absurd: if the underlying biological mechanisms and chemical optimization have been thoroughly refined in preclinical stages, why do drugs fail when tested in humans? Yet the reasons behind this apparent absurdity are profound. The effects of drug molecules within the complex human system—particularly in populations characterized by genetic and omics heterogeneity—are elusive, and their usability can vary across different dimensions.

If most drugs fail in Phase II clinical trials or later, the primary issue may not be that the drug itself is ineffective, but rather that it was administered incorrectly to the wrong patient population. Many drugs do not completely fail in clinical trials; if we can identify where we went wrong, it is possible to achieve new clinical endpoints by adjusting the dosing regimen and indications.

The true value lies in the ability to cut losses promptly, select appropriate indications in a timely manner, and improve success rates. This can be achieved by leveraging machine learning-driven insights into patient profiling. Before clinical trials commence, it is possible to assess the drug’s applicability in the broader population, identify which subgroups defined by specific markers are likely to respond, and, most critically, determine which marker-defined populations and clinical endpoints are unlikely to show efficacy. Such insights are worth hundreds of millions in Phase II and later-stage clinical trials.

As can be seen, the primary bottleneck in the current drug development process lies at the interface between the biological and clinical stages, whereas the chemical stage is relatively mature. Consequently, optimizing only this non-bottleneck segment will not significantly improve the time efficiency or return on investment of drug development.

Therefore, I believe that if AI-driven drug discovery projects are merely an extension of past computational chemistry simulations, omics, and automation in drug development—using AI tools to optimize and enhance existing R&D processes—this is certainly the most feasible initial strategy, but it does not represent a particularly valuable market. These AI startups targeting the “chemistry” phase of drug discovery are generally pursuing tasks that are easy to implement but not essential. If the sole benefit is improving current drug R&D efficiency, then the valuations of AI drug discovery companies are clearly somewhat inflated.

Large pharmaceutical companies’ attention to and support for these startups are driven less by confidence in the technology and more by financial KPI considerations. By investing in AI-driven drug discovery companies, they outsource non-core R&D activities to CROs and AI-enabled “virtual biotech” startups. This approach leverages the pharma giants’ strengths—ample cash flow and the capacity to take over clinical development—while the resulting gains do not immediately appear on the income statement. Instead, earnings are smoothed through acquisition-related goodwill adjustments, thereby improving the appearance of financial statements. From an investment perspective, I certainly agree with this business logic.

However, the true high-value opportunity lies in using AI to reconstruct the overall logic of drug discovery and development, which can be pursued along two directions:

1. At the biological stage, transcend the constraints of interpretability to more efficiently discover novel mechanisms and effective new targets through unsupervised learning, thereby expanding the druggable space.

2. In the medical development phase, integrate patient profiling into clinical trial decision-making to enhance the likelihood of accurate drug positioning and regulatory approval, thereby enabling early identification and termination of futile drug candidates with poor prospects to avoid costly failures in Phase III trials.

Both directions essentially reflect the same philosophy: AI should be leveraged to enhance the success rate of drug discovery and development, rather than merely improving operational efficiency. These two objectives differ fundamentally in quality versus quantity. If we can directly confront the “Hard Problems” in the “biology” and “medicine” stages and achieve disruptive innovation, I believe this will deliver far greater value than any incremental innovations made during the “chemistry” stage. Let us strive together toward this goal!

Author: Liu Zheng, Analyst at Timbre Capital, Singapore

Author's Email: justinian@timbrecapital.com.sg