Graph Transformer Foundation Model for modeling ADMET properties

Pre-trained on vast datasets, foundation models demonstrate strong generalization across different tasks, particularly in natural language processing and computer vision. This study explores the potential of applying foundation models to the domain of ADMET (Absorption, Distribution, Metabolism, Excretion, and Toxicity) property prediction, a critical task in early-stage drug discovery. We present a Graph Transformer Foundation Model (GTFM) that combines the strengths of graph neural networks (GNNs) and transformer architectures to model molecular graphs. We use self-supervised learning (SSL) to extract useful representations from large unlabeled datasets by predicting masked nodes/edges and the Joint Embedding Predictive Architecture (JEPA). The latter demonstrates superior performance in most tasks by learning robust and predictive features from molecular graphs. The GTFM is benchmarked against classical machine learning models using predefined molecular descriptors. The results demonstrate that the GTFM, especially when employing JEPA, outperforms classical approaches for ADMET property prediction in 8 out of 19 classification and 5 out of 9 regression tasks, being comparable in the rest. This shows foundation models, specifically GTFM, as a promising approach for ADMET modeling, providing a scalable and versatile solution for drug discovery applications.