![Author ORCID: We display the ORCID iD icon alongside authors names on our website to acknowledge that the ORCiD has been authenticated when entered by the user. To view the users ORCiD record click the icon. [opens in a new tab]](https://chemrxiv.org/engage/assets/public/chemrxiv/images/logos/orcid.png)
Abstract
Machine learning has become a crucial tool in drug discovery and chemistry at large, e.g., to predict molecular properties, such as bioactivity, with high accuracy. However, activity cliffs – pairs of molecules that are highly similar in their structure but exhibit large differences in potency – have been underinvestigated for their effect on model performance. Not only are these edge cases informative for molecule discovery and optimization, but models that are well-equipped to accurately predict the potency of activity cliffs have an increased potential for prospective applications. Our work aims to fill the current knowledge gap on best-practice machine learning methods in the presence of activity cliffs. We benchmarked a total of 720 machine and deep learning models on curated bioactivity data from 30 macromolecular targets for their performance on activity cliff compounds. While all methods struggled in the presence of activity cliffs, machine learning approaches based on molecular descriptors outperformed more complex deep learning methods. Our findings highlight large case-by-case differences in performance, advocating for (a) the inclusion of dedicated “activity-cliff-centered” metrics during model development and evaluation, and (b) the development of novel algorithms to better predict the properties of activity cliffs. To this end, the methods, metrics, and results of this study have been encapsulated into an open-access benchmarking platform named MoleculeACE (Activity Cliff Estimation, available on GitHub at: https://github.com/molML/MoleculeACE). MoleculeACE is designed to steer the community towards addressing the pressing but overlooked limitation of molecular machine learning models posed by activity cliffs.
Version notes
We elucidated the reasoning behind our taken splitting approach in-depth and mention the effects of the splitting approach on 1. data distributions, 2. the presence of activity cliffs in the test and train set, and 3. the occurrence and effects of all activity cliff ‘partners’ ending up in the test set. We included a supporting table. Secondly, to assess any bias of our splitting approach in favor of ECFP descriptors (used for splitting), we compared similarities between the train and test set for different molecular descriptors. We found no significant differences for other descriptors. We fixed several typos.
Content
Supplementary weblinks
MoleculeACE
MoleculeACE (Activity Cliff Estimation) is an open-access benchmarking platform to: (1) Systematically benchmark a model’s performance on activity cliffs compounds, in comparison with well-established machine- and deep learning methods; (2) evaluate the deck of chosen models on a new dataset not included in our benchmark, thanks to the data collection and curation pipeline; and (3) further expand the definition of activity cliffs, based on specific use cases.
