Theoretical and Computational Chemistry

Augmented Hill-Climb increases reinforcement learning efficiency for language-based de novo molecule generation

Authors

Abstract

A plethora of AI-based techniques now exists to conduct de novo molecule generation that can devise molecules conditioned towards a particular endpoint in the context of drug design. One popular approach is using reinforcement learning to update a recurrent neural network or language-based de novo molecule generator. However, reinforcement learning can be inefficient, sometimes requiring up to 10^5 molecules to be sampled to optimize more complex objectives, which poses a limitation when using computationally expensive scoring functions like docking or computer-aided synthesis planning models. In this work, we propose a reinforcement learning strategy called Augmented Hill-Climb based on a simple, hypothesis-driven hybrid between REINVENT and Hill-Climb that improves sample-efficiency by addressing the limitations of both currently used strategies. We compare its ability to optimize several docking tasks with REINVENT and benchmark this strategy against other commonly used reinforcement learning strategies including REINFORCE, REINVENT (version 1 & 2), Hill-Climb and best agent reminder. We find that optimization ability is improved ~1.5-fold and sample-efficiency is improved ~45-fold compared to REINVENT while still delivering appealing chemistry as output. Diversity filters were used, and their parameters were tuned to overcome observed failure modes that take advantage of certain diversity filter configurations. Lastly, we find that Augmented Hill-Climb outperforms the other reinforcement learning strategies used on six tasks, especially in the early stages of training or for more difficult objectives. Overall, we hence show that AHC improves sample-efficiency for language-based de novo molecule generation conditioning via reinforcement learning, compared to the current state-of-the-art. This makes more computationally expensive scoring functions, such as docking, more accessible on a relevant timescale.

Content

Thumbnail image of AugHC ChemRxiv.pdf

Supplementary material

Thumbnail image of AugHC Additional File 1 - Supplementary information.pdf
Supporting information
Supplementary tables and figures

Supplementary weblinks

Supporting data
Supporting data including the prior training datasets, trained priors and results.
SMILES-RNN
GitHub repository containing code used to generate the results.