Targeted Molecular Generation With Latent Reinforcement Learning

Computational methods for generating molecules with specific physiochemical properties or biolog- ical activity can greatly assist drug discovery efforts. Deep learning generative models constitute a significant step towards that direction. In this work, we introduce a novel approach that utilizes a Reinforcement Learning paradigm, called proximal policy optimization (PPO), for optimizing chemical molecules in the latent space of a pre-trained deep learning generative model. Working in the latent space of a generative model allows us to bypass the need for explicitly defining chemical rules when computationally designing molecules. The generation of molecules with desired properties is achieved through navigating the latent space for identifying regions that correspond to molecules with desired properties. Proximal policy optimization is a state-of-the-art policy gradient algorithm capable of operating in continuous high dimensional spaces in a sample-efficient manner. We have paired our optimization framework with the latent spaces of two autoencoder models, a variational autoencoder and an autoencoder trained with mutual information machine loss respectively, showing that the method is agnostic to the underlying architecture. We present results on commonly used benchmarks for molecule optimization that demonstrate that our method has comparable or even superior performance to state-of-the-art approaches. We additionally show how our method can generate molecules that contain a pre-specified substructure while simultaneously optimizing for molecular properties, a task highly relevant to real drug discovery scenarios.