How good are current pocket based 3D generative models? : The benchmark set and evaluation on protein pocket based 3D molecular generative models

The development of three-dimensional (3D) molecular generative model based on protein pockets has recently attracted a lot of attentions. This type of model aims to achieve the simultaneous generation of molecular graph and 3D binding conformation under the constraint of protein binding. Various pocket based generative models have been proposed, however, currently there is a lack of systematic and objective evaluation metrics for these models. To address this issue, a comprehensive benchmark dataset, named as POKMOL-3D, is proposed to evaluate protein pocket based 3D molecular generative models. It includes 32 protein targets together with their known active compounds as a test set to evaluate the versatility of generation models to mimick the real-world scenario. Additionally, a series of 2D and 3D evaluation metrics was integrated to assess the quality of generated molecular structures and their binding conformations. It is expected that this work can enhance our comprehension of the effectiveness and weakness of current 3D generative models, and stimulate the discussion on challenges and useful guidance for developing next wave of molecular generative models.