Expanding Molecular Design with Graph Variational Autoencoders: A Comparative Study of Pair-Encoding and Character Tokenization

Deep generative models are transforming early-stage drug discovery, yet most current approaches are not well suited for realistic, small-data settings and often rely on simplified molecular representations such as linear strings, overlooking the inherent graph-based structure of molecules. To address this, we first developed the Transformer Graph Variational Autoencoder (TGVAE), an AI model that learns directly from molecular graphs and generates Simplified Molecular Input Line Entry System (SMILES) strings. Building upon this foundation, here we introduce the improved architecture of TGVAE (TGVAEv2) and a DeeperGAT-VAE (DGVAE) model, an upgraded architecture that incorporates lightweight and deeper graph-attention blocks, designed to enable future applications on smaller molecular datasets. Both models address common challenges in training, such as over-smoothing in graph neural networks and posterior collapse in VAEs, ensuring stable and chemically precise molecule generation. Across standard benchmarks, TGVAEv2 and DGVAE achieve high validity, uniqueness, diversity, and novelty, while reproducing key drug-like property distributions. In addition to architectural improvements, we also expanded the input tokenization by incorporating SMILES pair-encoding, which captures larger and chemically relevant substructures compared to character-level tokens and supports the generation of more diverse and novel molecules. By comparing both approaches, we present how the DGVAE can achieve improved performance with fewer parameters and faster convergence while maintaining chemical accuracy. Evaluation against PubChem confirms that SMILES pair-encoding greatly expands the space of scaffolds and fragments unseen in public databases. These advancements not only broaden the accessible chemical space but also set a new benchmark for the application of graph-based AI models in molecular generation and drug discovery.