Librarian of Alexandria: An Extensible LLM-based Chemical Data Extraction Pipeline

Dataset creation is a critical component of predictive machine learning technology. In the field of chemistry, large experimental datasets are scarce, especially for niche chemical properties and topics, and their creation is both cumbersome and time-consuming. Here we present Librarian of Alexandria (LoA), an open-source automatic tool for large dataset generation via direct extraction by large- language models (LLM) from scientific literature. LoA is available on GitHub along with example inputs, outputs and a Colab-friendly Jupyter Notebook (see Supplementary Information). LoA checks the relevance of a research paper and performs data extraction using two separate LLMs which may be independently and modularly specified by the end-user. LoA can be easily updated via simplified user incorporation of the latest available LLMs. We also compared several LLMs for both relevance and extraction functions, and automated the collection of research papers for several popular chemical journals which provide open access. LoA provides a means by which enormous chemical datasets can be created with minimal effort and ∼80% accuracy for the purposes of training predictive machine learning models.